Homebrew Package Installation for Servers

Homebrew logo

One of our customers came to us a couple of weeks ago wanting to run Virtualmin on Mac OS X. I, foolishly, said, “Sure! I can help with that. I installed Virtualmin on OS X several years ago, how much different can it be? It shouldn’t take more than a couple of hours.” It turns out, it can be remarkably different. In the intervening years, Mac OS X has evolved in many interesting directions, mostly positive, some questionable. Mac OS X remains exceedingly weak for server usage, for reasons well out of the scope of this short article. But, it is quite strong for desktop/laptop use, and many people want to be able to develop their web applications on Mac OS X even if they will be deployed to Linux servers.

Enter Homebrew

The last time I setup Virtualmin on Mac OS X systems, the best package management tool, and the best way to install a lot of Open Source software, was Fink (Mac Ports, then Darwin Ports, was still quite new at the time).

Fink is an apt repository of dpkg packages built for Mac OS X. I love apt and dpkg (almost as much as yum and rpm), and Webmin/Virtualmin have great support for apt and dpkg, and so I was all set to choose Fink for this new deployment. But, there are some issues with Fink. First, it installs everything with its own packages, including stuff that is already available from the base system. For Virtualmin, whenever possible, I like to use the system standard packages for the services it manages. Homebrew is designed to work with what Apple provides when possible, which is somewhat more aligned with the Virtualmin philosophy.

Second, and perhaps more important, Fink is seemingly much less popular and much less actively maintained than Homebrew. I’m not sure why. Possibly because the Homebrew website looks good, and the documentation is very well-written, while the Fink home page is a little drab and looks complicated. And, Fink package versions tend to be quite a bit older than the packages provided by the very active Homebrew project. This can be a much more serious issue. Security updates are absolutely vital in a web server, and a package repository that is actively maintained is the best way to insure you’ll have security updates.

So, I’ve spent the past couple of days experimenting with Homebrew. It’s a pretty nice system, and its community and developers are active, responsive, and helpful. All great things. But, its primary advertised feature is also its biggest weakness and most dangerous mistake.

Installation Without sudo

Or: Homebrew Considered Harmful

One of the major advertised features of Homebrew is that you can install it, and any package, without root or sudo privileges. There are good reasons one might want this, but on a server, it has alarming side effects, and it is one of the first things I would need to correct for our use case (of installing a virtual hosting stack and Virtualmin). The example I’ll use here is that of MySQL. When you install the mysql package from Homebrew, it will be installed with the ownership of all files set to the user that installed the package. And, more dangerously, it will be setup to run as the user that installed it.

This decision was made because Homebrew often builds the software at install time, rather than providing a binary package (there is a new “bottles” feature installs binary packages, but that wasn’t intended to address the sudo problem). The risk of building software with sudo or root privileges is very real, and in this case it results in the choice to build the software as a non-root user.

Other package managers, like dpkg and rpm, resolve this problem with toolchains designed around building the packages within a chroot so that unwanted behavior is contained. For example, mock on Fedora and CentOS provides an easy to use tool for building packages across many distributions and versions inside of a chroot environment with only the dependencies specified by the package. The most popular Linux distributions distribute binary packages that were built in a controlled environment. But, Homebrew generally builds the software at install time, with no chroot to protect the system from broken or hostile build processes. And, so they insist you run it as a non-root user. This is, I suppose, a logical conclusion to come to, based on the premise of a package manager that builds software on the user’s system without being confined in a container or chroot, but it has negative consequences.

For example, when I install MySQL from Homebrew, everything is owned by joe:staff. The provided property list file for starting the server is also designed to start it as that user, when the user logs in. For a development system, this may not be a big deal, and even makes a certain sort of sense (I prefer my development environment to more closely mirror my deployment environment, but I can see reasonable arguments for the way they do it). But, for a server, it is simply untenable.

The most important reason it is a bad decision is that it leads to many, possibly all, of the services running with the privileges of the user that installed them. Which, in most cases, is probably a powerful user (mine is an administrative user with sudo privileges, for example). So, in the event any of the services are compromised, all of the services will be compromised, and likely so will the user account in question. The security implications of this really cannot be overstated. This is a huge problem.

This is why Linux and UNIX systems (and even Apple, who aren’t historically renowned for their strong multi-user security practices) run all services as different users, and with restricted privileges. On the average LAMP system, there will be an apache or www user that runs Apache, a mysql user that runs MySQL, a nobody user that runs Postfix, and web applications will usually be run as yet another user still. These special users often have very restricted accounts, and may not even have a shell associated with them, further limiting the damage that can occur in the event of an exploit of any one service. Likewise, they may be further restricted by SELinux or other RBACL-based security. Any one of these services or applications being compromised through any means won’t generally compromise other services or users. Homebrew throws that huge security benefit away to avoid having to sudo during installation.

It’s probably too late to convince the Homebrew folks to backtrack on that decision. But, it’s not terribly difficult to fix for one-off installations, and many do consider it a valuable feature of Homebrew. Fixing the installed services as I’ve done has some side effects that may also be dangerous, which I’ll go into at the end of the article, but since I figured out how to do it, I thought I’d document it. During my research I found that an alarming number of users are using Homebrew in server environments and I found a number of users asking similar questions about various services, so, maybe this will help some folks avoid a dangerous situation.

So, let’s get started. After installation of MySQL (using the command brew install mysql), here’s the changes you’ll want to make.

Update The Property Lists File

The way Homebrew recommends running MySQL after installation is to link the provided plist file in /usr/local/opt/mysql/homebrew.mxcl.mysql.plist into your ~/Library/LaunchAgents directory, and add it using the launchctl load command. This sets it up to run at all times when your user is logged in, which is great if you’re developing and only need it running when you’re logged in and working. But, we want it to run during system boot without having any users logged in, and even more importantly we want it to run as the _mysql user.

So, instead of linking it into your local LaunchAgents directory, as the documentation suggests, copy it into your system /Library/LaunchDaemons directory.

$ sudo cp /usr/local/opt/mysql/homebrew.mxcl.mysql.plist /Library/LaunchDaemons

Then edit the file to add user and group information (you’ll have to use sudo), add a –user option, and change the command to mysqld and WorkingDirectory to /usr/local/var/mysql. Mine looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>UserName</key>
  <string>_mysql</string>
  <key>GroupName</key>
  <string>_mysql</string>
  <key>KeepAlive</key>
  <true/>
  <key>Label</key>
  <string>homebrew.mxcl.mysql</string>
  <key>ProgramArguments</key>
  <array>
    <string>/usr/local/opt/mysql/bin/mysqld</string>
    <string>--bind-address=127.0.0.1</string>
    <string>--datadir=/usr/local/var/mysql</string>
    <string>--user=_mysql</string>
  </array>
  <key>RunAtLoad</key>
  <true/>
  <key>WorkingDirectory</key>
  <string>/usr/local/var/mysql</string>
</dict>
</plist>

Notice the addition of the UserName and GroupName keys, both set to _mysql, as well as several other altered lines.

Note: I am not a Mac OS X or launchd expert. There are a number of aspects of the Mac OS X privilege model that I do not understand. I would welcome comments about how the security of this configuration might be improved. Also, the entirety of my experience with launchd is the several hours I spent playing with it and reading about it to convince MySQL to run as a different user. But, I’m pretty much certain that the way Homebrew does servers is worse than what I’ve done here.

Change Ownership of MySQL Installation and Databases

The _mysql user does not have permissions to read things owned by the joe user (or the user you used to install MySQL with Homebrew), so you’ll need to change ownership of all MySQL data files to _mysql:wheel.

$ sudo chown -R _mysql:wheel /usr/local/var/mysql/

Change Ownership of the Property List File

Property list files in the Library/LaunchDaemons directory (or the LaunchAgents directory) must belong to root, for security reasons. So, you’ll need to update that, as well.

$ sudo chown root:wheel /Library/LaunchDaemons/homebrew.mxcl.mysql.plist

Load and Start the MySQL Daemon

The launchctl command manages agents and daemons, and we can add our new service by loading the property list:

$ sudo launchctl load /Library/LaunchDaemons/homebrew.mxcl.mysql.plist

And, then we can start MySQL:

$ sudo launchctl start homebrew.mxcl.mysql.plist

If anything goes wrong, check the system.log:

$ sudo tail -f /var/log/system.log
I found the documentation at launchd.info particularly helpful when working out how to use launchd.

Future Concerns

Since this corrects the big security issue with the Homebrew installation of MySQL, and this technique could reasonably easily be applied across every Homebrew-installed service, why aren’t we happy?

Updates are part of security, too.

The most important reason to use a package manager is not to make it easy to install software, contrary to popular belief, and if all the package manager does is provide easy to install packages, it is not an effective package manager. The most important reason to use a package manager is to make it easy to update software.

Homebrew makes installation and upgrades reasonably easy, but the steps I’ve taken in this article to make MySQL run as its own user seems likely to break updates, since some files created during installation have changed ownership. A newer version of MySQL isn’t available in the Homebrew repository, so I can’t test whether it does break upgrades or not. Nonetheless, fixing this issue compounded across many services (the Virtualmin installation process normally installs: Apache, MySQL/MariaDB, PostgreSQL, Postfix, Dovecot, BIND, Procmail, SpamAssassin, ClamAV, Mailman, and a bunch more) will likely prove to be a maintenance challenge that is probably not worth the effort.

So, despite having figured out how to make this work, I’m now going to spend the same amount of time and effort giving Mac Ports a thorough test drive. I have a pretty strong suspicion it will be a better fit for server usage. This way of working feels like fighting against the way Homebrew wants to operate, and when you find yourself having to work so hard against the tool, it’s probably the wrong one for the job.

And What Does All This Mean for Virtualmin on Mac OS X?

Well, I’m a sucker for the sunk cost fallacy, so I’m planning to spend another couple of days working out a basic install script for Virtualmin on Mac OS X, probably using Mac Ports (I can see a way forward using Homebrew, but I don’t like the terrain). I’ll likely never recommend Mac OS X for a production server deployment, but it’s certainly not the worst OS out there for the purpose.

Multi-account Twittering from the command line

While working on our migration from Joomla to Drupal over at Virtualmin.com, I’ve been keeping folks apprised of what I’m up to by posting the highlights to Twitter on our virtualmin Twitter account. So, I’ve had to switch back and forth quite a bit from my personal account. It also means I have to have another tab open quite frequently for Twitter. I looked around for a standalone Twitter client that supports multiple accounts, and found the pickings were pretty slim on Linux (a few Air clients run on Linux, but not very well, unfortunately). But, I got to thinking that a command line client would be ideal…so I searched the web, and found a brief article here with a simple one-liner to submit a tweet to Twitter. The fact that Twitter works with a one-line shell script is awesome, and explains why there are so many Twitter clients.

Anyway, I first thought, “I could just make copies of that for each account.” The I could call, “twitter-swelljoe” for my personal account and “twitter-virtualmin” for my business account. But that seemed wrong, somehow.

So, I wrote a version that accepts a command line argument of the username, and then sends the rest as a tweet.

#!/bin/bash
USER=$1
shift
TWIT=$@
PASS=password
curl -u $USER:$PASS -d status="$TWIT" http://twitter.com/statuses/update.xml>/dev/null

Nothing really fancy going on here, but the “shift” is saying, “drop the first token from the arguments list”, and then $@ says “give me the rest”. So, no quotes required for the tweet. I saved the filed as tw, and use it like so:

tw swelljoe Tweeting from the command line!
tw virtualmin Business-y kinds of tweets from the command line.

This assumes one password for all Twitter accounts, which is exactly what I do anyway (I use SuperGenPass, which always generates the same password when given the same URL and passphrase). You could instead store the passwords in an associative array (gotta be using Bash 4.0+ for that), something like this (I can’t test this, as I don’t have bash 4.0!):

#!/bin/bash
USER=$1
shift
TWIT=$@
 
declare -A PASS
PASS[swelljoe]="passwd"
PASS[virtualmin]="notapasswd"
 
curl -u $USER:${PASS[$USER]} -d status="$TWIT" http://twitter.com/statuses/update.xml>/dev/null

There are ways to fake associative arrays in older versions of bash, or use other tools, but since I don’t need multiple passwords, I’m going to just hand-wave that away.

I find it amusing that the first version of this twitter client, at 137 characters, is a valid tweet. It makes me want to go golfing and see how small I can make an even more functional version…at the very least, I can’t resist:

tw swelljoe `cat tw`

And then I have to go change my password after realizing I tweeted the real version rather than the demonstration version. Good thing nobody follows my personal tweets.

A couple of nice articles about Webmin and Virtualmin

I use Google’s awesome Alerts feature to keep an ear out for anything interesting on the web about some of my favorite topics.  Obviously, Webmin and Virtualmin are among those topics.

A great article about Webmin showed up today at Free Software Magazine, by Gary Richmond, that is a “bird’s eye view” of the subject, just touching on the high points.  In it, he gets to the heart of some of the things that make Webmin great, like it’s ability to edit configuration files directly and safely, rather than generating them from templates, as most similar configuration tools do.  All in all, a nice little “boss-friendly, while still being accurate” introduction to Webmin.

And Monday, the awesome blog The Next Web covered Virtualmin, the business and the project, kicking off their new series on profitable web businesses (rather than those that are running on dreams and VC cash) with this article: Companies who make money: Virtualmin. Lots of great stuff over there.

And, of course, no discussion of the blogosphere would be complete without mentioning the new browser from Google. I haven’t tried it yet, as it’s only available for Windows and I’m a Linux user when I’m working, but Jamie tells me that Webmin works just fine (expected) and there was only one quirky bit in the Virtualmin theme (surprising, since we test against Konqueror and Safari, both WebKit browsers, but easily fixed). So, if you’re running Windows, give it a go: Chrome. If you’re an IE user, please give it a go.

Disqus comments

I’ve just converted In the box over to the Disqus comment network, which provides a really nice comments UI, good spam filtering, and a discussion board. And, with the new version of their WordPress API it integrates seamlessly. It also allows anyone with a Disqus account to comment without signing up for yet another blog. It’s a cool service, and makes time spent commenting on blogs more effective.

Community, or not community

I chatted with an applicant to the Y Combinator Summer Founders Program this afternoon, and he brought up a really interesting question that I’d never really thought about before.  His question was, roughly, “Do community-based or value-based Y Combinator backed businesses do better?”  Let’s assume that a “community-based” business is one where the user base creates the vast majority of the value (like reddit), and a “value-based” business is one where there is value even with only one user (like Zenter).  So, by this definition, a community-based site must reach a certain level of popularity before random folks on the Internet want to use it, while a value-based site can make life easier for a single person and that single person will keep coming back.

The reason this is interesting, to me, is that Y Combinator likes community-based companies, and my impression (admittedly supported by merely anecdotal and circumstantial evidence) is that they prefer them to value-based businesses, though this preference may be changing over time.  So, the question it raises is whether having a preference (even a mild one) for community-based businesses results in better outcomes for Y Combinator.  I suspect that it has a negative impact, but I don’t know that for sure.

We can break it down by exit, though:

Community-based YC exits

  • Reddit – Rumored 12-ish million dollar acquisition by Condé Nast

Group of men at Ellis Island

Value-based YC exits

  • Zenter – Undisclosed acquisition by Google
  • Parakey – Undisclosed acquisition by Facebook
  • Auctomatic – 5 million dollar cash+stock acquisition by Live Communicate
  • TextPayMe – Undisclosed acquisition by Amazon
  • Anywhere.fm* – Undisclosed acquisition by Imeem

Farmer turning sod

I’ve put an asterisk by Anywhere.fm above, as I’m certain some folks would consider it a community-based business…but it provides most of its value even with only one user, so it fits my requirements for a value-based business, and I think it’s fair to say they were not acquired based predominantly on the size of their community or traffic growth but on the strength of the application they’d built.

Of course, since all but one of those acquisitions in the “value-based” category are for an undisclosed sum, and I’m basing the reddit acquisition price on rumors (though rumors from disparate sources, I’ve never heard the numbers from any of the people actually involved), it’s hard to say with any confidence that the five exits in the value-based group are worth more to YC than the one community-based exit.  But, one can safely say that value-based businesses have made a lot more young YC-backed entrepreneurs rich than community-based businesses.

This isn’t the whole picture, of course.  Some of the biggest potential YC success stories have yet to have an exit.   It’s hard to argue that Loopt isn’t going to  be a big success story for YC, and it’s pretty much purely community-based, though they have the added hurdle of having to sell to mobile providers to attain their customer base.  Scribd has seen tremendous traffic growth and it’s a community-based site with a small amount of value independent of the community.  Obviously, “going viral” is a fantastic marketing opportunity that is only generally available to community-based businesses, but historically it seems that building a useful application is more likely to result in a successful exit than building a product that relies on a large community to have value.

I found this result surprising when I began listing off the exits and thinking about what kind of business they were.  I’ve always thought of Virtualmin as an outlier among YC companies, since we build a traditional installable web-based application, but when broken down into value vs. community, we’re not so far out.

Anyway, I don’t think I’ve stumbled onto anything earth-shaking in this little exercise, of course, but I do think it would be interesting to break down all YC companies (and perhaps even everything in the TechCrunch Crunchbase) into this categorization and see how things shake out with regard to who is still around and growing.  Maybe later…right now I’ve gotta go build some value.

Extending Virtualmin with plugins

Plugins can be big

Not many people know that Virtualmin’s already extensive list of built-in features can be extended by writing plugins, which are basically Webmin modules that export a special API. Why would you want to do this, you may ask? Let’s say their is a mailing list application, log analyzer, database or source code control system that you want to make available on a per-domain basis .. if so, a plugin is the way to do it.

A plugin is typically used to a new feature to Virtualmin. In it’s parlance, a feature is something that is enabled on a per-domain basis, such as a website, DNS domain or MySQL database. Let’s say you have discovered an awesome new log analysis program that you want run on each domain’s log files – a plugin would be the way to implement it.

A plugin can also add options to mailbox users. The most common use of this is to grant access on a per-user basis to some resource, such as statistics, an application or database. Plugins can also create new database types, add links to the left menu in the Virtualmin framed theme, and add sections to it’s system information page.

Some of the existing plugins give you an idea of what’s possible :

  1. The DAV plugin adds a feature which makes a virtual server’s web pages editable from applications that support the protocol, such as Windows and OSX. It also lets you enable DAV logins for each mailbox in the domain.
  2. The Bootup Actions plugin allows domain owners to have their long-running server processes started when the system boots.
  3. The Mail Relay plugin lets you forward email for a domain to another server, which can be configured by the domain owner.
  4. The Admin Notes feature adds a new section to the right-hand frame for entering comments about the system, for sharing status between master admins.

To see a full list of plugins that exist, check out the third-party modules database.

If you know Perl, have written a regular Webmin module, and want to write your own plugin, check out the extensive documentation on the API.

Sharing JavaScript Code in Webmin

I posted a while back on my personal blog about some UI enhancement work that I’ve been doing in Webmin using the ExtJS JavaScript toolkit. Several folks had questions about whether Webmin was getting a new “official” JavaScript toolkit (it has some ancient and ugly API calls to generate a few JavaScript helpers for things like field graying and validation and such, but they aint got that AJAX religion), and, if not, how one could add a JavaScript library to Webmin to cleanly share it across modules and themes.

So, the answer to the first question is that Webmin is not getting an “official” JavaScript toolkit at this time. Webmin has as one of its core goals that it can be used by anyone anywhere with any browser. AJAX and heavy JavaScript usage makes that goal far more complicated. For example, we consider it a serious bug if a blind user using a screen reader can’t use Webmin. That said, we also recognize that AJAX is the best way to handle huge classes of user interaction problems, and with our commercial offering we have a strong interest in having the best looking, and most pleasant to use, UI in the field. So, I’ve begun to build a “semi-official” Webmin module that contains ExtJS and some helper functions and classes. The first example usage of this will be our new TheJAX Virtualmin theme, and soon after a few new modules.

For the second question, I’d just like to show how I’ve created this new ExtJS module for Webmin, and how one can use it. It only takes a few minutes to wrap something up into a module, and since most AJAX frameworks are making use of good JavaScript design practices and using their own namespaces, you can actually mix and match without too much pain.

Hidden Modules

So, Webmin has a very powerful module system, that allows you to package code for easy distribution and installation. A Webmin module is simply a directory with some files in it. Only one file is mandatory to make the directory into a “module”: module.info

So, we create a directory named extjs within the Webmin directory (/usr/libexec/webmin on my system), and make a file called module.info with the following contents:

name=ExtJS
desc=ExtJS AJAX Toolkit
depends=1.360
version=0.1
hidden=1

Here I’ve given it a name, and a short description, noted the version of Webmin it depends on, given it a version (I’m going to stat it at 0.1, though the contained ExtJS version is 2.0b), and set it to be hidden. The hidden option means that users won’t be able to see this module in the UI, but other modules can make calls to it. Later, if I decide to add configurable options to this library that I do want users to be able to see, I can make it visible and add an icon and a UI.

Now, I can start dropping in my files. I merely unzipped the ExtJS bundle, deleted the extraneous files, and dropped it into an ext directory within the module directory. That’s just to make it easy to update ExtJS components separately from the helper functions that I write in Perl in the top-level directory.

Helper Functions

So, the simplest thing to automate away is the inclusion of the script tags that load the library. So, I’ll create a header_text function in a file called extjs-lib.pl (Webmin has a convention of calling function libraries modulename-lib.pl), which looks like this:

 # extjs-lib.pl
 
do '../web-lib.pl';
&amp;init_config();
 
my $debug=''; # Set to '-debug' to use non-stripped library
 
# header_text()
# Text to load JavaScript and CSS for use of extjs
sub header_text {
  return <EOF;
<script src="/extjs/ext/adapter/ext/ext-base.js" type="text/javascript"></script>
<script src="/extjs/ext/ext-all.js" type="text/javascript"></script>
<link href="/extjs/ext/resources/css/ext-all.css" rel="stylesheet" type="text/css" />
<link href="/extjs/ext/resources/css/xtheme-$config%7B" rel="stylesheet" type="text/css" />
EOF
}
 
1;

Here we pull in the Webmin core library, pull in the configuration for this module (which I’ll cover in a couple of days when I’ve completed the configuration code for this module), and build the function to return the bits of text we need to properly load ExtJS and its stylesheets.

Using It

Believe it or not, we’ve now got a library that can be used by other Webmin modules or by themes. Webmin has a foreign_require function that will pull libraries like this in under their own namespace. So, when I need to use ExtJS, I can do this:

foreign_require("extjs", "extjs-lib.pl");
print extjs::header_text();

All done! In a few days I’ll be finished with the first full-featured version of this library, and will wrap it up for distribution, along with some proof-of-concept modules that show how to use a full-featured AJAX interface without breaking text-mode browsers and readers, among other things.

The new face of Webmin

After much deliberation (a bloodbath!), Jamie, Kevin, and I (and a little help from some friends) have chosen a new logo for Webmin, from the seemingly infinite great options offered up by our recent logo contest at SitePoint. Without further ado, I present to you the new face of Webmin:

The figure represents the letters W and M, but there is also the additional symbolism of three Ms to represent Webmin, Usermin, and Virtualmin. We agonized quite a bit over leaving behind the spider web themed branding of the old logo, but in the end, decided that a web was simply meaningless in a modern context where everything is web-based. When Webmin began, almost nothing for system administration ran on the web, and it was the defining characteristic of Webmin. Jamie is still justifiably proud of being so far ahead of the curve on adopting the web-based paradigm, but felt it was time to move on. The future of Webmin is virtualization and grid computing, mobile devices, public APIs, clustering, and more. The web-based UI is merely one aspect of a large swath of interesting facets.

We’d like to thank all of the designers who took part in the contest. They were patient, enthusiastic, and really good. We came into the final round of judging with a dozen or more entries that at least two of us loved, and only after two days of debate and seeking advice from friends and family, did we finally come to a concensus on one that we could all love. We’re extremely excited about the new logo, and plan to roll it out to the website, and the default Webmin theme in the next few days. We’ll also be printing some T-shirts, as soon as I find someone good here in the Bay Area to make them for us.

Thanks also to Kevin Hale, of Particle Tree and Wufoo fame for being our celebrity judge and providing adult supervision during the contest. His magical designer-y ways kept us thoroughly on the right path. Jamie’s sister Lara Cameron
also loaned us her eyes and expertise.

See also

Webmin Logo Contest

Getting a great logo: Reducing the field

It’s just not a contest until you see a goatse

As I’ve mentioned here and here, we’re holding a logo contest for Webmin’s tenth anniversary. We’ve gotten a ton of fantastic entries, and we’re coming down to the final hours of the contest. We are feeling really good about quite a few of the entries, but today the entries finally achieved what all great contests should aspire to: an unintentional goatse troll.

The finest goatse logo troll of all time, of course, appeared during the BBC Olympic “Lisa Simpson doing something naughty” logo coverage (pay attention at about :29 on the clock). But, now we’ve got one of our very own:

I’m so proud.

See also: Article at the Register