Homebrew Package Installation for Servers

Homebrew logo

One of our customers came to us a couple of weeks ago wanting to run Virtualmin on Mac OS X. I, foolishly, said, “Sure! I can help with that. I installed Virtualmin on OS X several years ago, how much different can it be? It shouldn’t take more than a couple of hours.” It turns out, it can be remarkably different. In the intervening years, Mac OS X has evolved in many interesting directions, mostly positive, some questionable. Mac OS X remains exceedingly weak for server usage, for reasons well out of the scope of this short article. But, it is quite strong for desktop/laptop use, and many people want to be able to develop their web applications on Mac OS X even if they will be deployed to Linux servers.

Enter Homebrew

The last time I setup Virtualmin on Mac OS X systems, the best package management tool, and the best way to install a lot of Open Source software, was Fink (Mac Ports, then Darwin Ports, was still quite new at the time).

Fink is an apt repository of dpkg packages built for Mac OS X. I love apt and dpkg (almost as much as yum and rpm), and Webmin/Virtualmin have great support for apt and dpkg, and so I was all set to choose Fink for this new deployment. But, there are some issues with Fink. First, it installs everything with its own packages, including stuff that is already available from the base system. For Virtualmin, whenever possible, I like to use the system standard packages for the services it manages. Homebrew is designed to work with what Apple provides when possible, which is somewhat more aligned with the Virtualmin philosophy.

Second, and perhaps more important, Fink is seemingly much less popular and much less actively maintained than Homebrew. I’m not sure why. Possibly because the Homebrew website looks good, and the documentation is very well-written, while the Fink home page is a little drab and looks complicated. And, Fink package versions tend to be quite a bit older than the packages provided by the very active Homebrew project. This can be a much more serious issue. Security updates are absolutely vital in a web server, and a package repository that is actively maintained is the best way to insure you’ll have security updates.

So, I’ve spent the past couple of days experimenting with Homebrew. It’s a pretty nice system, and its community and developers are active, responsive, and helpful. All great things. But, its primary advertised feature is also its biggest weakness and most dangerous mistake.

Installation Without sudo

Or: Homebrew Considered Harmful

One of the major advertised features of Homebrew is that you can install it, and any package, without root or sudo privileges. There are good reasons one might want this, but on a server, it has alarming side effects, and it is one of the first things I would need to correct for our use case (of installing a virtual hosting stack and Virtualmin). The example I’ll use here is that of MySQL. When you install the mysql package from Homebrew, it will be installed with the ownership of all files set to the user that installed the package. And, more dangerously, it will be setup to run as the user that installed it.

This decision was made because Homebrew often builds the software at install time, rather than providing a binary package (there is a new “bottles” feature installs binary packages, but that wasn’t intended to address the sudo problem). The risk of building software with sudo or root privileges is very real, and in this case it results in the choice to build the software as a non-root user.

Other package managers, like dpkg and rpm, resolve this problem with toolchains designed around building the packages within a chroot so that unwanted behavior is contained. For example, mock on Fedora and CentOS provides an easy to use tool for building packages across many distributions and versions inside of a chroot environment with only the dependencies specified by the package. The most popular Linux distributions distribute binary packages that were built in a controlled environment. But, Homebrew generally builds the software at install time, with no chroot to protect the system from broken or hostile build processes. And, so they insist you run it as a non-root user. This is, I suppose, a logical conclusion to come to, based on the premise of a package manager that builds software on the user’s system without being confined in a container or chroot, but it has negative consequences.

For example, when I install MySQL from Homebrew, everything is owned by joe:staff. The provided property list file for starting the server is also designed to start it as that user, when the user logs in. For a development system, this may not be a big deal, and even makes a certain sort of sense (I prefer my development environment to more closely mirror my deployment environment, but I can see reasonable arguments for the way they do it). But, for a server, it is simply untenable.

The most important reason it is a bad decision is that it leads to many, possibly all, of the services running with the privileges of the user that installed them. Which, in most cases, is probably a powerful user (mine is an administrative user with sudo privileges, for example). So, in the event any of the services are compromised, all of the services will be compromised, and likely so will the user account in question. The security implications of this really cannot be overstated. This is a huge problem.

This is why Linux and UNIX systems (and even Apple, who aren’t historically renowned for their strong multi-user security practices) run all services as different users, and with restricted privileges. On the average LAMP system, there will be an apache or www user that runs Apache, a mysql user that runs MySQL, a nobody user that runs Postfix, and web applications will usually be run as yet another user still. These special users often have very restricted accounts, and may not even have a shell associated with them, further limiting the damage that can occur in the event of an exploit of any one service. Likewise, they may be further restricted by SELinux or other RBACL-based security. Any one of these services or applications being compromised through any means won’t generally compromise other services or users. Homebrew throws that huge security benefit away to avoid having to sudo during installation.

It’s probably too late to convince the Homebrew folks to backtrack on that decision. But, it’s not terribly difficult to fix for one-off installations, and many do consider it a valuable feature of Homebrew. Fixing the installed services as I’ve done has some side effects that may also be dangerous, which I’ll go into at the end of the article, but since I figured out how to do it, I thought I’d document it. During my research I found that an alarming number of users are using Homebrew in server environments and I found a number of users asking similar questions about various services, so, maybe this will help some folks avoid a dangerous situation.

So, let’s get started. After installation of MySQL (using the command brew install mysql), here’s the changes you’ll want to make.

Update The Property Lists File

The way Homebrew recommends running MySQL after installation is to link the provided plist file in /usr/local/opt/mysql/homebrew.mxcl.mysql.plist into your ~/Library/LaunchAgents directory, and add it using the launchctl load command. This sets it up to run at all times when your user is logged in, which is great if you’re developing and only need it running when you’re logged in and working. But, we want it to run during system boot without having any users logged in, and even more importantly we want it to run as the _mysql user.

So, instead of linking it into your local LaunchAgents directory, as the documentation suggests, copy it into your system /Library/LaunchDaemons directory.

$ sudo cp /usr/local/opt/mysql/homebrew.mxcl.mysql.plist /Library/LaunchDaemons

Then edit the file to add user and group information (you’ll have to use sudo), add a –user option, and change the command to mysqld and WorkingDirectory to /usr/local/var/mysql. Mine looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>UserName</key>
  <string>_mysql</string>
  <key>GroupName</key>
  <string>_mysql</string>
  <key>KeepAlive</key>
  <true/>
  <key>Label</key>
  <string>homebrew.mxcl.mysql</string>
  <key>ProgramArguments</key>
  <array>
    <string>/usr/local/opt/mysql/bin/mysqld</string>
    <string>--bind-address=127.0.0.1</string>
    <string>--datadir=/usr/local/var/mysql</string>
    <string>--user=_mysql</string>
  </array>
  <key>RunAtLoad</key>
  <true/>
  <key>WorkingDirectory</key>
  <string>/usr/local/var/mysql</string>
</dict>
</plist>

Notice the addition of the UserName and GroupName keys, both set to _mysql, as well as several other altered lines.

Note: I am not a Mac OS X or launchd expert. There are a number of aspects of the Mac OS X privilege model that I do not understand. I would welcome comments about how the security of this configuration might be improved. Also, the entirety of my experience with launchd is the several hours I spent playing with it and reading about it to convince MySQL to run as a different user. But, I’m pretty much certain that the way Homebrew does servers is worse than what I’ve done here.

Change Ownership of MySQL Installation and Databases

The _mysql user does not have permissions to read things owned by the joe user (or the user you used to install MySQL with Homebrew), so you’ll need to change ownership of all MySQL data files to _mysql:wheel.

$ sudo chown -R _mysql:wheel /usr/local/var/mysql/

Change Ownership of the Property List File

Property list files in the Library/LaunchDaemons directory (or the LaunchAgents directory) must belong to root, for security reasons. So, you’ll need to update that, as well.

$ sudo chown root:wheel /Library/LaunchDaemons/homebrew.mxcl.mysql.plist

Load and Start the MySQL Daemon

The launchctl command manages agents and daemons, and we can add our new service by loading the property list:

$ sudo launchctl load /Library/LaunchDaemons/homebrew.mxcl.mysql.plist

And, then we can start MySQL:

$ sudo launchctl start homebrew.mxcl.mysql.plist

If anything goes wrong, check the system.log:

$ sudo tail -f /var/log/system.log
I found the documentation at launchd.info particularly helpful when working out how to use launchd.

Future Concerns

Since this corrects the big security issue with the Homebrew installation of MySQL, and this technique could reasonably easily be applied across every Homebrew-installed service, why aren’t we happy?

Updates are part of security, too.

The most important reason to use a package manager is not to make it easy to install software, contrary to popular belief, and if all the package manager does is provide easy to install packages, it is not an effective package manager. The most important reason to use a package manager is to make it easy to update software.

Homebrew makes installation and upgrades reasonably easy, but the steps I’ve taken in this article to make MySQL run as its own user seems likely to break updates, since some files created during installation have changed ownership. A newer version of MySQL isn’t available in the Homebrew repository, so I can’t test whether it does break upgrades or not. Nonetheless, fixing this issue compounded across many services (the Virtualmin installation process normally installs: Apache, MySQL/MariaDB, PostgreSQL, Postfix, Dovecot, BIND, Procmail, SpamAssassin, ClamAV, Mailman, and a bunch more) will likely prove to be a maintenance challenge that is probably not worth the effort.

So, despite having figured out how to make this work, I’m now going to spend the same amount of time and effort giving Mac Ports a thorough test drive. I have a pretty strong suspicion it will be a better fit for server usage. This way of working feels like fighting against the way Homebrew wants to operate, and when you find yourself having to work so hard against the tool, it’s probably the wrong one for the job.

And What Does All This Mean for Virtualmin on Mac OS X?

Well, I’m a sucker for the sunk cost fallacy, so I’m planning to spend another couple of days working out a basic install script for Virtualmin on Mac OS X, probably using Mac Ports (I can see a way forward using Homebrew, but I don’t like the terrain). I’ll likely never recommend Mac OS X for a production server deployment, but it’s certainly not the worst OS out there for the purpose.

6 Responses

  1. Jay Vaughan April 13, 2015 / 3:50 am

    What you’re supposed to do: run homebrew with a non admin-privileged user account, and if you need process-per-user isolation well, add the user, run homebrew from there. It takes very little effort to add MySQL-User, then homebrew the install, and isolate things that way – I don’t get why you didn’t do this in the first place, though. I guess the notion that homebrew users == Administrative (owner) users is not something I would apply to anything than my personal (development) macbook, but for real serious server development, its logical that you’d use a dscl script to do proper isolation on OSX ..

    • Joe Cooper April 16, 2015 / 1:14 am

      Because that’s just as clumsy as what I did (actually moreso); dependencies will end up spread across many, many, many users. Now, when you want to update your system, you don’t have to run “brew update” once. You have to run it for every single service on the system, sudo-ing to each one. I appreciate the advice, but it’s just as toxic as what Homebrew does already, and is prone to user error on a scale that I’m really not comfortable with. Homebrew may have its place, but it is not on servers.

      I am, however, very interested in this part of your comment, “you’d use a dscl script to do proper isolation on OSX”. Do you have any links to documentation on this? The Apple docs for dscl are opaque to me. I spend hours trying to reason out how to use dscl to deal with passwd/shadow now that nidump is gone (this is how Webmin used to authenticate on Mac OS X, if not using LDAP or PAM). I can’t make sense of the thing; I’d love to see some examples of usage for what you’re suggesting for isolating services.

      • Jay Vaughan April 16, 2015 / 2:37 am

        Well, to be of any use to you, you have to be willing to a) accept that you want to isolate services like mysqld in such a way that giving it its own user is sensible. And, b) that you don’t just “upgrade your system” habitually on a system where process/user-isolation are important; presumably that means a ‘production’ machine, but even in a development environment I’ve been known to throw the irons at any development team-member who habitually updates a development, in-use, system. Ymmv, cowboy.

        So, if you’re still tracking, use dscl to set up your user:

        dscl / -create /Users/mysql_user UserShell /bin/false
        dscl / -passwd /Users/mysql_user $PASSWORD
        dscl / -append /Groups/admin GroupMembership mysql_user
        dscl / -append /Groups/daemon GroupMembership mysql_user
        #&etc.

        The magic words are “dscl directory editing”, e.g.

        http://www.symantec.com/connect/articles/mac-commands-directory-editor-dscl-and-custom-inventory

        • Joe Cooper April 16, 2015 / 8:47 am

          Why wouldn’t you update whenever security updates arrive for your applications?

          Apache had ~15 reported vulnerabilities in 2014, nginx had four, MySQL had ~19, OpenSSL had at least two serious vulnerabilities in 2014 which would have required updates in every service that uses SSL/TLS (which is likely a lot of services). There is no such thing as server software that never needs updating.

          Staying on top of updates is the single most important feature of a package management system…far more important than initial installation. The installation only happens once. Updates happen for the rest of the life of your server, and have serious security implications. A package manager that doesn’t make updates easy, reliable, and safe, is a package manager that fails at its most important task.

          Also, thanks for the tips on dscl!

  2. Jay Vaughan April 16, 2015 / 11:06 am

    >>There is no such thing as server software that never needs updating.

    True, but so? There are problems updating homebrew? News to me, but I’ve only been running it a couple of years. And anyway, everyone knows you use Linux for easy/reliable servers these days. The only capacity I can imagine using OSX in server mode is internal, within my own organization, and in that capacity I sure _don’t_ just always auto-upgrade whenever I get the feeling about updates. Instead, I manage and pick the things I want to upgrade – and user isolation is handy for that.

    • Joe Cooper April 16, 2015 / 11:40 pm

      “And anyway, everyone knows you use Linux for easy/reliable servers these days.”

      Which is one of my points about Homebrew. If it’s the best thing going for Mac OS X, then Mac OS X isn’t ready for the server. 😉

Comments are closed.