Homebrew Package Installation for Servers

Homebrew logo

One of our customers came to us a couple of weeks ago wanting to run Virtualmin on Mac OS X. I, foolishly, said, “Sure! I can help with that. I installed Virtualmin on OS X several years ago, how much different can it be? It shouldn’t take more than a couple of hours.” It turns out, it can be remarkably different. In the intervening years, Mac OS X has evolved in many interesting directions, mostly positive, some questionable. Mac OS X remains exceedingly weak for server usage, for reasons well out of the scope of this short article. But, it is quite strong for desktop/laptop use, and many people want to be able to develop their web applications on Mac OS X even if they will be deployed to Linux servers.

Enter Homebrew

The last time I setup Virtualmin on Mac OS X systems, the best package management tool, and the best way to install a lot of Open Source software, was Fink (Mac Ports, then Darwin Ports, was still quite new at the time).

Fink is an apt repository of dpkg packages built for Mac OS X. I love apt and dpkg (almost as much as yum and rpm), and Webmin/Virtualmin have great support for apt and dpkg, and so I was all set to choose Fink for this new deployment. But, there are some issues with Fink. First, it installs everything with its own packages, including stuff that is already available from the base system. For Virtualmin, whenever possible, I like to use the system standard packages for the services it manages. Homebrew is designed to work with what Apple provides when possible, which is somewhat more aligned with the Virtualmin philosophy.

Second, and perhaps more important, Fink is seemingly much less popular and much less actively maintained than Homebrew. I’m not sure why. Possibly because the Homebrew website looks good, and the documentation is very well-written, while the Fink home page is a little drab and looks complicated. And, Fink package versions tend to be quite a bit older than the packages provided by the very active Homebrew project. This can be a much more serious issue. Security updates are absolutely vital in a web server, and a package repository that is actively maintained is the best way to insure you’ll have security updates.

So, I’ve spent the past couple of days experimenting with Homebrew. It’s a pretty nice system, and its community and developers are active, responsive, and helpful. All great things. But, its primary advertised feature is also its biggest weakness and most dangerous mistake.

Installation Without sudo

Or: Homebrew Considered Harmful

One of the major advertised features of Homebrew is that you can install it, and any package, without root or sudo privileges. There are good reasons one might want this, but on a server, it has alarming side effects, and it is one of the first things I would need to correct for our use case (of installing a virtual hosting stack and Virtualmin). The example I’ll use here is that of MySQL. When you install the mysql package from Homebrew, it will be installed with the ownership of all files set to the user that installed the package. And, more dangerously, it will be setup to run as the user that installed it.

This decision was made because Homebrew often builds the software at install time, rather than providing a binary package (there is a new “bottles” feature installs binary packages, but that wasn’t intended to address the sudo problem). The risk of building software with sudo or root privileges is very real, and in this case it results in the choice to build the software as a non-root user.

Other package managers, like dpkg and rpm, resolve this problem with toolchains designed around building the packages within a chroot so that unwanted behavior is contained. For example, mock on Fedora and CentOS provides an easy to use tool for building packages across many distributions and versions inside of a chroot environment with only the dependencies specified by the package. The most popular Linux distributions distribute binary packages that were built in a controlled environment. But, Homebrew generally builds the software at install time, with no chroot to protect the system from broken or hostile build processes. And, so they insist you run it as a non-root user. This is, I suppose, a logical conclusion to come to, based on the premise of a package manager that builds software on the user’s system without being confined in a container or chroot, but it has negative consequences.

For example, when I install MySQL from Homebrew, everything is owned by joe:staff. The provided property list file for starting the server is also designed to start it as that user, when the user logs in. For a development system, this may not be a big deal, and even makes a certain sort of sense (I prefer my development environment to more closely mirror my deployment environment, but I can see reasonable arguments for the way they do it). But, for a server, it is simply untenable.

The most important reason it is a bad decision is that it leads to many, possibly all, of the services running with the privileges of the user that installed them. Which, in most cases, is probably a powerful user (mine is an administrative user with sudo privileges, for example). So, in the event any of the services are compromised, all of the services will be compromised, and likely so will the user account in question. The security implications of this really cannot be overstated. This is a huge problem.

This is why Linux and UNIX systems (and even Apple, who aren’t historically renowned for their strong multi-user security practices) run all services as different users, and with restricted privileges. On the average LAMP system, there will be an apache or www user that runs Apache, a mysql user that runs MySQL, a nobody user that runs Postfix, and web applications will usually be run as yet another user still. These special users often have very restricted accounts, and may not even have a shell associated with them, further limiting the damage that can occur in the event of an exploit of any one service. Likewise, they may be further restricted by SELinux or other RBACL-based security. Any one of these services or applications being compromised through any means won’t generally compromise other services or users. Homebrew throws that huge security benefit away to avoid having to sudo during installation.

It’s probably too late to convince the Homebrew folks to backtrack on that decision. But, it’s not terribly difficult to fix for one-off installations, and many do consider it a valuable feature of Homebrew. Fixing the installed services as I’ve done has some side effects that may also be dangerous, which I’ll go into at the end of the article, but since I figured out how to do it, I thought I’d document it. During my research I found that an alarming number of users are using Homebrew in server environments and I found a number of users asking similar questions about various services, so, maybe this will help some folks avoid a dangerous situation.

So, let’s get started. After installation of MySQL (using the command brew install mysql), here’s the changes you’ll want to make.

Update The Property Lists File

The way Homebrew recommends running MySQL after installation is to link the provided plist file in /usr/local/opt/mysql/homebrew.mxcl.mysql.plist into your ~/Library/LaunchAgents directory, and add it using the launchctl load command. This sets it up to run at all times when your user is logged in, which is great if you’re developing and only need it running when you’re logged in and working. But, we want it to run during system boot without having any users logged in, and even more importantly we want it to run as the _mysql user.

So, instead of linking it into your local LaunchAgents directory, as the documentation suggests, copy it into your system /Library/LaunchDaemons directory.

$ sudo cp /usr/local/opt/mysql/homebrew.mxcl.mysql.plist /Library/LaunchDaemons

Then edit the file to add user and group information (you’ll have to use sudo), add a –user option, and change the command to mysqld and WorkingDirectory to /usr/local/var/mysql. Mine looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>UserName</key>
  <string>_mysql</string>
  <key>GroupName</key>
  <string>_mysql</string>
  <key>KeepAlive</key>
  <true/>
  <key>Label</key>
  <string>homebrew.mxcl.mysql</string>
  <key>ProgramArguments</key>
  <array>
    <string>/usr/local/opt/mysql/bin/mysqld</string>
    <string>--bind-address=127.0.0.1</string>
    <string>--datadir=/usr/local/var/mysql</string>
    <string>--user=_mysql</string>
  </array>
  <key>RunAtLoad</key>
  <true/>
  <key>WorkingDirectory</key>
  <string>/usr/local/var/mysql</string>
</dict>
</plist>

Notice the addition of the UserName and GroupName keys, both set to _mysql, as well as several other altered lines.

Note: I am not a Mac OS X or launchd expert. There are a number of aspects of the Mac OS X privilege model that I do not understand. I would welcome comments about how the security of this configuration might be improved. Also, the entirety of my experience with launchd is the several hours I spent playing with it and reading about it to convince MySQL to run as a different user. But, I’m pretty much certain that the way Homebrew does servers is worse than what I’ve done here.

Change Ownership of MySQL Installation and Databases

The _mysql user does not have permissions to read things owned by the joe user (or the user you used to install MySQL with Homebrew), so you’ll need to change ownership of all MySQL data files to _mysql:wheel.

$ sudo chown -R _mysql:wheel /usr/local/var/mysql/

Change Ownership of the Property List File

Property list files in the Library/LaunchDaemons directory (or the LaunchAgents directory) must belong to root, for security reasons. So, you’ll need to update that, as well.

$ sudo chown root:wheel /Library/LaunchDaemons/homebrew.mxcl.mysql.plist

Load and Start the MySQL Daemon

The launchctl command manages agents and daemons, and we can add our new service by loading the property list:

$ sudo launchctl load /Library/LaunchDaemons/homebrew.mxcl.mysql.plist

And, then we can start MySQL:

$ sudo launchctl start homebrew.mxcl.mysql.plist

If anything goes wrong, check the system.log:

$ sudo tail -f /var/log/system.log
I found the documentation at launchd.info particularly helpful when working out how to use launchd.

Future Concerns

Since this corrects the big security issue with the Homebrew installation of MySQL, and this technique could reasonably easily be applied across every Homebrew-installed service, why aren’t we happy?

Updates are part of security, too.

The most important reason to use a package manager is not to make it easy to install software, contrary to popular belief, and if all the package manager does is provide easy to install packages, it is not an effective package manager. The most important reason to use a package manager is to make it easy to update software.

Homebrew makes installation and upgrades reasonably easy, but the steps I’ve taken in this article to make MySQL run as its own user seems likely to break updates, since some files created during installation have changed ownership. A newer version of MySQL isn’t available in the Homebrew repository, so I can’t test whether it does break upgrades or not. Nonetheless, fixing this issue compounded across many services (the Virtualmin installation process normally installs: Apache, MySQL/MariaDB, PostgreSQL, Postfix, Dovecot, BIND, Procmail, SpamAssassin, ClamAV, Mailman, and a bunch more) will likely prove to be a maintenance challenge that is probably not worth the effort.

So, despite having figured out how to make this work, I’m now going to spend the same amount of time and effort giving Mac Ports a thorough test drive. I have a pretty strong suspicion it will be a better fit for server usage. This way of working feels like fighting against the way Homebrew wants to operate, and when you find yourself having to work so hard against the tool, it’s probably the wrong one for the job.

And What Does All This Mean for Virtualmin on Mac OS X?

Well, I’m a sucker for the sunk cost fallacy, so I’m planning to spend another couple of days working out a basic install script for Virtualmin on Mac OS X, probably using Mac Ports (I can see a way forward using Homebrew, but I don’t like the terrain). I’ll likely never recommend Mac OS X for a production server deployment, but it’s certainly not the worst OS out there for the purpose.

Virtualmin Memory Usage (and Other Tales of Wonder and Woe!)

I’ve noticed over the years that one of the most common sources of confusion for new Virtualmin users, or users who are new to Linux and web hosting in general, is memory usage. I’ve written up documentation about Virtualmin on Low Memory Systems in the past, but it focuses mostly on helping folks with low memory systems reduce memory usage of their Virtualmin (and all of its related packages, like Apache, PHP, MySQL, and Postfix) installation. It goes into interesting detail about Webmin memory usage, library caching in Virtualmin, etc. but doesn’t go into things like the memory usage of various services in a Virtualmin (or any LAMP stack) system. This article will briefly address each of these subjects and provide real world numbers for how much memory one should expect a Virtualmin installation to require.

A side story in all of this is how Virtualmin compares to other web hosting control panels. Somehow, this is considered interesting data for some folks, though I can’t really fathom why, given the huge differences in functionality available, particularly when comparing control panels with extremely limited web-focused functionality with full-featured control panels (like Virtualmin, cPanel, or Plesk) that provide mail processing with anti-virus and spam filtering, database management, etc.  But, it comes up a lot. So, let’s get some hard numbers for Virtualmin and talk about where those numbers come from. If anyone happens to have data about memory usage of other control panels, feel free to post them in the comments (though, I doubt any control panel will use vastly more or less memory than Virtualmin, unless it’s written in Java, or something similar).

Where does the memory go‽

The first thing I want to do is break down memory usage in a production Virtualmin system, and talk about which components require large amounts of memory, and which ones can be reduced through options or disabling features.

Virtualmin system top

top sorted by memory usage on a very busy 8GB server

The above image is the output of the top command on a Virtualmin system that has several active websites, including a large Drupal deployment (the one for Virtualmin.com which has ~30,000 registered users, ~100,000 posts and comments, and receives about 100,000 visitors a month, at time of writing) and all of our software download repositories. As you can see the system has 8GB of RAM and 2GB of swap memory. Here’s what we see is using the majority of memory on this system, in order of size:

  • mysqld – This is the MySQL server process. It is configured with quite large buffers and cache settings, in order to maximize performance for our Drupal instance and other applications that access the database, such as the Virtualmin Technical Support module (which can create tickets in our issue tracker). This is the largest single process on the system, which is likely to be true on most systems with large database-backed websites. It has 2.3GB of memory allocated, though all but 418MB is not necessarily dedicated to this process or in physical RAM. See the note below about virtual vs. resident size.
  • clamd – This one always surprises people, and folks often forget about it when calculating their expected memory usage. ClamAV is very demanding, because it loads a large database of virus signatures into memory. Virtualmin allows it to be configured as either a daemon or a standalone executable…but the standalone version is extremely slow to start, and causes a spike of both CPU and disk activity when starting. So, if you plan to process mail (on any system, regardless of whether Virtualmin is involved), you should expect to give up a few hundred megabytes to the spam and AV filtering stack. The ClamAV server has 305MB resident size.
  • php-cgi – There are several of these, and they represent the pool of mod_fcgid spawned PHP processes that are serving the Drupal website. They are owned by user “virtualmin”, because we use suexec on this system, and the site in question is virtualmin.com, and the username for that account is virtualmin. The PHP process is quite large here, larger than most, for a few reasons. Primarily, it is because we make use of a large number of Drupal modules, and some of those modules are quite demanding, so we’ve had to increase PHP memory limits for this user. These processes have ~135MB resident size, and much larger virtual size, but all of the virtual memory usage is shared across every php-cgi process for every user.
  • lookup-domain-daemon.pl – This is part of the mail processing stack, and is a server provided by Virtualmin. It allows SpamAssassin and ClamAV to have user-level and domain-level configuration settings, and allows some types of configuration for these services to be modified by end users safely. This process is 55MB with another ~40MB shared with other processes.
  • spamd – The SpamAssassin server. See, I told you mail processing was heavy! At ~50MB for each of the SpamAssassin child processes, this adds up on a heavily loaded system.
  • perl – Finally, this is actually the Webmin/Virtualmin process! My system currently has library caching fully enabled, and the total virtual process size is ~135MB (this would be smaller on a 32 bit system), and a resident size of 46M. If I were on a low-memory system, I would disable pre-caching, and Virtualmin would shrink to about 15MB (less on a 32 bit system). This can be set in Virtualmin->System Settings->Preload Virtualmin libraries at startup? The options are “All Libraries”, “Only Core”, and “No”, which will cause the Webmin process to be 40-45MB resident, 20-25MB resident, or 12-17MB resident,depending on whether the system is 32 or 64 bit.
  • named – This is the BIND DNS server. It’s memory usage is quite modest compared to a lot of the other services on this system, and is probably never something one would worry about tuning, unless you serve a very high volume of DNS requests.One thing to bear in mind, however, is that if you have enabled the caching nameserver features of BIND, and many users are using it for DNS service, the process size could grow quite large. We recommend only enabling recursive lookups for the Virtualmin server itself (or, possibly even better, forwarding those recursive lookups to another server).
  • httpd – This is the pool of Apache web server processes. Notice the virtual size is quite large, while the resident size is quite small. Much of the memory usage of these processes is shared across all of them (of which there are probably 100+ on my system at any given time, due to number of concurrent users). The size of these processes is determined mostly by the number of modules you have installed. But, even on this system, with a number of modules enabled and actively used, the resident size is only 9MB per process. Given my 3.4 GB of currently free memory, Apache could spawn over 300 additional processes (beyond the 100 or more already running) without bumping into the memory limitations of this system. Apache often gets accused of being a memory hog compared to other web servers, but that’s often an unfair comparison between an Apache with a bunch of large modules (like mod_php, or mod_perl, neither of which are needed for most Virtualmin systems) and a stripped down lightweight server, like nginx that simply doesn’t have any large modules that can be enabled.

Note: VIRT and RES are indicators of the type of memory that has been allocated; VIRT includes the resident memory, as well as memory-mapped files on disk, shared libraries which share RAM with other processes, etc., while RES is the resident memory usage, which roughly reflects how mush RAM is dedicated to this process.

There are many other processes on this system, including the rest of the httpd processes, but these few processes already explain where the vast majority of memory on the system is going, and so we won’t dig any deeper into it for this story.

Just for fun, let’s see a somewhat smaller system’s memory usage:

small-virtualmin-top

Memory-sorted top output on a moderately loaded 4GB Virtualmin server

This is a ~4GB virtual machine, and I’ve temporarily disabled library pre-caching in Virtualmin, which makes the process size about 17MB (it is a 64 bit system). Since it’s so small, it doesn’t even show up in the list, when sorted by memory usage. In this case, the large processes start with MySQL, once again configured with somewhat large buffers and caches. Java shows up here, which is uncommon for me, since Java is such a best to work with, but I have a Jenkins CI instance running on this box. And, then the mail filtering stack is next, and slightly smaller than on the above system. I don’t have ClamAV running on this box, since the only email it processes is received by people running Linux and we don’t worry so much about viruses in email. And, then comes php-cgi, which is much smaller on this system, since it only runs moderately small WordPress instances, and a pretty hard-working MediaWiki installation for doxfer.webmin.com.

It’s also possible to run Virtuamin in a very small amount of memory, particularly if you don’t need to process mail on the system. We recommend at least 256MB for a 32 bit system, and 384MB for a 64 bit system, even if you won’t be running a mail stack. While Virtualmin itself doesn’t need more memory, the performance of most web applications would be pretty abysmal on anything less. MySQL performance is directly correlated with the amount of memory you can devote to it. Using nginx (which is also supported by Virtualmin) may help in reducing the needed memory usage, though a minimal Apache configuration won’t be much larger.

tl;dr

Virtualmin uses somewhere between 12MB and 46MB resident memory, and up to ~150MB virtual, depending on whether library caching is enabled and whether it is a 32 or 64 bit system.

If you’re processing mail with spam and antivirus, Virtualmin will, by default, also run a 45-55MB daemon for assisting with processing mail.

All of this is dwarfed by the actual services being managed by Virtualmin, like Apache, MySQL, ClamAV, SpamAssassin, Postfix,etc.

If you need to run Virtualmin on a very small-memory system, the best thing you can do is off-load email to some other system or service, since the full mail processing stack with SpamAssassin, ClamAV, Postfix, and Dovecot can easily add up to a few hundred MB.

Interesting Links

My favorite site to refer people to when they’re wondering about what memory usage information means on a Linux system is Linux Ate My RAM!