Virtualmin Memory Usage (and Other Tales of Wonder and Woe!)

I’ve noticed over the years that one of the most common sources of confusion for new Virtualmin users, or users who are new to Linux and web hosting in general, is memory usage. I’ve written up documentation about Virtualmin on Low Memory Systems in the past, but it focuses mostly on helping folks with low memory systems reduce memory usage of their Virtualmin (and all of its related packages, like Apache, PHP, MySQL, and Postfix) installation. It goes into interesting detail about Webmin memory usage, library caching in Virtualmin, etc. but doesn’t go into things like the memory usage of various services in a Virtualmin (or any LAMP stack) system. This article will briefly address each of these subjects and provide real world numbers for how much memory one should expect a Virtualmin installation to require.

A side story in all of this is how Virtualmin compares to other web hosting control panels. Somehow, this is considered interesting data for some folks, though I can’t really fathom why, given the huge differences in functionality available, particularly when comparing control panels with extremely limited web-focused functionality with full-featured control panels (like Virtualmin, cPanel, or Plesk) that provide mail processing with anti-virus and spam filtering, database management, etc.  But, it comes up a lot. So, let’s get some hard numbers for Virtualmin and talk about where those numbers come from. If anyone happens to have data about memory usage of other control panels, feel free to post them in the comments (though, I doubt any control panel will use vastly more or less memory than Virtualmin, unless it’s written in Java, or something similar).

Where does the memory go‽

The first thing I want to do is break down memory usage in a production Virtualmin system, and talk about which components require large amounts of memory, and which ones can be reduced through options or disabling features.

Virtualmin system top

top sorted by memory usage on a very busy 8GB server

The above image is the output of the top command on a Virtualmin system that has several active websites, including a large Drupal deployment (the one for Virtualmin.com which has ~30,000 registered users, ~100,000 posts and comments, and receives about 100,000 visitors a month, at time of writing) and all of our software download repositories. As you can see the system has 8GB of RAM and 2GB of swap memory. Here’s what we see is using the majority of memory on this system, in order of size:

  • mysqld – This is the MySQL server process. It is configured with quite large buffers and cache settings, in order to maximize performance for our Drupal instance and other applications that access the database, such as the Virtualmin Technical Support module (which can create tickets in our issue tracker). This is the largest single process on the system, which is likely to be true on most systems with large database-backed websites. It has 2.3GB of memory allocated, though all but 418MB is not necessarily dedicated to this process or in physical RAM. See the note below about virtual vs. resident size.
  • clamd – This one always surprises people, and folks often forget about it when calculating their expected memory usage. ClamAV is very demanding, because it loads a large database of virus signatures into memory. Virtualmin allows it to be configured as either a daemon or a standalone executable…but the standalone version is extremely slow to start, and causes a spike of both CPU and disk activity when starting. So, if you plan to process mail (on any system, regardless of whether Virtualmin is involved), you should expect to give up a few hundred megabytes to the spam and AV filtering stack. The ClamAV server has 305MB resident size.
  • php-cgi – There are several of these, and they represent the pool of mod_fcgid spawned PHP processes that are serving the Drupal website. They are owned by user “virtualmin”, because we use suexec on this system, and the site in question is virtualmin.com, and the username for that account is virtualmin. The PHP process is quite large here, larger than most, for a few reasons. Primarily, it is because we make use of a large number of Drupal modules, and some of those modules are quite demanding, so we’ve had to increase PHP memory limits for this user. These processes have ~135MB resident size, and much larger virtual size, but all of the virtual memory usage is shared across every php-cgi process for every user.
  • lookup-domain-daemon.pl – This is part of the mail processing stack, and is a server provided by Virtualmin. It allows SpamAssassin and ClamAV to have user-level and domain-level configuration settings, and allows some types of configuration for these services to be modified by end users safely. This process is 55MB with another ~40MB shared with other processes.
  • spamd – The SpamAssassin server. See, I told you mail processing was heavy! At ~50MB for each of the SpamAssassin child processes, this adds up on a heavily loaded system.
  • perl – Finally, this is actually the Webmin/Virtualmin process! My system currently has library caching fully enabled, and the total virtual process size is ~135MB (this would be smaller on a 32 bit system), and a resident size of 46M. If I were on a low-memory system, I would disable pre-caching, and Virtualmin would shrink to about 15MB (less on a 32 bit system). This can be set in Virtualmin->System Settings->Preload Virtualmin libraries at startup? The options are “All Libraries”, “Only Core”, and “No”, which will cause the Webmin process to be 40-45MB resident, 20-25MB resident, or 12-17MB resident,depending on whether the system is 32 or 64 bit.
  • named – This is the BIND DNS server. It’s memory usage is quite modest compared to a lot of the other services on this system, and is probably never something one would worry about tuning, unless you serve a very high volume of DNS requests.One thing to bear in mind, however, is that if you have enabled the caching nameserver features of BIND, and many users are using it for DNS service, the process size could grow quite large. We recommend only enabling recursive lookups for the Virtualmin server itself (or, possibly even better, forwarding those recursive lookups to another server).
  • httpd – This is the pool of Apache web server processes. Notice the virtual size is quite large, while the resident size is quite small. Much of the memory usage of these processes is shared across all of them (of which there are probably 100+ on my system at any given time, due to number of concurrent users). The size of these processes is determined mostly by the number of modules you have installed. But, even on this system, with a number of modules enabled and actively used, the resident size is only 9MB per process. Given my 3.4 GB of currently free memory, Apache could spawn over 300 additional processes (beyond the 100 or more already running) without bumping into the memory limitations of this system. Apache often gets accused of being a memory hog compared to other web servers, but that’s often an unfair comparison between an Apache with a bunch of large modules (like mod_php, or mod_perl, neither of which are needed for most Virtualmin systems) and a stripped down lightweight server, like nginx that simply doesn’t have any large modules that can be enabled.

Note: VIRT and RES are indicators of the type of memory that has been allocated; VIRT includes the resident memory, as well as memory-mapped files on disk, shared libraries which share RAM with other processes, etc., while RES is the resident memory usage, which roughly reflects how mush RAM is dedicated to this process.

There are many other processes on this system, including the rest of the httpd processes, but these few processes already explain where the vast majority of memory on the system is going, and so we won’t dig any deeper into it for this story.

Just for fun, let’s see a somewhat smaller system’s memory usage:

small-virtualmin-top

Memory-sorted top output on a moderately loaded 4GB Virtualmin server

This is a ~4GB virtual machine, and I’ve temporarily disabled library pre-caching in Virtualmin, which makes the process size about 17MB (it is a 64 bit system). Since it’s so small, it doesn’t even show up in the list, when sorted by memory usage. In this case, the large processes start with MySQL, once again configured with somewhat large buffers and caches. Java shows up here, which is uncommon for me, since Java is such a best to work with, but I have a Jenkins CI instance running on this box. And, then the mail filtering stack is next, and slightly smaller than on the above system. I don’t have ClamAV running on this box, since the only email it processes is received by people running Linux and we don’t worry so much about viruses in email. And, then comes php-cgi, which is much smaller on this system, since it only runs moderately small WordPress instances, and a pretty hard-working MediaWiki installation for doxfer.webmin.com.

It’s also possible to run Virtuamin in a very small amount of memory, particularly if you don’t need to process mail on the system. We recommend at least 256MB for a 32 bit system, and 384MB for a 64 bit system, even if you won’t be running a mail stack. While Virtualmin itself doesn’t need more memory, the performance of most web applications would be pretty abysmal on anything less. MySQL performance is directly correlated with the amount of memory you can devote to it. Using nginx (which is also supported by Virtualmin) may help in reducing the needed memory usage, though a minimal Apache configuration won’t be much larger.

tl;dr

Virtualmin uses somewhere between 12MB and 46MB resident memory, and up to ~150MB virtual, depending on whether library caching is enabled and whether it is a 32 or 64 bit system.

If you’re processing mail with spam and antivirus, Virtualmin will, by default, also run a 45-55MB daemon for assisting with processing mail.

All of this is dwarfed by the actual services being managed by Virtualmin, like Apache, MySQL, ClamAV, SpamAssassin, Postfix,etc.

If you need to run Virtualmin on a very small-memory system, the best thing you can do is off-load email to some other system or service, since the full mail processing stack with SpamAssassin, ClamAV, Postfix, and Dovecot can easily add up to a few hundred MB.

Interesting Links

My favorite site to refer people to when they’re wondering about what memory usage information means on a Linux system is Linux Ate My RAM!

Extending Virtualmin with plugins

Plugins can be big

Not many people know that Virtualmin’s already extensive list of built-in features can be extended by writing plugins, which are basically Webmin modules that export a special API. Why would you want to do this, you may ask? Let’s say their is a mailing list application, log analyzer, database or source code control system that you want to make available on a per-domain basis .. if so, a plugin is the way to do it.

A plugin is typically used to a new feature to Virtualmin. In it’s parlance, a feature is something that is enabled on a per-domain basis, such as a website, DNS domain or MySQL database. Let’s say you have discovered an awesome new log analysis program that you want run on each domain’s log files – a plugin would be the way to implement it.

A plugin can also add options to mailbox users. The most common use of this is to grant access on a per-user basis to some resource, such as statistics, an application or database. Plugins can also create new database types, add links to the left menu in the Virtualmin framed theme, and add sections to it’s system information page.

Some of the existing plugins give you an idea of what’s possible :

  1. The DAV plugin adds a feature which makes a virtual server’s web pages editable from applications that support the protocol, such as Windows and OSX. It also lets you enable DAV logins for each mailbox in the domain.
  2. The Bootup Actions plugin allows domain owners to have their long-running server processes started when the system boots.
  3. The Mail Relay plugin lets you forward email for a domain to another server, which can be configured by the domain owner.
  4. The Admin Notes feature adds a new section to the right-hand frame for entering comments about the system, for sharing status between master admins.

To see a full list of plugins that exist, check out the third-party modules database.

If you know Perl, have written a regular Webmin module, and want to write your own plugin, check out the extensive documentation on the API.