Doing Email Right (Part 1: SMTP)

picture of a mailbox made from an old V8 engine

Mail should be fast

Email service has a reputation, even among Linux-savvy users and web developers, as being extremely difficult to setup and maintain. The reputation may be well-earned, compared to setting up Apache or MySQL, but email doesn’t need to be something to fear. When you break it down into its component pieces, it becomes far less intimidating, or at least more comprehensible. And, though it seems from the outside that all of the pieces are cohesive, the pieces of a mail server are generally distinct and can be configured without much knowledge of the other pieces.

In this article, I’m going to explain the components of a mail server that can send and receive mail as well as deliver mail to local user mailboxes. Because of the depth of the subject I won’t go into detail about POP/IMAP service for clients like Thunderbird and Outlook, or spam and anti-virus filtering, or DNS for mail service (which is a quite complex subject in and of itself, particularly with DKIM and other email security features that use DNS), but I will cover those in additional articles soon.

MTA, MDA, MUA, IMAP, POP3, SMTP, WTF‽

The first thing to understand about “email service” is that it is not just one service. It is actually made up of several interconnected components, each of which communicates via one or more protocols to other services locally or remotely.

Message Transfer Agent (MTA)

The central component of any email system is the MTA, or Message Transfer Agent, aka Mail Transfer Agent. This is a server that communicates with clients and other servers using SMTP (Simple Mail Transfer Protocol). Some popular examples (in order of popularity and quality of current maintenance at the time of writing) include Postfix, Exim, Sendmail, and QMail. My examples in this article will mostly use Postfix, as it is likely the most popular option, has a good security record, good performance, and is easy to configure.

In its simplest form, the MTA either initiates or accepts an SMTP connection, in order to send or receive an email on behalf of a user. I won’t yet go into the details of the SMTP protocol, though it is a simple protocol when performing simple tasks. For now, it’s enough to know that an MTA like Postfix or Sendmail sends and receives mail via the SMTP protocol. I will show the entire life cycle of an email once I’ve discussed each component.

Mail Delivery Agent (MDA)

Once an email reaches its destination server, the MTA will hand it off to a Mail Delivery Agent, also called a Local Delivery Agent, for delivery to a local user mailbox. The mailbox could be in a variety of formats, Maildir and mbox being the most popular.

There are several mail delivery agents available for Linux and UNIX platforms, with procmail and maildrop being the most popular. The Dovecot and Cyrus mail suites both include a mail delivery agent, in addition to the POP3/IMAP servers they are best known for.

Mail User Agent (MUA)

The Mail User Agent, also simply called “mail client”, is the software used to read (and also send, though via two different protocols) email. Examples of this include Mozilla Thunderbird, Geary, Microsoft Outlook, and Evolution. Other, less obvious examples of mail clients include webmail clients like Roundcube, Squirrelmail, and Usermin, and classic command line clients like mutt and pine. The mail client will use the POP3 or IMAP protocol to retrieve mail from a POP/IMAP server, but will use SMTP to send mail.

There is another class of retrieval software called Mail Retrieval Agent, which includes fetchmail and getmail. This category of tool has become much less popular in recent years, due to evolving user needs, and won’t be covered in this article.

POP3/IMAP

This is another type of MTA which provides a standardized interface for MUAs to retrieve mail from the mail server. Popular examples include Dovecot, Cyrus, and Courier. I will primarily cover Dovecot in this series, as it is among the most popular, very fast and efficient, flexible, easy to configure, and actively maintained. It’s also what we use in a default installation of Virtualmin, so it is what I know best.

A Very Short and Somewhat Boring Story About A Single Email

Once upon a time, the duck wanted to send an email to the fox. So, the duck opened up her email client (Thunderbird, of course), and clicked “Write”. She typed what he needed to say to the fox, and clicked “Send”.

Mail Client to Mail Server Communication

Her mail client then opened a connection to her Postfix mail server running on a server in a data center in Sheboygan, using SMTP. Once established, the conversation between the MUA (Thunderbird) and the MTA (Postfix) went something like this (output from Postfix is indicated with a “S”, as it is operating as the receiving server in this case, while output from Thunderbird is indicated with a “C”, as it is operating as the sending client in this scenario). The server response happens immediately upon opening a connection to port 25 (or possibly other ports in the case of SSL or TLS connections):

 S: 220 smtp.fancyduckenterprises.com ESMTP Postfix
 C: HELO duckhome.wildbirdnetworks.com
 S: 250 Hello duckhome.wildbirdnetworks.com, I am glad to meet you
 C: MAIL FROM:<duck@fancyduckenterprises.com>
 S: 250 Ok
 C: RCPT TO:<fox@foxlocksdocksandclocks.com>
 S: 250 Ok
 C: DATA
 S: 354 End data with <CR><LF>.<CR><LF>
 C: From: "Duck" <duck@fancyduckenterprises.com>
 C: To: "Fox" <fox@foxlocksdocksandclocks.com>
 C: Date: Wed, 15 Apr 2015 16:02:43 -0500
 C: Subject: Fancy Clocks?
 C:
 C: Hello Fox,
 C: Do you have any fancy clocks?
 C: Your friend,
 C: Duck
 C: .
 S: 250 Ok: queued as 34A1E36F0
 C: QUIT
 S: 221 Bye

This almost looks like a casual conversation between server and client, but there is a reasonably well-defined protocol that determines the negotiation and sending of mail. In the above example, the server and the client both belong to the duck; the next section will cover the conversation between servers that delivers the mail from Duck’s server to a server belonging to Fox.

The first line is the server introducing itself, telling the client the status of the server (220, which means “service ready”), the name of the server (smtp.fancyduckenterprises.com), the protocol that the server speaks (ESMTP is an acronym of Extended Simple Mail Transfer Protocol), and the MTA software (Postfix, in this case).

The next line is simply the client acknowledging the connection by identifying itself with its name (duckhome.wildbirdnetworks.com). “HELO” is simply the protocol defined way this is done.

In the third line, the server acknowledges (250, which means “Requested mail action okay, completed”) the connection and agrees to communicate with the client.

In the fourth line, the client tells the server that it has mail it would like to send, from sender duck@fancyduckenterprises.com.

On the fifth line, the server confirms that it is willing to send mail on behalf of the sender. Postfix (and Sendmail, etc.) has a variety of ways to determine who can send mail and where they can send it from, which I’ll discuss in a later article. Likewise, for the sixth and seventh lines, the client tells the server who the mail is for, and the server agrees to accept it for sending.

The next several lines are the client saying it is ready to send mail data, the server agreeing to accept it (reply code 354, or “Start mail input; end with <CRLF>.<CRLF>”), and the mail data from the client.

Once the client is finished sending mail (indicated by a single “.” on a line by itself), the server queues the mail and gives the client a unique identification number for the mail. This mail ID number can be useful in tracking down mail server troubles. End users rarely need to concern themselves with such implementation details, but system administrators need to know how to troubleshoot mail delivery problems, and mail ID is extremely important for that purpose.

Once queued on the server, the connection is closed, and now the server will attempt to send it to the appropriate mail server for the destination address on the email (the RCPT TO field in the above session).

Server to Server Communication

The Postfix server that queued the mail in the previous section will use the SMTP protocol (again, but as a client rather than a server) to connect to the MTA that serves the foxlocksdocksandclocks.com domain. The sending MTA will use DNS to figure out what server to send the mail to. DNS is out of the scope of this article, but in my next article about DNS I’ll be covering various types of DNS records, including MX records. But, in the meantime, just know that mail servers query DNS for the MX (mail exchange) record of the receiving domain to find out what server to send mail to.

The conversation between the Postfix server on smtp.fancyduckenterprises.com and the MTA (which could be Postfix or any of a wide variety of MTAs) on foxlocksdocksandclocks.com will look strikingly similar to the conversation between Thunderbird and Postfix in the previous example. This is unsurprising, since they both use the SMTP protocol to transmit mail.

So, we won’t go into detail on that part of the story of this particular email, but I will point out that we could find the email in the queue of the server (while it is in the queue, which could be less than a second on a lightly loaded system and if the receiving server is also fast) with the email ID we talked about earlier by using the mailq or postqueue -p command.

Since we’ll be covering POP3/IMAP and client-side communications in the next article in this series, I’m going to skip over that part of the story for now, and get right to the basic configuration needed to achieve SMTP-to-SMTP conversations, both sending and receiving, and we’ll pick up the riveting tale of email ID 34A1E36F0 in the next article.

Installing Postfix

On modern Linux distributions, installing software is incredibly easy. CentOS has yum, while Debian and Ubuntu have apt. Both are easy to use, automatically resolve dependencies, and will fetch and install software with very little user involvement.

To install Postfix on CentOS, run the following command:

# yum install postfix

Or, on Debian or Ubuntu:

# apt-get update
# apt-get install postfix

Assuming your system is connected to the Internet, and the package manager is functional, within a few seconds you’ll have Postfix installed and ready to configure.

A Basic Postfix Configuration

Postfix, freshly installed from OS-provided packages in CentOS, Debian, or Ubuntu (and most other modern Linux distributions) is usually ready to start, and requires little to no configuration. But, since simply saying, “Start Postfix” seems like a cop-out, and this article is for people who want to learn how email works on a deeper level than merely starting the daemon and hoping for the best, I’ll go into more detail about the most commonly changed options in Postfix, as well as some troubleshooting information.

The primary concerns with your initial configuration, the “Just get it working!” phase, is in getting mail accepted for your domain, allowing it to send on behalf of your domain, and delivering mail to the right place on the system, so those are the configuration directives I’ll cover now.

myorigin – This directive determines the origin domain name that is used for mail sent from this system. By default, it will be set to $myhostname, which is a special variable that refers to the hostname of the system (in our example above that would be smtp.fancyduckenterprises.com). But, it may be preferable, particularly if you have multiple sending mail servers, to use the parent domain name (fancyduckenterprises.com), instead. To do that, you would change this option to:

myorigin = $mydomain

Note that this option only effects information that is used during SMTP communication. It does not limit or alter mail being sent through the server; the client sending email determines what the From address will be on any mail sent through your system, and it is that field that determines what appears in mail clients for the recipient of mail. If you only have one mail server for your domain, you probably do not need to change this option, and changing it adds some configuration complexity if you do change it (you have to also setup an alias for each user).

virtual_alias_maps – This directive  is used if your mail server will be receiving mail for users in multiple domains. e.g. if you are virtually hosting many mail domains on this server. I’ll go into more detail about virtual domains in a future article, but in the meantime, I wrote a brief tutorial about virtual hosting with Postfix for my Webmin book a few years back, which is still accurate in current Postfix versions. This option will generally point to a local map file, but could also use an LDAP directory or an SQL database on a remote machine, if you have multiple mail servers. In its simplest form, you’d configure it like this:

virtual_alias_maps = hash:/etc/postfix/virtual

Where hash indicates the type of map in use (it could also be ldap, mysql, or one of many other types. The file, /etc/postfix/virtual, will be a specially formatted list of email addresses and the destination for those addresses,

home_mailbox – This directive determines where user mail should be delivered on the system. Most modern systems use the Maildir format for mail delivery, as it can be more efficient for very large mailboxes than the next most popular option, mbox. It also makes incrementally backing up mail systems easier in some cases. If this option has a “/” at the end, it will use Maildir format. Otherwise, it will use mbox. So, to deliver mail to a Maildir in the users home directory, you’d configure it like this:

home_mailbox = Maildir/

That seems simple, and it is for very simple use cases, but in most environments, you will also want to enable spam and antivirus filtering, which generally requires more complexity, possibly using the mailbox_command option, which I will cover in a later article. In the meantime, this one option will allow Postfix to deliver mail to users on the system.

smtpd_recipient_restrictions – The final piece of the puzzle for a basic Postfix configuration is to insure that only the clients you want to allow to send mail can do so. You may have heard of an “open relay”, which is a term for any server that allows non-authenticated users to send mail through the server; being an open relay provides spammers with the ability to send their abusive email through your system (which can have disastrous consequences for your ability to deliver mail). I usually configure this option like so:

smtpd_recipient_restrictions = permit_mynetworks permit_sasl_authenticated reject_unauth_destination

These options determine which networks can send through this system ($mynetworks is another special variable that means the networks that the server is connected directly to, by default, but can be expanded to include other networks), allows SASL authenticated users to send through this system (for clients that are not within $mynetworks), and rejects mail sent to destinations other than the ones this server accepts mail for. Note that these options are applied in order, so the first that matches is the one that applies to any given message. Thus, if the mail is coming from $mynetworks, it will not need to pass the other checks. SASL authentication is a complex subject and will be covered in a future article.

Starting Postfix and Troubleshooting

After configuring the few options that you choose to alter, you can start Postfix and test your installation.

On CentOS 7, Debian 8, and other systemd-equipped systems, you could start it with:

# systemctl start postfix

If there are any problems, you can check the systemd journal using the journalctl command for why. It’s also useful to look in the mail log (/var/log/maillog on CentOS and /var/log/mail.log on Debian/Ubuntu).

On Ubuntu 14.04 and other upstart-based systems, you’d use the start command:

# start postfix

If there are any problems, the mail.log or the system log in /var/log/messages can be consulted to see what went wrong.

Sending Mail

A simple way to test a newly installed mail server is to use a local mail client to send mail from the server itself. This provides a very simple test case; external connections and clients are not needed, so you can isolate any failures down to a few specific misconfigurations, rather than trying to guess which part of a very complex system has failed.

There are several ways to send email from the command line, but one that is always available on any Postfix or Sendmail system is the sendmail command (Postfix provides a sendmail program to provide compatibility with Sendmail), so I’ll use that:

$ sendmail -t <<EOF
 > to:joe@virtualmin.com
 > from:joe@virtualmin.com
 > subject:Testing
 >
 > Tested!
 > EOF

If things go as expected, you should see the message appear in the inbox of the user you sent the message to. If they don’t go as expected, you can check the maillog or mail.log for clues about why.

Next Time

This is getting to be quite long, and we still haven’t really dug into client-side protocols, spam and anti-virus filtering, or DNS for mail service. But, fear not, in the next couple of weeks, I’ll be posting two new installments of this series on email, as well as another article about DNS, that cover just that.

Image credit: Kevin Dooley – Mailbox

Homebrew Package Installation for Servers

Homebrew logo

One of our customers came to us a couple of weeks ago wanting to run Virtualmin on Mac OS X. I, foolishly, said, “Sure! I can help with that. I installed Virtualmin on OS X several years ago, how much different can it be? It shouldn’t take more than a couple of hours.” It turns out, it can be remarkably different. In the intervening years, Mac OS X has evolved in many interesting directions, mostly positive, some questionable. Mac OS X remains exceedingly weak for server usage, for reasons well out of the scope of this short article. But, it is quite strong for desktop/laptop use, and many people want to be able to develop their web applications on Mac OS X even if they will be deployed to Linux servers.

Enter Homebrew

The last time I setup Virtualmin on Mac OS X systems, the best package management tool, and the best way to install a lot of Open Source software, was Fink (Mac Ports, then Darwin Ports, was still quite new at the time).

Fink is an apt repository of dpkg packages built for Mac OS X. I love apt and dpkg (almost as much as yum and rpm), and Webmin/Virtualmin have great support for apt and dpkg, and so I was all set to choose Fink for this new deployment. But, there are some issues with Fink. First, it installs everything with its own packages, including stuff that is already available from the base system. For Virtualmin, whenever possible, I like to use the system standard packages for the services it manages. Homebrew is designed to work with what Apple provides when possible, which is somewhat more aligned with the Virtualmin philosophy.

Second, and perhaps more important, Fink is seemingly much less popular and much less actively maintained than Homebrew. I’m not sure why. Possibly because the Homebrew website looks good, and the documentation is very well-written, while the Fink home page is a little drab and looks complicated. And, Fink package versions tend to be quite a bit older than the packages provided by the very active Homebrew project. This can be a much more serious issue. Security updates are absolutely vital in a web server, and a package repository that is actively maintained is the best way to insure you’ll have security updates.

So, I’ve spent the past couple of days experimenting with Homebrew. It’s a pretty nice system, and its community and developers are active, responsive, and helpful. All great things. But, its primary advertised feature is also its biggest weakness and most dangerous mistake.

Installation Without sudo

Or: Homebrew Considered Harmful

One of the major advertised features of Homebrew is that you can install it, and any package, without root or sudo privileges. There are good reasons one might want this, but on a server, it has alarming side effects, and it is one of the first things I would need to correct for our use case (of installing a virtual hosting stack and Virtualmin). The example I’ll use here is that of MySQL. When you install the mysql package from Homebrew, it will be installed with the ownership of all files set to the user that installed the package. And, more dangerously, it will be setup to run as the user that installed it.

This decision was made because Homebrew often builds the software at install time, rather than providing a binary package (there is a new “bottles” feature installs binary packages, but that wasn’t intended to address the sudo problem). The risk of building software with sudo or root privileges is very real, and in this case it results in the choice to build the software as a non-root user.

Other package managers, like dpkg and rpm, resolve this problem with toolchains designed around building the packages within a chroot so that unwanted behavior is contained. For example, mock on Fedora and CentOS provides an easy to use tool for building packages across many distributions and versions inside of a chroot environment with only the dependencies specified by the package. The most popular Linux distributions distribute binary packages that were built in a controlled environment. But, Homebrew generally builds the software at install time, with no chroot to protect the system from broken or hostile build processes. And, so they insist you run it as a non-root user. This is, I suppose, a logical conclusion to come to, based on the premise of a package manager that builds software on the user’s system without being confined in a container or chroot, but it has negative consequences.

For example, when I install MySQL from Homebrew, everything is owned by joe:staff. The provided property list file for starting the server is also designed to start it as that user, when the user logs in. For a development system, this may not be a big deal, and even makes a certain sort of sense (I prefer my development environment to more closely mirror my deployment environment, but I can see reasonable arguments for the way they do it). But, for a server, it is simply untenable.

The most important reason it is a bad decision is that it leads to many, possibly all, of the services running with the privileges of the user that installed them. Which, in most cases, is probably a powerful user (mine is an administrative user with sudo privileges, for example). So, in the event any of the services are compromised, all of the services will be compromised, and likely so will the user account in question. The security implications of this really cannot be overstated. This is a huge problem.

This is why Linux and UNIX systems (and even Apple, who aren’t historically renowned for their strong multi-user security practices) run all services as different users, and with restricted privileges. On the average LAMP system, there will be an apache or www user that runs Apache, a mysql user that runs MySQL, a nobody user that runs Postfix, and web applications will usually be run as yet another user still. These special users often have very restricted accounts, and may not even have a shell associated with them, further limiting the damage that can occur in the event of an exploit of any one service. Likewise, they may be further restricted by SELinux or other RBACL-based security. Any one of these services or applications being compromised through any means won’t generally compromise other services or users. Homebrew throws that huge security benefit away to avoid having to sudo during installation.

It’s probably too late to convince the Homebrew folks to backtrack on that decision. But, it’s not terribly difficult to fix for one-off installations, and many do consider it a valuable feature of Homebrew. Fixing the installed services as I’ve done has some side effects that may also be dangerous, which I’ll go into at the end of the article, but since I figured out how to do it, I thought I’d document it. During my research I found that an alarming number of users are using Homebrew in server environments and I found a number of users asking similar questions about various services, so, maybe this will help some folks avoid a dangerous situation.

So, let’s get started. After installation of MySQL (using the command brew install mysql), here’s the changes you’ll want to make.

Update The Property Lists File

The way Homebrew recommends running MySQL after installation is to link the provided plist file in /usr/local/opt/mysql/homebrew.mxcl.mysql.plist into your ~/Library/LaunchAgents directory, and add it using the launchctl load command. This sets it up to run at all times when your user is logged in, which is great if you’re developing and only need it running when you’re logged in and working. But, we want it to run during system boot without having any users logged in, and even more importantly we want it to run as the _mysql user.

So, instead of linking it into your local LaunchAgents directory, as the documentation suggests, copy it into your system /Library/LaunchDaemons directory.

$ sudo cp /usr/local/opt/mysql/homebrew.mxcl.mysql.plist /Library/LaunchDaemons

Then edit the file to add user and group information (you’ll have to use sudo), add a –user option, and change the command to mysqld and WorkingDirectory to /usr/local/var/mysql. Mine looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>UserName</key>
  <string>_mysql</string>
  <key>GroupName</key>
  <string>_mysql</string>
  <key>KeepAlive</key>
  <true/>
  <key>Label</key>
  <string>homebrew.mxcl.mysql</string>
  <key>ProgramArguments</key>
  <array>
    <string>/usr/local/opt/mysql/bin/mysqld</string>
    <string>--bind-address=127.0.0.1</string>
    <string>--datadir=/usr/local/var/mysql</string>
    <string>--user=_mysql</string>
  </array>
  <key>RunAtLoad</key>
  <true/>
  <key>WorkingDirectory</key>
  <string>/usr/local/var/mysql</string>
</dict>
</plist>

Notice the addition of the UserName and GroupName keys, both set to _mysql, as well as several other altered lines.

Note: I am not a Mac OS X or launchd expert. There are a number of aspects of the Mac OS X privilege model that I do not understand. I would welcome comments about how the security of this configuration might be improved. Also, the entirety of my experience with launchd is the several hours I spent playing with it and reading about it to convince MySQL to run as a different user. But, I’m pretty much certain that the way Homebrew does servers is worse than what I’ve done here.

Change Ownership of MySQL Installation and Databases

The _mysql user does not have permissions to read things owned by the joe user (or the user you used to install MySQL with Homebrew), so you’ll need to change ownership of all MySQL data files to _mysql:wheel.

$ sudo chown -R _mysql:wheel /usr/local/var/mysql/

Change Ownership of the Property List File

Property list files in the Library/LaunchDaemons directory (or the LaunchAgents directory) must belong to root, for security reasons. So, you’ll need to update that, as well.

$ sudo chown root:wheel /Library/LaunchDaemons/homebrew.mxcl.mysql.plist

Load and Start the MySQL Daemon

The launchctl command manages agents and daemons, and we can add our new service by loading the property list:

$ sudo launchctl load /Library/LaunchDaemons/homebrew.mxcl.mysql.plist

And, then we can start MySQL:

$ sudo launchctl start homebrew.mxcl.mysql.plist

If anything goes wrong, check the system.log:

$ sudo tail -f /var/log/system.log
I found the documentation at launchd.info particularly helpful when working out how to use launchd.

Future Concerns

Since this corrects the big security issue with the Homebrew installation of MySQL, and this technique could reasonably easily be applied across every Homebrew-installed service, why aren’t we happy?

Updates are part of security, too.

The most important reason to use a package manager is not to make it easy to install software, contrary to popular belief, and if all the package manager does is provide easy to install packages, it is not an effective package manager. The most important reason to use a package manager is to make it easy to update software.

Homebrew makes installation and upgrades reasonably easy, but the steps I’ve taken in this article to make MySQL run as its own user seems likely to break updates, since some files created during installation have changed ownership. A newer version of MySQL isn’t available in the Homebrew repository, so I can’t test whether it does break upgrades or not. Nonetheless, fixing this issue compounded across many services (the Virtualmin installation process normally installs: Apache, MySQL/MariaDB, PostgreSQL, Postfix, Dovecot, BIND, Procmail, SpamAssassin, ClamAV, Mailman, and a bunch more) will likely prove to be a maintenance challenge that is probably not worth the effort.

So, despite having figured out how to make this work, I’m now going to spend the same amount of time and effort giving Mac Ports a thorough test drive. I have a pretty strong suspicion it will be a better fit for server usage. This way of working feels like fighting against the way Homebrew wants to operate, and when you find yourself having to work so hard against the tool, it’s probably the wrong one for the job.

And What Does All This Mean for Virtualmin on Mac OS X?

Well, I’m a sucker for the sunk cost fallacy, so I’m planning to spend another couple of days working out a basic install script for Virtualmin on Mac OS X, probably using Mac Ports (I can see a way forward using Homebrew, but I don’t like the terrain). I’ll likely never recommend Mac OS X for a production server deployment, but it’s certainly not the worst OS out there for the purpose.