Maildir

The Maildir e-mail format is a common way of storing e-mail messages, where each message is kept in a separate file with a unique name, and each folder is a directory. The local filesystem handles file locking as messages are added, moved and deleted.

Internal structure nl:Bestand:Maildir.png.

Specifications

A Maildir directory (often named Maildir) usually has three subdirectories named tmp, new, and cur.

Maildir

The original Maildir specification was written by Daniel J. Bernstein, the author of qmail, djbdns, and other software.^[1] Although the original specification was written specifically for Bernstein's qmail, it is general enough to be implemented in many programs.

Maildir++

Sam Varshavchik, the author of the Courier Mail Server and other software, wrote an extension^[2] to the Maildir format called Maildir++ to support subfolders and mail quotas. Maildir++ directories contain subdirectories with names that start with a '.' (dot) that are also Maildir++ folders. This extension is therefore a violation of the Maildir specification, which provides an exhaustive list of the possible contents of a Maildir, however it is a compatible violation and other Maildir software supports Maildir++.

Technical operation

The process that delivers an e-mail message writes it to a file in the tmp directory with a unique filename. The current algorithm for generating the unique filename combines the time, the host name, and a number of pseudo-random parameters to ensure uniqueness.^[1]

The delivery process stores the message in the maildir by creating and writing to tmp/unique, and then moving this file to new/unique. The moving is commonly done by hard linking the file to new and then unlinking the file from tmp, but some implementations simply rename() it there. This sequence guarantees that a maildir-reading program will not see a partially-written message, as MUAs never look in tmp.

When the mail user agent process finds messages in the new directory it moves them to cur (using rename() - link then unlink strategy may result in having the message duplicated) and appends an informational suffix to the filename before reading them. The information suffix consists of a colon (to separate the unique part of the filename from the actual information), a '2', a comma and various flags. The '2' specifies, loosely speaking, the version of the information that follows the comma. '2' is the only currently officially specified version, '1' being an experimental version. One can only assume that it was used while the Maildir format was under development. The specification defines flags which show whether the message has been read, deleted and so on: the initial (capital) letter of Passed, Replied, Seen, Trashed, Draft, and Flagged.^[1] Dovecot uses lowercase letters to match 26 IMAP keywords,^[3] which may include standardised keywords such as $MDNSent, and user defined flags.

Technical issues

Inconsistent state with lockless operation

Daniel J. Bernstein designed Maildir to be safely writable by multiple concurrent writers without any form of explicit locking, even over NFS. This works reasonably well in practice, but can result in strangeness. During directory listing, any files that are renamed after the first readdir() system call and before the last readdir() call may not appear in the listing. This causes the listing process to believe that the message was deleted, while in reality only its flags were changed. When the process lists the messages again, the "deleted" message suddenly reappears. Some mail-accessing programs layer their own locking on top of Maildir in an attempt to prevent this kind of problem. Dovecot, for example, uses its own non-standard locking with Maildir.

Locking & scaling

There remains the implicit locking used by the filesystem in updating the directories; non-cluster filesystems will typically only allow one kernel thread at a time to be updating what is in a directory, so the rename() system call will provide the locking needed. Maildir is not lock-free, only explicit-lock-free. For many small to medium mail-systems, this scales adequately even over NFS, but as you grow to large systems handling many concurrent deliveries, the constant changing of the directory contents across multiple directories will result in repeated cache invalidation of the different NFS clients, so READDIR RPCs will have to re-issued, scaling poorly. Further, many filesystems have per-directory file-count limits.

Maildir thus suffers from the inherent scaling limitations of any one-email-per-file email storage design.

Filesystem compatibility

The Maildir standard cannot be implemented without modification on systems that do not accept colons in filenames. This includes Microsoft Windows and some configurations of Novell Storage Services.

Software running on these systems can use an alternative separator (such as ";", or "-"), and it is often trivial to patch free and open source software to use a different separator.^[4]

As there is currently no agreement on what character this alternative separator should be, there can be interoperability difficulties between different Maildir-supporting programs on these systems. However, not all Maildir-related software needs to know what the separator character is, because not all Maildir-related software needs to be able to read or modify the flags of a message ("read", "replied to" etc.); software that merely delivers to a Maildir, or archives old messages from it based only on date, should work no matter what separator is in use. If only the MUA needs to read or modify message flags, and only one is used, then non-standard alternative separators may be used without interoperability problems.

Software that supports Maildir directly

Mail servers

• bincimap IMAP server
• Dovecot IMAP server
• Courier Mail Server SMTP and IMAP server, for which the Maildir++ format was invented
• Exim SMTP server
• Postfix SMTP server
• Qmail SMTP server, for which Maildir format was invented
• XMail, an open source cross-platform (*nix and Windows) SMTP and POP3 server
• MeTA1 SMTP server

Delivery agents

• Procmail
• Maildrop
• Getmail, a Maildir-aware mail-retrieval and delivery agent alternative to Fetchmail
• mbsync
• OfflineIMAP
• mswatch
• mpop

Mail readers

• Balsa previously the official GNOME mail reader (prior to Evolution)
• Cone a curses-based mail reader
• Gnus
• mailx
• GNUMail
• KMail, KDE mail reader
• Mutt
• Evolution, official GNOME mail client
• Opera Mail
• Wanderlust

Mail index and search tools

• Beagle (software) can index Maildirs and many other information storage formats
• Mairix is a program for indexing and searching email messages stored in maildir, MH or mbox folders
• notmuch is a program for indexing and searching email messages stored in maildir

Software that supports Maildir by implication

The list of software that can be used with Maildir is in fact much larger if you consider how this software can be plugged together, and the role of network access protocols.

For example:

• The Sendmail MTA does not support any mail delivery format (although many assume that it does). Sendmail uses a separate delivery process called mail.local. Procmail (and other programs that support Maildir) can be used in place of mail.local, so Sendmail can rightly be said to support Maildir as much as it supports any other format.
• Many mail readers do not support Maildir but do support remote access formats such as IMAP. Since there are several IMAP mail stores that support Maildir, any mail reader that supports IMAP such as Microsoft Outlook, Pine, or Mozilla Thunderbird can be used to access Maildir folders.
• Fetchmail does not support Maildir (or any local delivery format) but since it talks to an SMTP server or local delivery agent, any of those listed above can be used to deliver mail from Fetchmail to Maildirs.

Notes and references

1. ^ ^{a b c} Daniel J. Bernstein. (1995) Using maildir format (the original specification)
2. ^ Varshavchik, Sam (1998) Maildir++ and Maildir quotas which has the Maildir++ specification buried within it
3. ^ Dovecot Wiki: maildir format
4. ^ mutt maildir support: workaround for filesystems that don't accept colons

See also

• mbox
• MH Message Handling System
• MIX (Email)

External links

• manual page for maildir
• maildir specifications