- Simple Mail Transfer Protocol
-
This article is about the Internet standard for electronic mail transmission. For the email delivery company, see SMTP (company).
Internet protocol suite Application layer Transport layer Internet layer Link layer Simple Mail Transfer Protocol (SMTP) is an Internet standard for electronic mail (e-mail) transmission across Internet Protocol (IP) networks. SMTP was first defined by RFC 821 (1982, eventually declared STD 10),[1] and last updated by RFC 5321 (2008)[2] which includes the extended SMTP (ESMTP) additions, and is the protocol in widespread use today. SMTP is specified for outgoing mail transport and uses TCP port 25. The protocol for new submissions is effectively the same as SMTP, but it uses port 587 instead. SMTP connections secured by SSL are known by the shorthand SMTPS, though SMTPS is not a protocol in its own right.
While electronic mail servers and other mail transfer agents use SMTP to send and receive mail messages, user-level client mail applications typically only use SMTP for sending messages to a mail server for relaying. For receiving messages, client applications usually use either the Post Office Protocol (POP) or the Internet Message Access Protocol (IMAP) or a proprietary system (such as Microsoft Exchange or Lotus Notes/Domino) to access their mail box accounts on a mail server.
Contents
History
Various forms of one-to-one electronic messaging were used in the 1960s. People communicated with one another using systems developed for specific mainframe computers. As more computers were interconnected, especially in the US Government's ARPANET, standards were developed to allow users of different systems to e-mail one another. SMTP grew out of these standards developed during the 1970s.
SMTP can trace its roots to two implementations described in 1971, the Mail Box Protocol, which has been disputed to actually have been implemented,[3] but is discussed in RFC 196 and other RFCs, and the SNDMSG program, which, according to RFC 2235, Ray Tomlinson of BBN "invents" for TENEX computers the sending of mail across the ARPANET.[4][5][6] Fewer than 50 hosts were connected to the ARPANET at this time.[7]
Further implementations include FTP Mail [8] and Mail Protocol, both from 1973.[9] Development work continued throughout the 1970s, until the ARPANET converted into the modern Internet around 1980. Jon Postel then proposed a Mail Transfer Protocol in 1980 that began to remove the mail's reliance on FTP.[10] SMTP was published as RFC 821 in August 1982, also by Postel.
The SMTP standard was developed around the same time as Usenet, a one-to-many communication network with some similarities.
SMTP became widely used in the early 1980s. At the time, it was a complement to Unix to Unix Copy Program (UUCP) mail, which was better suited for handling e-mail transfers between machines that were intermittently connected. SMTP, on the other hand, works best when both the sending and receiving machines are connected to the network all the time. Both use a store and forward mechanism and are examples of push technology. Though Usenet's newsgroups are still propagated with UUCP between servers,[11] UUCP mail has virtually disappeared[12] along with the "bang paths" it used as message routing headers.
The article about sender rewriting contains technical background info about the early SMTP history and source routing before RFC 1123.
Sendmail was one of the first (if not the first) mail transfer agents to implement SMTP.[citation needed] Some other popular SMTP server programs include Postfix, qmail, Novell GroupWise, Exim, Novell NetMail, Microsoft Exchange Server, Sun Java System Messaging Server.
Message submission (RFC 2476) and SMTP-AUTH (RFC 2554) were introduced in 1998 and 1999, both describing new trends in e-mail delivery. Originally, SMTP servers were typically internal to an organization, receiving mail for the organization from the outside, and relaying messages from the organization to the outside. But as time went on, SMTP servers (Mail transfer agents), in practice, were expanding their roles to become message submission agents for Mail user agents, some of which were now relaying mail from the outside of an organization. (e.g. a company executive wishes to send e-mail while on a trip using the corporate SMTP server.) This issue, a consequence of the rapid expansion and popularity of the World Wide Web, meant that SMTP had to include specific rules and methods for relaying mail and authenticating users to prevent abuses such as relaying of unsolicited e-mail (spam).
As this protocol started out purely ASCII text-based, it did not deal well with binary files. Standards such as Multipurpose Internet Mail Extensions (MIME) were developed to encode binary files for transfer through SMTP. Mail transfer agents (MTAs) developed after Sendmail also tended to be implemented 8-bit-clean, so that the alternate "just send eight" strategy could be used to transmit arbitrary text data (in any 8-bit ASCII-like character encoding) via SMTP. 8-bit-clean MTAs today tend to support the 8BITMIME extension, permitting binary files to be transmitted almost as easily as plain text.
Many people contributed to the core SMTP specifications, among them Jon Postel, Eric Allman, Dave Crocker, Ned Freed, Randall Gellens, John Klensin, and Keith Moore.
Mail processing model
Blue arrows can be implemented using SMTP variations.
Email is submitted by a mail client (MUA, mail user agent) to a mail server (MSA, mail submission agent) using SMTP on TCP port 587. Most mailbox providers still allow submission on traditional port 25. From there, the MSA delivers the mail to its mail transfer agent (MTA, mail transfer agent). Often, these two agents are just different instances of the same software launched with different options on the same machine. Local processing can be done either on a single machine, or split among various appliances; in the former case, involved processes can share files; in the latter case, SMTP is used to transfer the message internally, with each host configured to use the next appliance as a smart host. Each process is an MTA in its own right; that is, an SMTP server.
The boundary MTA has to locate the target host. It uses the Domain name system (DNS) to look up the mail exchanger record (MX record) for the recipient's domain (the part of the address on the right of @). The returned MX record contains the name of the target host. The MTA next connects to the exchange server as an SMTP client. (The article on MX record discusses many factors in determining which server the sending MTA connects to.)
Once the MX target accepts the incoming message, it hands it to a mail delivery agent (MDA) for local mail delivery. An MDA is able to save messages in the relevant mailbox format. Again, mail reception can be done using many computers or just one —the picture displays two nearby boxes in either case. An MDA may deliver messages directly to storage, or forward them over a network using SMTP, or any other means, including the Local Mail Transfer Protocol (LMTP), a derivative of SMTP designed for this purpose.
Once delivered to the local mail server, the mail is stored for batch retrieval by authenticated mail clients (MUAs). Mail is retrieved by end-user applications, called email clients, using Internet Message Access Protocol (IMAP), a protocol that both facilitates access to mail and manages stored mail, or the Post Office Protocol (POP) which typically uses the traditional mbox mail file format or a proprietary system such as Microsoft Exchange/Outlook or Lotus Notes/Domino. Webmail clients may use either method, but the retrieval protocol is often not a formal standard.
SMTP defines message transport, not the message content. Thus, it defines the mail envelope and its parameters, such as the envelope sender, but not the header or the body of the message itself. STD 10 and RFC 5321 define SMTP (the envelope), while STD 11 and RFC 5322 define the message (header and body), formally referred to as the Internet Message Format.
Protocol overview
SMTP is a connection-oriented, text-based protocol in which a mail sender communicates with a mail receiver by issuing command strings and supplying necessary data over a reliable ordered data stream channel, typically a Transmission Control Protocol (TCP) connection. An SMTP session consists of commands originated by an SMTP client (the initiating agent, sender, or transmitter) and corresponding responses from the SMTP server (the listening agent, or receiver) so that the session is opened, and session parameters are exchanged. A session may include zero or more SMTP transactions. An SMTP transaction consists of three command/reply sequences (see example below.) They are:
- MAIL command, to establish the return address, a.k.a. Return-Path, 5321.From, mfrom, or envelope sender. This is the address for bounce messages.
- RCPT command, to establish a recipient of this message. This command can be issued multiple times, one for each recipient. These addresses are also part of the envelope.
- DATA to send the message text. This is the content of the message, as opposed to its envelope. It consists of a message header and a message body separated by an empty line. DATA is actually a group of commands, and the server replies twice: once to the DATA command proper, to acknowledge that it is ready to receive the text, and the second time after the end-of-data sequence, to either accept or reject the entire message.
Besides the intermediate reply for DATA, each server's reply can be either positive (2xx reply codes) or negative. Negative replies can be permanent (5xx codes) or transient (4xx codes). A reject is a permanent failure by an SMTP server; in this case the SMTP client should send a bounce message. A drop is a positive response followed by message discard rather than delivery.
The initiating host, the SMTP client, can be either an end-user's email client, functionally identified as a mail user agent (MUA), or a relay server's mail transfer agent (MTA), that is an SMTP server acting as an SMTP client, in the relevant session, in order to relay mail. Fully capable SMTP servers maintain queues of messages for retrying message transmissions that resulted in transient failures.
A MUA knows the outgoing mail SMTP server from its configuration. An SMTP server acting as client, i.e. relaying, typically determines which SMTP server to connect to by looking up the MX (Mail eXchange) DNS resource record for each recipient's domain name. Conformant MTAs (not all) fall back to a simple A record in case no MX record can be found. Relaying servers can also be configured to use a smart host.
An SMTP server acting as client initiates a TCP connection to the server on the "well-known port" designated for SMTP: port 25. MUAs should use port 587 to connect to an MSA. The main difference between an MTA and an MSA is that SMTP Authentication is mandatory for the latter only.
SMTP vs mail retrieval
SMTP is a delivery protocol only. It cannot pull messages from a remote server on demand. Other protocols, such as the Post Office Protocol (POP) and the Internet Message Access Protocol (IMAP) are specifically designed for retrieving messages and managing mail boxes. However, SMTP has a feature to initiate mail queue processing on a remote server so that the requesting system may receive any messages destined for it (see Remote Message Queue Starting below). POP and IMAP are preferred protocols when a user's personal computer is only intermittently powered up, or Internet connectivity is only transient and hosts cannot receive message during off-line periods.
Remote Message Queue Starting
Remote Message Queue Starting is a feature of SMTP that permits a remote host to start processing of the mail queue on a server so it may receive messages destined to it by sending the TURN command. This feature however was deemed insecure[13] and was extended in RFC 1985 with the ETRN command which operates more securely using an authentication method based on Domain Name System information.
On-Demand Mail Relay
Main article: On-Demand Mail RelayOn-Demand Mail Relay (ODMR) is an SMTP extension standardized in RFC 2645 that allows e-mail to be relayed to an authenticated recipient.
Internationalization
RFC 5336 describes internationalization features for SMTP, the UTF8SMTP extension, which provides support for multi-byte and non-ASCII characters in email addresses, such as Pelé@live.com (simple diacritic), δοκιμή@παράδειγμα.δοκιμή, and 测试@测试.测试.
Outgoing mail SMTP server
An e-mail client needs to know the IP address of an SMTP server and this has to be given as part of its configuration (usually given as a DNS name). The server will deliver outgoing messages on behalf of the user.
Outgoing mail server access restrictions
Server administrators need to impose some control on which clients can use the server. This enables them to deal with abuse, for example spamming. Two solutions have been in common use:
- In the past, many systems imposed usage restrictions by the location of the client, only permitting usage by clients whose IP address is one that the server administrators control. Usage from any other client IP address is disallowed.
- Modern SMTP servers typically offer an alternative system that requires authentication of clients by credentials before allowing access.
Restricting access by location
Under this system, an ISP's SMTP server will not allow access by users who are 'outside the ISP's network'. More precisely, the server may only allow access to users with an IP address provided by the ISP, which is equivalent to requiring that they are connected to the Internet using that same ISP. A mobile user may often be on a network other than that of their normal ISP, and will then find that sending email fails because the configured SMTP server choice is no longer accessible.
This system has several variations. For example, an organisation's SMTP server may only provide service to users on the same network, enforcing this by firewalling to block access by users on the wider Internet. Or the server may perform range checks on the client's IP address. These methods were typically used by corporations and institutions such as universities which provided an SMTP server for outbound mail only for use internally within the organisation. However, most of these bodies now use client authentication methods, as described below.
By restricting access to certain IP addresses, server administrators can readily recognise the IP address of any abuser. As it will be a meaningful address to them, the administrators can deal with the rogue machine or user.
Where a user is mobile, and may use different ISPs to connect to the internet, this kind of usage restriction is onerous, and altering the configured outbound email SMTP server address is impractical. It is highly desirable to be able to use email client configuration information that does not need to change.
Client authentication
Modern SMTP servers typically require authentication of clients by credentials before allowing access, rather than restricting access by location as described earlier. This more flexible system is friendly to mobile users and allows them to have a fixed choice of configured outbound SMTP server.
Open relay
A server that is accessible on the wider Internet and does not enforce these kinds of access restrictions is known as an open relay. This is now generally considered a bad practice worthy of blacklisting.
Ports
Server administrators choose whether clients use TCP port 25 (SMTP) or port 587 (Submission), as formalized in RFC 6409, for relaying outbound mail to a mail server. The specifications and many servers support both. Although some servers support port 465 for legacy secure SMTP in violation of the specifications, it is preferable to use standard ports and standard ESMTP commands[14] according to RFC 3207 if a secure session needs to be used between the client and the server.
Some servers are set up to reject all relaying on port 25, but valid users authenticating on port 587 are allowed to relay mail to any valid address.
Some Internet service providers intercept port 25, redirecting traffic to their own SMTP server regardless of the destination address. This means that it is not possible for their users to access an SMTP server outside the ISP's network using port 25.
Some SMTP servers support authenticated access on an additional port other than 25 to allow users to connect to them even if port 25 is blocked.
SMTP transport example
A typical example of sending a message via SMTP to two mailboxes (alice and theboss) located in the same mail domain (example.com) is reproduced in the following session exchange.
For illustration purposes here (not part of protocol), the protocol exchanges are prefixed for the server (S:) and the client (C:).
After the message sender (SMTP client) establishes a reliable communications channel to the message receiver (SMTP server), the session is opened with a greeting by the server, usually containing its fully qualified domain name (FQDN), in this case smtp.example.com. The client initiates its dialog by responding with a
HELOcommand identifying itself in the command's parameter with its FQDN (or an address literal if none is available).[2]S: 220 smtp.example.com ESMTP Postfix C: HELO relay.example.org S: 250 Hello relay.example.org, I am glad to meet you C: MAIL FROM:<bob@example.org> S: 250 Ok C: RCPT TO:<alice@example.com> S: 250 Ok C: RCPT TO:<theboss@example.com> S: 250 Ok C: DATA S: 354 End data with <CR><LF>.<CR><LF> C: From: "Bob Example" <bob@example.org> C: To: "Alice Example" <alice@example.com> C: Cc: theboss@example.com C: Date: Tue, 15 Jan 2008 16:02:43 -0500 C: Subject: Test message C: C: Hello Alice. C: This is a test message with 5 header fields and 4 lines in the message body. C: Your friend, C: Bob C: . S: 250 Ok: queued as 12345 C: QUIT S: 221 Bye {The server closes the connection}The client notifies the receiver of the originating email address of the message in a
MAIL FROMcommand. In this example, the email message is sent to two mailboxes on the same SMTP server: one each for each recipient listed in the To and Cc header fields. The corresponding SMTP command isRCPT TO. Each successful reception and execution of a command is acknowledged by the server with a result code and response message (e.g., 250 Ok).The transmission of the body of the mail message is initiated with a
DATAcommand after which it is transmitted verbatim line by line and is terminated with an end-of-data sequence. This sequence consists of a new-line (<CR><LF>), a single full stop (period), followed by another new-line. Since a message body can contain a line with just a period as part of the text, the client sends two periods every time a line starts with a period; correspondingly, the server replaces every sequence of two periods at the beginning of a line with a single one. Such escaping method is called dot-stuffing.The server's positive reply to the end-of-data, as exemplified, implies that the server has taken the responsibility of delivering the message. A message can be doubled if there is a communication failure at this time, e.g. due to a power shortage: Until the sender has received that 250 reply, it must assume the message was not delivered. On the other hand, after the receiver has decided to accept the message, it must assume the message has been delivered to it. Thus, during this time span, both agents have active copies of the message that they will try to deliver.[15] The probability that a communication failure occurs exactly at this step is directly proportional to the amount of filtering that the server performs on the message body, most often for anti-spam purposes. The limiting timeout is specified to be 10 minutes.[16]
The
QUITcommand ends the session. If the second recipient were located elsewhere, the client wouldQUITand connect to the appropriate SMTP server after the first message had been queued. The information that the client sends in theHELOandMAIL FROMcommands are added (not seen in example code) as additional header fields to the message by the receiving server. It adds aReceivedandReturn-Pathheader field, respectively.Optional extensions
Although optional and not shown in this example, many clients ask the server for the SMTP extensions that the server supports, by using the
EHLOgreeting of the extended SMTP specification (RFC 1870). Clients fall back toHELOonly if the server does not respond toEHLO.Modern clients may use the ESMTP extension keyword
SIZEto query the server for the maximum message size that will be accepted. Older clients and servers may try to transfer excessively sized messages that will be rejected after consuming network resources, including connect time to network links that is paid by the minute.Users can manually determine in advance the maximum size accepted by ESMTP servers. The client replaces the
HELOcommand with theEHLOcommand.S: 220 smtp2.example.com ESMTP Postfix C: EHLO bob.example.org S: 250-smtp2.example.com Hello bob.example.org [192.0.2.201] S: 250-SIZE 14680064 S: 250-PIPELINING S: 250 HELP
Thus smtp2.example.com declares that it will accept a fixed maximum message size no larger than 14,680,064 octets (8-bit bytes). Depending on the server's actual resource usage, it may be currently unable to accept a message this large. In the simplest case, an ESMTP server will declare a maximum SIZE with only the EHLO user interaction.
Security and spamming
Main article: Anti-spam techniques (e-mail)The original SMTP specification did not include a facility for authentication of senders. Subsequently, the SMTP-AUTH extension was defined by RFC 2554.[17] The SMTP extension (ESMTP) provides a mechanism for email clients to specify a security mechanism to a mail server, authenticate the exchange, and negotiate a security profile (Simple Authentication and Security Layer, SASL) for subsequent message transfers.
Microsoft products implement the proprietary Secure Password Authentication (SPA) protocol through the use of the SMTP-AUTH extension.
However, the impracticality of widespread SMTP-AUTH implementation and management means that E-mail spamming is not and cannot be addressed by it.
Modifying SMTP extensively, or replacing it completely, is not believed to be practical, due to the network effects of the huge installed base of SMTP. Internet Mail 2000 was one such proposal for replacement.
Spam is enabled by several factors, including vendors implementing broken MTAs (that do not adhere to standards, and therefore make it difficult for other MTAs to enforce standards), security vulnerabilities within the operating system (often exacerbated by always-on broadband connections) that allow spammers to remotely control end-user PCs and cause them to send spam, and a lack of "intelligence" in many MTAs.
There are a number of proposals for sideband protocols that will assist SMTP operation. The Anti-Spam Research Group (ASRG) of the Internet Research Task Force (IRTF) is working on a number of E-mail authentication and other proposals for providing simple source authentication that is flexible, lightweight, and scalable. Recent Internet Engineering Task Force (IETF) activities include MARID (2004) leading to two approved IETF experiments in 2005, and DomainKeys Identified Mail in 2006.
Implementations
Main article: List of mail serversRelated Requests For Comments
- RFC 1123 – Requirements for Internet Hosts—Application and Support (STD 3)
- RFC 1870 – SMTP Service Extension for Message Size Declaration (оbsoletes: RFC 1653)
- RFC 2505 – Anti-Spam Recommendations for SMTP MTAs (BCP 30)
- RFC 2920 – SMTP Service Extension for Command Pipelining (STD 60)
- RFC 3030 – SMTP Service Extensions for Transmission of Large and Binary MIME Messages
- RFC 3207 – SMTP Service Extension for Secure SMTP over Transport Layer Security (obsoletes RFC 2487)
- RFC 3461 – SMTP Service Extension for Delivery Status Notifications (obsoletes RFC 1891)
- RFC 3462 – The Multipart/Report Content Type for the Reporting of Mail System Administrative Messages (obsoletes RFC 1892)
- RFC 3463 – Enhanced Status Codes for SMTP (obsoletes RFC 1893 )
- RFC 3464 – An Extensible Message Format for Delivery Status Notifications (obsoletes RFC 1894)
- RFC 3834 – Recommendations for Automatic Responses to Electronic Mail
- RFC 4952 – Overview and Framework for Internationalized E-mail
- RFC 4954 – SMTP Service Extension for Authentication (obsoletes RFC 2554)
- RFC 5068 – E-mail Submission Operations: Access and Accountability Requirements (BCP 134)
- RFC 5321 – The Simple Mail Transfer Protocol (obsoletes RFC 821 aka STD 10, RFC 974, RFC 1869, RFC 2821)
- RFC 5322 – Internet Message Format (obsoletes RFC 822 aka STD 11, and RFC 2822)
- RFC 5336 - SMTP Extension for Internationalized Email Addresses (updates RFC 2821, RFC 2822, and RFC 4952)
- RFC 5504 - Downgrading Mechanism for Email Address Internationalization
- RFC 6409 – Message Submission for Mail (obsoletes RFC 4409, RFC 2476)
See also
- Bounce messages (SMTP non-delivery reports), bounce address
- Comparison of mail servers
- E-mail authentication
- E-mail encryption
- Extended SMTP (ESMTP)
- Ident
- List of mail servers
- POP before SMTP / SMTP after POP
- Sender Policy Framework (SPF)
- SMTP-AUTH (ESMTPA)
- SMTPS
- Variable envelope return path
References
- ^ RFC 821, Simple Mail Transfer Protocol, J.B. Postel, The Internet Society (August 1982)
- ^ a b RFC 5321, Simple Mail Transfer Protocol, J. Klensin, The Internet Society (October 2008)
- ^ The History of Electronic Mail, Tom Van Vleck: "It is not clear this protocol was ever implemented"
- ^ The First Network Email, Ray Tomlinson, BBN
- ^ Picture of "The First Email Computer" by Dan Murphy, a PDP-10
- ^ Dan Murphy's TENEX and TOPS-20 Papers
- ^ RFC 2235
- ^ RFC 469 - Network Mail Meeting Summary
- ^ RFC 524 - A Proposed Mail Protocol
- ^ RFC 772 - Mail Transfer Protocol
- ^ Tldp.org
- ^ draft-barber-uucp-project-conclusion-05 - The Conclusion of the UUCP Mapping Project
- ^ RFC 1985, SMTP Service Extension for Remote Message Queue Starting, J. De Winter, The Internet Society (August 1996)
- ^ RFC 3207 specifies only the well-known port 25 and the "Submission port," which is TCP port 587, for the STARTTLS command, the precursor for an encrypted SMTP session using TLS. It makes no mention of the unofficial port 465.
- ^ RFC 1047
- ^ rfc5321#section-4.5.3.2.6
- ^ RFC 2554, SMTP Service Extension for Authentication, J. Myers (March 1999)
Further reading
- Hughes, L (1998). Internet e-mail Protocols, Standards and Implementation. Artech House Publishers. ISBN 0890069395.
- Hunt, C (2003). sendmail Cookbook. O'Reilly Media. ISBN 0596004710.
- Johnson, K (2000). Internet Email Protocols: A Developer's Guide. Addison-Wesley Professional. ISBN 0201432889.
- Loshin, P (1999). Essential Email Standards: RFCs and Protocols Made Practical. John Wiley & Sons. ISBN 0471345970.
- Rhoton, J (1999). Programmer's Guide to Internet Mail: SMTP, POP, IMAP, and LDAP. Elsevier. ISBN 1555582125.
- Wood, D (1999). Programming Internet Mail. O'Reilly. ISBN 1565924797.
External links
- Essential Internet Protocols - SMTP
- SMTP Sequence Diagram (PDF)
- Diagram of e-mail flow (PDF, PNG )
- The Case For E-mail Security - Security and Insecurity in SMTP, POP and IMAP.
- Picture of the first computers to send and receive a network email, 2 PDP-10s
- Email Address Internationalization IETF Working Group
- SMTP TLS Transport real-time test - Live version of above example, mostly for TLS (i.e. secure) email but useful for non-TLS too
Categories:- Internet mail protocols
Wikimedia Foundation. 2010.
