Mail Services (TCP/IP Network Administration, 3rd Edition)

3.4. Mail Services

Users consider electronic mail the most important network service because they use it for interpersonal communications. Some applications are newer and fancier; others consume more network bandwidth; and others are more important for the continued operation of the network. But email is the application people use to communicate with each other. It isn't very fancy, but it is vital.

TCP/IP provides a reliable, flexible email system built on a few basic protocols. These protocols are Simple Mail Transfer Protocol (SMTP), Post Office Protocol (POP), Internet Message Access Protocol (IMAP), and Multipurpose Internet Mail Extensions (MIME). There are other TCP/IP mail protocols that have some interesting features, but they are not yet widely implemented.

Our coverage concentrates on the four protocols you are most likely to use building your network: SMTP, POP, IMAP, and MIME. We start with SMTP, the foundation of all TCP/IP email systems.

3.4.1. Simple Mail Transfer Protocol

SMTP is the TCP/IP mail delivery protocol. It moves mail across the Internet and across your local network. SMTP is defined in RFC 821, A Simple Mail Transfer Protocol. It runs over the reliable, connection-oriented service provided by Transmission Control Protocol (TCP), and it uses well-known port number 25.[23] Table 3-1 lists some of the simple, human-readable commands used by SMTP.

[23]Most standard TCP/IP applications are assigned a well-known port so that remote systems know how to connect the service.

Table 3-1. SMTP commands

Command	Syntax	Function
Hello	HELO <sending-host> EHLO <sending-host>	Identify sending SMTP
From	MAIL FROM:<from-address>	Sender address
Recipient	RCPT TO:<to-address>	Recipient address
Data	DATA	Begin a message
Reset	RSET	Abort a message
Verify	VRFY <string>	Verify a username
Expand	EXPN <string>	Expand a mailing list
Help	HELP [string]	Request online help
Quit	QUIT	End the SMTP session

SMTP is such a simple protocol you can literally do it yourself. telnet to port 25 on a remote host and type mail in from the command line using the SMTP commands. This technique is sometimes used to test a remote system's SMTP server, but we use it here to illustrate how mail is delivered between systems. The example below shows mail that Daniel on rodent.wrotethebook.com manually input and sent to Tyler on crab.wrotethebook.com.

$ telnet crab 25
Trying 172.16.12.1...
Connected to crab.wrotethebook.com.
Escape character is '^]'.
220 crab.wrotethebook.com ESMTP Sendmail 8.9.3+Sun/8.9.3; Thu, 19 Apr 2001 16:28:01-0400 (EDT)
HELO rodent.wrotethebook.com
250 crab.wrotethebook.com Hello rodent [172.16.12.2], pleased to meet you 
MAIL FROM:<daniel@rodent.wrotethebook.com>
250 <daniel@rodent.wrotethebook.com>... Sender ok
RCPT TO:<tyler@crab.wrotethebook.com>
250 <tyler@crab.wrotethebook.com>... Recipient ok
DATA
354 Enter mail, end with "." on a line by itself
Hi Tyler!
.
250 QAA00316 Message accepted for delivery
QUIT
221 crab.wrotethebook.com closing connection
Connection closed by foreign host.

The user input is shown in bold type. All of the other lines are output from the system. This example shows how simple it is. A TCP connection is opened. The sending system identifies itself. The From address and the To address are provided. The message transmission begins with the DATA command and ends with a line that contains only a period (.). The session terminates with a QUIT command. Very simple, and very few commands are used.

There are other commands (SEND, SOML, SAML, and TURN) defined in RFC 821 that are optional and not widely implemented. Even some of the commands that are implemented are not commonly used. The commands HELP, VRFY, and EXPN are designed more for interactive use than for the normal machine-to-machine interaction used by SMTP. The following excerpt from a SMTP session shows how these odd commands work.

HELP
214-This is Sendmail version 8.9.3+Sun
214-Topics:
214-    HELO    EHLO    MAIL    RCPT    DATA
214-    RSET    NOOP    QUIT    HELP    VRFY
214-    EXPN    VERB    ETRN    DSN
214-For more info use "HELP <topic>". 
214-For local information contact postmaster at this site. 
214 End of HELP info 
HELP RSET 
214-RSET 
214-    Resets the system. 
214 End of HELP info 
VRFY <jane> 
250 <jane@brazil.wrotethebook.com> 
VRFY <mac> 
250 Kathy McCafferty <<mac>> 
EXPN <admin> 
250-<sara@horseshoe.wrotethebook.com> 
250 David Craig <<david>>
250-<tyler@wrotethebook.com>

The HELP command prints out a summary of the commands implemented on the system. The HELP RSET command specifically requests information about the RSET command. Frankly, this help system isn't very helpful!

The VRFY and EXPN commands are more useful but are often disabled for security reasons because they provide user account information that might be exploited by network intruders. The EXPN <admin> command asks for a listing of the email addresses in the mailing list admin, and that is what the system provides. The VRFY command asks for information about an individual instead of a mailing list. In the case of the VRFY <mac> command, mac is a local user account, and the user's account information is returned. In the case of VRFY <jane>, jane is an alias in the /etc/aliases file. The value returned is the email address for jane found in that file. The three commands in this example are interesting but rarely used. SMTP depends on the other commands to get the real work done.

SMTP provides direct end-to-end mail delivery. Other mail systems, like UUCP and X.400, use store and forward protocols that move mail toward its destination one hop at a time, storing the complete message at each hop and then forwarding it on to the next system. The message proceeds in this manner until final delivery is made. Figure 3-3 illustrates both store-and-forward and direct-delivery mail systems. The UUCP address clearly shows the path that the mail takes to its destination, while the SMTP mail address implies direct delivery.[24]

[24]The address doesn't have anything to do with whether a system is store and forward or direct delivery. It just happens that UUCP provides an address that helps to illustrate this point.

Figure 3-3. Mail delivery systems

Direct delivery allows SMTP to deliver mail without relying on intermediate hosts. If the delivery fails, the local system knows it right away. It can inform the user that sent the mail or queue the mail for later delivery without reliance on remote systems. The disadvantage of direct delivery is that it requires both systems to be fully capable of handling mail. Some systems cannot handle mail, particularly small systems such as PCs or mobile systems such as laptops. These systems are usually shut down at the end of the day and are frequently offline. Mail directed from a remote host fails with a "cannot connect" error when the local system is turned off or is offline. To handle these cases, features in the DNS system are used to route the message to a mail server in lieu of direct delivery. The mail is then moved from the server to the client system when the client is back online. One of the protocols TCP/IP networks use for this task is POP.

3.4.2. Post Office Protocol

There are two versions of Post Office Protocol: POP2 and POP3. POP2, defined in RFC 937, uses port 109, and POP3, defined in RFC 1725 , uses port 110. These are incompatible protocols that use different commands, although they perform the same basic functions. The POP protocols verify the user's login name and password and move the user's mail from the server to the user's local mail reader. POP2 is rarely used anymore, so this section focuses on POP3.

A sample POP3 session clearly illustrates how a POP protocol works. POP3 is a simple request/response protocol, and just as with SMTP, you can type POP3 commands directly into its well-known port (110) and observe their effect. Here's an example with the user input shown in bold type:

% telnet crab 110 
Trying 172.16.12.1 ... 
Connected to crab.wrotethebook.com. 
Escape character is '^]'. 
+OK crab POP3 Server Process 3.3(1) at Mon 16-Apr-2001 4:48PM-EDT 
USER hunt 
+OK User name (hunt) ok. Password, please. 
PASS Watts?Watt? 
+OK 3 messages in folder NEWMAIL (V3.3 Rev B04) 
STAT 
+OK 3 459 
RETR 1 
+OK 146 octets   
...The full text of message 1... 
DELE 1
+OK message # 1 deleted 
RETR 2 
+OK 155 octets   
...The full text of message 2...
DELE 2
+OK message # 2 deleted 
RETR 3 
+OK 158 octets   
...The full text of message 3... 
DELE 3
+OK message # 3 deleted 
QUIT 
+OK POP3 crab Server exiting (0 NEWMAIL messages left) Connection closed by foreign host.

The USER command provides the username, and the PASS command provides the password for the account of the mailbox that is being retrieved. (This is the same username and password the user would use to log into the mail server.) In response to the STAT command, the server sends a count of the number of messages in the mailbox and the total number of bytes contained in those messages. In the example, there are three messages that contain a total of 459 bytes. RETR 1 retrieves the full text of the first message. DELE 1 deletes that message from the server. Each message is retrieved and deleted in turn. The client ends the session with the QUIT command. Simple! Table 3-2 lists the full set of POP3 commands.

Table 3-2. POP3 commands

Command	Function
USER username	The user's account name
PASS password	The user's password
STAT	Display the number of unread messages/bytes
RETR n	Retrieve message number n
DELE n	Delete message number n
LAST	Display the number of the last message accessed
LIST [n]	Display the size of message n or of all messages
RSET	Undelete all messages; reset message number to 1
TOP n l	Print the headers and l lines of message n
NOOP	Do nothing
QUIT	End the POP3 session

The retrieve (RETR) and delete (DELE) commands use message numbers that allow messages to be processed in any order. Additionally, there is no direct link between retrieving a message and deleting it. It is possible to delete a message that has never been read or to retain a message even after it has been read. However, POP clients do not normally take advantage of these possibilities. On an average POP server, the entire contents of the mailbox are moved to the client and either deleted from the server or retained as if never read. Deletion of individual messages on the client is not reflected on the server because all of the messages are treated as a single unit that is either deleted or retained after the initial transfer of data to the client. Email clients that want to remotely maintain a mailbox on the server are more likely to use IMAP.

3.4.3. Internet Message Access Protocol

Internet Message Access Protocol (IMAP) is an alternative to POP. It provides the same basic service as POP and adds features to support mailbox synchronization, which is the ability to read individual mail messages on a client or directly on the server while keeping the mailboxes on both systems completely up to date. IMAP provides the ability to manipulate individual messages on the client or the server and to have those changes reflected in the mailboxes of both systems.

IMAP uses TCP for reliable, sequenced data delivery. The IMAP port is TCP port 143.[25] Like the POP protocol, IMAP is also a request/response protocol with a small set of commands. The IMAP command set is somewhat more complex than the one used by POP because IMAP does more, yet there are still fewer than 25 IMAP commands. Table 3-3 lists the basic set of IMAP commands as defined in RFC 2060, Internet Message Access Protocol - Version 4rev1.

[25]The /etc/services file lists two different ports for IMAP: 143 and 220. Port 220 is used by IMAP 3. IMAP 4 uses port number 143, which is the same port used by IMAP 2

Table 3-3. IMAP4 commands

Command	Function
CAPABILITY	List the features supported by the server
NOOP	Literally "No Operation"
LOGOUT	Close the connection
AUTHENTICATE	Request an alternate authentication method
LOGIN	Provide the username and password for plain-text authentication
SELECT	Open a mailbox
EXAMINE	Open a mailbox as read-only
CREATE	Create a new mailbox
DELETE	Remove a mailbox
RENAME	Change the name of a mailbox
SUBSCRIBE	Add a mailbox to the list of active mailboxes
UNSUBSCRIBE	Delete a mailbox name from the list of active mailboxes
LIST	Display the requested mailbox names from the set of all mailbox names
LSUB	Display the requested mailbox names from the set of active mailboxes
STATUS	Request the status of a mailbox
APPEND	Add a message to the end of the specified mailbox
CHECK	Force a checkpoint of the current mailbox
CLOSE	Close the mailbox and remove all messages marked for deletion
EXPUNGE	Remove from the current mailbox all messages marked for deletion
SEARCH	Display all messages in the mailbox that match the specified search criterion
FETCH	Retrieve a message from the mailbox
STORE	Modify a message in the mailbox
COPY	Copy the specified messages to the end of the specified mailbox
UID	Locate a message based on the message's unique identifier

This command set clearly illustrates the "mailbox" orientation of IMAP. The protocol is designed to remotely maintain mailboxes that are stored on the server. The protocol commands show that. Despite the increased complexity of the protocol, it is still possible to run a simple test of your IMAP server using telnet and a small number of the IMAP commands.

$ telnet localhost 143
Trying 127.0.0.1...
Connected to rodent.wrotethebook.com.
Escape character is '^]'.
* OK rodent.wrotethebook.com IMAP4rev1 v12.252 server ready
a0001 LOGIN craig Wats?Watt?
a0001 OK LOGIN completed
a0002 SELECT inbox
* 3 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 965125671] UID validity status
* OK [UIDNEXT 5] Predicted next UID
* FLAGS (\Answered \Flagged \Deleted \Draft \Seen)
* OK [PERMANENTFLAGS (\* \Answered \Flagged \Deleted \Draft \Seen)] Permanent flags
* OK [UNSEEN 1] first unseen message in /var/spool/mail/craig
a0002 OK [READ-WRITE] SELECT completed 
a0003 FETCH 1 BODY[TEXT]
* 1 FETCH (BODY[TEXT] {1440}
... an e-mail message that is 1440 bytes long ...
* 1 FETCH (FLAGS (\Seen))
a0003 OK FETCH completed 
a0004 STORE 1 +FLAGS \DELETED
* 1 FETCH (FLAGS (\Seen \Deleted))
a0004 OK STORE completed
a0005 CLOSE
a0005 OK CLOSE completed
a0006 LOGOUT
* BYE rodent.wrotethebook.com IMAP4rev1 server terminating connection
a0006 OK LOGOUT completed
Connection closed by foreign host.

The first three lines and the last line come from telnet; all other messages come from IMAP. The first IMAP command entered by the user is LOGIN, which provides the username and password from /etc/passwd used to authenticate this user. Notice that the command is preceded by the string A0001. This is a tag, which is a unique identifier generated by the client for each command. Every command must start with a tag. When you manually type in commands for a test, you are the source of the tags.

IMAP is a mailbox-oriented protocol. The SELECT command selects the mailbox that will be used. In the example, the user selects a mailbox named "inbox". The IMAP server displays the status of the mailbox, which contains three messages. Associated with each message are a number of flags. The flags are used to manage the messages in the mailbox by marking them as Seen, Unseen, Deleted, and so on.

The FETCH command downloads a message from the mailbox. In the example, the user downloads the text of the message, which is what you normally see when reading a message. It is possible, however, to download only the headers or flags.

After the message is downloaded, the user deletes it. This is done by writing the Deleted flag with the STORE command. The DELETE command is not used to delete messages; it deletes entire mailboxes. Individual messages are marked for deletion by setting the Delete flag. Messages with the Delete flag set are not deleted until either the EXPUNGE command is issued or the mailbox is explicitly closed with the CLOSE command, as is done in the example. The session is then terminated with the LOGOUT command.

Clearly, the IMAP protocol is more complex than POP; it is just about at the limits of what can reasonably be typed in manually. Of course, you don't really enter these commands manually. The desktop system and the server exchange them automatically. They are shown here only to give you a sense of the IMAP protocol. About the only IMAP test you would ever do manually is to test if imapd is up and running. To do that, you don't even need to log in; if the server answers the telnet, you know it is up and running. All you then need to do is send the LOGOUT command to gracefully close the connection.

3.4.4. Multipurpose Internet Mail Extensions

The last email protocol on our quick tour is Multipurpose Internet Mail Extensions (MIME).[26] As its name implies, MIME is an extension of the existing TCP/IP mail system, not a replacement for it. MIME is more concerned with what the mail system delivers than with the mechanics of delivery. It doesn't attempt to replace SMTP or TCP; it extends the definition of what constitutes "mail."

[26] MIME is also an integral part of the Web and HTTP.

The structure of the mail message carried by SMTP is defined in RFC 822, Standard for the Format of ARPA Internet Text essages. RFC 822 defines a set of mail headers that are so widely accepted they are used by many mail systems that do not use SMTP. This is a great benefit to email because it provides a common ground for mail translation and delivery through gateways to different mail networks. MIME extends RFC 822 into two areas not covered by the original RFC:

Support for various data types. The mail system defined by RFC 821 and RFC 822 transfers only 7-bit ASCII data. This is suitable for carrying text data composed of U.S. ASCII characters, but it does not support several languages that have richer character sets, nor does it support binary data transfer.
Support for complex message bodies. RFC 822 doesn't provide a detailed description of the body of an electronic message. It concentrates on the mail headers.

MIME addresses these two weaknesses by defining encoding techniques for carrying various forms of data and by defining a structure for the message body that allows multiple objects to be carried in a single message. RFC 1521, Multipurpose Internet Mail Extensions Part One: Format of Internet Message Bodies, defines two headers that give structure to the mail message body and allow it to carry various forms of data. These are the Content-Type header and the Content-Transfer-Encoding header.

As the name implies, the Content-Type header defines the type of data being carried in the message. The header has a Subtype field that refines the definition. Many subtypes have been defined since the original RFC was released. A current list of MIME types can be obtained from the Internet.[27] The original RFC defines seven initial content types and a few subtypes:

[27] Go to ftp://ftp.isi.edu/in-notes/iana/assignments/media-types to retrieve the file media-types.

text: Text data. RFC 1521 defines text subtypes plain and richtext. More than 30 subtypes have since been added, including enriched, xml and html.
application: Binary data. The primary subtype defined in RFC 1521 is octet-stream, which indicates the data is a stream of 8-bit binary bytes. One other subtype, PostScript, is defined in the standard. Since then more than 200 subtypes have been defined. They specify binary data formatted for a particular application. For example, msword is an application subtype.
image: Still graphic images. Two subtypes are defined in RFC 1521: jpeg and gif. More than 20 additional subtypes have since been added, including widely used image data standards such as tiff, cgm, and g3fax.
video: Moving graphic images. The initially defined subtype was mpeg, which is a widely used standard for computer video data. A few others have since been added, including quicktime.
audio: Audio data. The only subtype initially defined for audio was basic, which means the sounds are encoded using pulse code modulation (PCM). About 20 additional audio types, such as MP4A-LATM, have since been added.
multipart: Data composed of multiple independent sections. A multipart message body is made up of several independent parts. RFC 1521 defines four subtypes. The primary subtype is mixed, which means that each part of the message can be data of any content type. Other subtypes are alternative, meaning that the same data is repeated in each section in different formats; parallel, meaning that the data in the various parts is to be viewed simultaneously; and digest, meaning that each section is data of the type message. Several subtypes have since been added, including support for voice messages (voice-message) and encrypted messages.
message: Data that is an encapsulated mail message. RFC 1521 defines three subtypes. The primary subtype, rfc822, indicates that the data is a complete RFC 822 mail message. The other subtypes, partial and External-body, are both designed to handle large messages. partial allows large encapsulated messages to be split among multiple MIME messages. External-body points to an external source for the contents of a large message body so that only the pointer, not the message itself, is contained in the MIME message. Two additional subtypes that have been defined are news for carrying network news and http for HTTP traffic formatted to comply with MIME content typing.

The Content-Transfer-Encoding header identifies the type of encoding used on the data. Traditional SMTP systems forward only 7-bit ASCII data with a line length of less than 1000 bytes. Since the data from a MIME system may be forwarded through gateways that support only 7-bit ASCII, the data can be encoded. RFC 1521 defines six types of encoding. Some types are used to identify the encoding inherent in the data. Only two types are actual encoding techniques defined in the RFC. The six encoding types are:

7bit

Figure 3-4. base64 encoding

The number of supported data types and encoding techniques grows as new data formats appear and are used in message transmissions. New RFCs constantly define new data types and encoding. Read the latest RFCs to keep up with MIME developments.

MIME defines data types that SMTP was not designed to carry. To handle these and other future requirements, RFC 1869, SMTP Service Extensions, defines a technique for making SMTP extensible. The RFC does not define new services for SMTP; in fact, the only service extensions mentioned in the RFC are defined in other RFCs. What this RFC does define is a simple mechanism for systems to negotiate which SMTP extensions are supported. The RFC defines a new hello command (EHLO) and the legal responses to that command. One response is for the receiving system to return a list of the SMTP extensions it supports. This response allows the sending system to know what extended services can be used, and to avoid those that are not implemented on the remote system. SMTP implementations that support the EHLO command are called Extended SMTP (ESMTP).

Several ESMTP service extensions have been defined for MIME mailers. Table 3-4 lists some of these. The table lists the EHLO keyword associated with each extension, the number of the RFC that defines it, and its purpose. These service extensions are just an example. Other have been defined to support SMTP enhancements.

Table 3-4. SMTP service extensions

Keyword	RFC	Function
8BITMIME	1652	Accept 8bit binary data
CHUNKING	1830	Accept messages cut into chunks
CHECKPOINT	1845	Checkpoint/restart mail transactions
PIPELINING	1854	Accept multiple commands in a single send
SIZE	1870	Display maximum acceptable message size
DSN	1891	Provide delivery status notifications
ETRN	1985	Accept remote queue processing requests
ENHANCEDSTATUSCODES	2034	Provide enhanced error codes
STARTTLS	2487	Use Transport Layer Security to encrypt the email exchange
AUTH	2554	Use strong authentication to identify the email source

It is easy to check which extensions are supported by your server by using the EHLO command. The following example is from a generic Solaris 8 system, which comes with sendmail 8.9.3:

> telnet localhost 25 
Trying 127.0.0.1... 
Connected to localhost. 
Escape character is '^]'. 
220 crab.wrotethebook.com ESMTP Sendmail 8.9.3+Sun/8.9.3; Mon, 23 Apr 2001 11:00:35-0400 (EDT)
EHLO crab 
250-crab.wrotethebook.com Hello localhost [127.0.0.1], pleased to meet you
250-EXPN 
250 HELP 
250-8BITMIME 
250-SIZE 
250-DSN 
250-ETRN 
250-VERB 
250-ONEX 
250-XUSR
QUIT 
221 crab.foobirds.org closing connection
Connection closed by foreign host.

The sample system lists nine commands in response to the EHLO greeting. Two of these, EXPN and HELP, are standard SMTP commands that aren't implemented on all systems (the standard commands are listed in Table 3-1). 8BITMIME, SIZE, DSN, and ETRN are ESMTP extensions, all of which are described in Table 3-4. The last three keywords in the response are VERB, ONEX, and XUSR. All of these are specific to sendmail version 8. None is defined in an RFC. VERB simply places the sendmail server in verbose mode. ONEX limits the session to a single message transaction. XUSR is equivalent to the -U sendmail command-line argument.[28] As the last three keywords indicate, the RFCs allow for private ESMTP extensions.

[28] See Appendix E, "A sendmail Reference" for a list of the sendmail command-line arguments.

The specific extensions implemented on each system are different. For example, on a generic Solaris 2.5.1 system, only three keywords (EXPN, SIZE, and HELP) are displayed in response to EHLO. The extensions available depend on the version of sendmail that is running and on how sendmail is configured.[29] The purpose of EHLO is to identify these differences at the beginning of the SMTP mail exchange.

[29] See Chapter 10, "sendmail " for the details of sendmail configuration.

ESMTP and MIME are important because they provide a standard way to transfer non-ASCII data through email. Users share lots of application-specific data that is not 7-bit ASCII. Many users depend on email as a file transfer mechanism.

SMTP, POP, IMAP, and MIME are essential parts of the mail system, but other email protocols may also be essential in the future. The one certainty is that the network will continue to change. You need to track current developments and include helpful technologies in your planning. Two technologies that users find helpful are file sharing and printer sharing. In the next section we look at file and print servers.