Introduction To Email

Master on Free Software

Introduction to InternetEMail

Alberto Garcia Gonzalez


What is E-Mail?

● Method for sending messages over the Internet● First defined in RFC 822 (1982)● Expanded in RFC 2822 and some others

● http://tools.ietf.org/html/rfc822● http://tools.ietf.org/html/rfc2822

● An email message consists of two parts:● Header● Body

● EMail is usually sent with the SMTP protocol.

http://tools.ietf.org/html/rfc822

http://tools.ietf.org/html/rfc2822


E-Mail header

● Consists of a number of header lines● Each line has the format “Name: value”● They contain additional data about a message● Headers are 7bit ASCII characters● The MIME standard allows us to represent other non

ASCII characters● There are many types of headers, some of them

standard and others added by any client.


Some standard headers

● From● To● Subject● Date● MessageID● Nonstandard headers begin with "X"● Complete list here: http://www.iana.org/assignments/message-headers/perm-headers.html

http://www.iana.org/assignments/message-headers/perm-headers.html


Message body

● Contains the text of the message.● It can also contain attached files (if any)● The body was originally ASCII only● Now it can contain 8bit data.● The body appears after the headers separated by a

blank line


Sample message

To: [email protected]: TestMessage-Id: <[email protected]>From: Alberto Garcia Gonzalez <[email protected]>Date: Wed, 26 Mar 2008 23:51:00 +0100

This is a test


How to store e-mails in disk

● There is no real standard: each email program can store messages using any method.

● Some email clients have their own systems of storing emails.

● However, there are several widespread methods.● Some of the most popular are:

mbox maildir


The mbox format

● Very simple method for storing email.● All messages are stored in a single file, separated by

blank lines● A line starting with "From " appears at the beginnig of

each message, before the headers.● Each mbox file represents a mail folder.● Very simple to use, but not very efficient.● A damaged message can corrupt the whole folder.● A locking mechanism is needed so several programs

can access the same mbox file.


Sample mbox file

From [email protected] Tue Mar 25 18:11:12 2008To: [email protected]: TestMessageId: <E1Jf0w00005zG[email protected]>From: Alberto Garcia Gonzalez <[email protected]>Date: Tue, 25 Mar 2008 18:11:10 +0100

This is a test

From [email protected] Wed Mar 26 23:51:02 2008MessageId: <E1Jf15700067e[email protected]>From: Alberto Garcia Gonzalez <[email protected]>Date: Wed, 26 Mar 2008 23:51:00 +0100To: [email protected]: Another test

This is another test


The maildir format

● A simple yet robust method for storing email● Each message is stored in a different file● Several programs can access the same folder without

needing locking mechanisms● Each mail folder needs a separate directory with three

subdirectories in it (cur, new, tmp)● Example:maildir/cur/maildir/new/maildir/new/1206660208.26880_1.deimosmaildir/new/1206660208.26880_3.deimosmaildir/tmp/


The mailx e-mail client

● mailx is a simple, textbased, email client● It is a standard Unix tool, but not very friendly● Useful to quickly send emails:$ mail [email protected]: This is a testTest.Cc:● It can also be used to read mbox files:$ mail -f mboxMail version 8.1.2 01/15/2001. Type ? for help."mbox": 2 messages 2 new>N 1 [email protected] Tue Mar 25 18:11 10/259 Test N 2 [email protected] Wed Mar 26 23:51 10/273 Another test

mailto:[email protected]


The mutt e-mail client

● A powerful text email client● It has lots of options and supports many protocols.● Small and fully configurable● It reads mbox and maildir among others.$ mutt -f mbox$ mutt -f maildir/$ mutt -f pop://pop.server.com/$ mutt -f imap://imap.server.com/

imap://imap.server.com/


MIME

● “Multipurpose Internet Mail Extensions”● Extensions to the email standard to support things

such as: Attachments NonASCII text

● It is now used in other protocols besides email (e.g HTTP)

● Defined in several different RFCs


MIME type

● Indicates the format of a file or document.● Consists of a type and a subtype.● Examples:

● text/plain● text/html● audio/mpeg● image/png

● Nonstandard mime types begin with “x”:● audio/xwav● application/xshockwaveflash


Character sets

● Code that maps a character to a byte or sequence of bytes.

● ASCII is the most widely spread. It encodes some letters, symbols and control characters to 7bit numbers (0127)

● Other popular character sets: ISO88591 (Latin1), GB2312, UTF8

● Unicode: standard that tries to define a list of all known characters. It is a superset of all charsets mentioned above


Converting between charsets

● iconv and recode are tools that convert text between different character sets.

● iconv is more widespread, but recode has more features.

● Example: how to convert a file from latin1 to utf8 using both tools

$ cat file | iconv -f latin1 -t utf8$ cat file | recode latin1..utf8


The Content-Type header

● Used to indicate the MIME type of a message or attachment:● Content-Type: text/plain● Content-Type: image/png

● The character set can be specified as well:● Content-Type: text/plain; charset=iso-8859-1

● If not specified, the default is text/plain and ASCII charset


Content-Transfer-Encoding

● The basic version of SMTP protocol only supports ASCII for sending emails

● So messages with nonASCII characters must be encoded in some way to be sent via SMTP.

● Note that this is different from the character set explained previously.

● The ContentTransferEncoding header indicates the encoding used.


Encoding types

● ASCIIbased encodings:● 7bit: basic ASCII text● quotedprintable: used when the text is mostly ASCII● base64: used when the data is mostly nonASCII

● Binary encodings supported by some SMTP servers:● 8bit● binary


● The recode tool can convert between the encodings used in EMail

● Example: convert a UTF8 text into Latin1 and encode it using quotedprintable:

$ echo camión | recode utf8..latin1/qp● Example: reencode a text from quotedprintable into

base64 without changing its charset:$ echo cami=F3n | recode /qp../base64

Converting between encodings


Sample UTF-8 e-mail

Date: Wed, 26 Mar 2008 01:12:48 +0100From: Alberto Garcia <[email protected]>To: [email protected]: TestMessage-ID: <[email protected]>MIME-Version: 1.0Content-Type: text/plain; charset=utf-8Content-Disposition: inlineContent-Transfer-Encoding: quoted-printable

Cami=C3=B3n


Encoded-Word

● Sometimes it's necessary to included nonASCII text in headers

● Subject, From and To are the most typical examples● The EncodedWord format is used to encode the text.

It specifies both the charset and the encoding.● The encoding can be 'Q' (quotedprintable) or

'B' (base64)● Format: =?<charset>?<enc>?<text>?=● Example:Subject: =?utf-8?Q?Sa=C3=BAde!?=


Multipart● The body of the message can contain several different

parts.● Examples: an attachment, the text version of an HTML

message● The multipart/* MIME types are used for this:

● multipart/mixed: A typical message with several attachments

● multipart/alternative: Each part contains an alternative version of the same content. Typically text and HTML

● There are more types: multipart/signed, multipart/encrypted, ...


Sample multipart messageDate: Wed, 26 Mar 2008 17:32:50 +0100From: Alberto Garcia <[email protected]>To: [email protected]: Multipart messageMessage-ID: <[email protected]>MIME-Version: 1.0Content-Type: multipart/mixed; boundary="dDRMvlgZJXvWKvBx"Content-Disposition: inline

--dDRMvlgZJXvWKvBxContent-Type: text/plain; charset=us-asciiContent-Disposition: inline

This is the first part

--dDRMvlgZJXvWKvBxContent-Type: text/plain; charset=us-asciiContent-Disposition: attachment; filename="part2.txt"

This is the second part

--dDRMvlgZJXvWKvBx--


The SMTP protocol

● Simple Mail Transfer Protocol● The most widely used protocol to send email via

Internet● Defined in RFC 821 (SMTP) and extended in RFC

2821● A simple textbased protocol (the basic version only

allows ASCII)● It uses a TCP connection, and port 25 by default


SMTP commands: HELO

● Used to start a SMTP connection● Identifies the calling host and initiates the session.● Example:$ telnet gmail-smtp-in.l.google.com 25Trying 72.14.221.27...Connected to gmail-smtp-in.l.google.com.Escape character is '^]'.220 mx.google.com ESMTP h7si386132nfh.19HELO mail.local250 mx.google.com at your service


SMTP commands:MAIL and RCPT

● Used to indicate the email sender and recipient(s) of a message.

● Example:MAIL FROM:<[email protected]>250 2.1.0 OKRCPT TO:<[email protected]>250 2.1.0 OKRCPT TO:<[email protected]>250 2.1.0 OK


SMTP: envelope headers

● The sender and recipient set during the SMTP connection are called the envelope headers.

● They are not necessarily related to the message's From: and To: headers (although they usually are the same).

● For example, it is possible to send and email to an address different from the one in the message's header.


SMTP commands: DATA

● Used to send the raw contents of the email (both headers and body). Must finish with a line containing a single dot ('.').

● Example:DATA354 Go aheadFrom: Alberto <[email protected]>To: John <[email protected]>Subject: This is a test

This is a test.250 2.0.0 OK 1206634142 h7si386132nfh.19


SMTP commands: QUIT

● Finishes a connection. Tipically used after the message has been sent (or rejected).

● Example:QUIT221 2.0.0 mx.google.com closing connection h7si386132nfh.19

● And now we've seen a complete SMTP session.


ESMTP: Extended SMTP

● An extension to the SMTP protocol● Introduces new commands: authentication,

cryptography, ...● Supported by most modern SMTP servers.● Indicated by the word ESMTP in the initial banner:220 mx.google.com ESMTP b33si1239227ika.5


ESMTP commands: EHLO

● Stands for Extended HELO● If the client sends this command it means that it

understands ESMTP.● The server replies with the list of extensions that is

supports:EHLO mail.local250-mx.google.com at your service, [83.49.12.59]250-SIZE 28311552250-8BITMIME250 ENHANCEDSTATUSCODES


ESMTP commands: AUTH

● Used to authenticate the client, providing a username and a password.

● There are several authentication methods. The SMTP server must indicate which ones it supports.

● Example:EHLO mail.local250-mx.google.com at your service, [83.49.12.59]250-SIZE 28311552250-8BITMIME250-AUTH LOGIN PLAIN250 ENHANCEDSTATUSCODES


LOGIN authentication

● Simple authentication method.● Supported by many email clients and servers.● The client sends the user and the password encoded

in base64.● Example:AUTH LOGIN334 VXNlcm5hbWU6dGVzdHVzZXI=334 UGFzc3dvcmQ6dGVzdHB3235 2.7.0 Accepted


PLAIN authentication

● Similar to LOGIN, easy to use and widespread.● The client sends the following string to the server

encoded in base64: '\0user\0password'● Example:AUTH PLAIN334AHRlc3R1c2VyAHRlc3Rwdw==235 2.7.0 Accepted


ESMTP commands: STARTTLS

● Used to encrypt the communication between client and server.

● With this, all authentication methods mentioned before become secure.

● It is a binary protocol so it cannot be easily tested via telnet ;)STARTTLS220 TLS go ahead


How clients send e-mail

● Users usually have a SMTP server (called “smarthost”) configured in their email clients.

● All the email is sent via that SMTP server, which is tipically from the ISP.

● That server is normally configured to accept email only from authorized clients (normally using their IP or authentication).

● If necessary, it sends the message to another server.


How e-mail is routed

● DNS is used to decide where a message should be sent.

● The MX record of the recipient's domain is checked.● The message is sent to all server with the lowest

priority number.


Basic DNS queries

● There's a tool called 'dig' used to make DNS queries.● Basic usage:$ dig [ +short ] [ @<dns-server> ] query [ <type> ]

● Example: getting the MX record of gmail.com$ dig +short gmail.com mx5 gmail-smtp-in.l.google.com.10 alt1.gmail-smtp-in.l.google.com.10 alt2.gmail-smtp-in.l.google.com.50 gsmtp147.google.com.50 gsmtp183.google.com.


Basic configuration of a SMTP server: Exim

● We'll test the basic configuration of the Exim SMTP server included in most GNU/Linux distributions.

$ dpkg-reconfigure exim4-config● This is the Debian configuration script.● Each SMTP server has a completely different set of

configuration options.


The POP3 protocol

● POP3 (Post Office Protocol version 3)● Very simple protocol to download emails from a

server● It's the most widespread protocol nowadays● Very few features an insecure● Goes over a TCP connection and its default port is 110


POP3 commands:USER and PASS

● Used to log in to a POP3 server● User and password are sent in cleartext● Example:$ telnet pop3-server.com pop-3Trying 192.168.1.1...Connected to pop3-server.com.Escape character is '^]'.+OKuser testuser+OKpass testpasswd+OK


POP3 commands:STAT and LIST

● Used to view the status of a mailbox● STAT gets the number of messages in the server and

their total size● LIST shows the size of each individual message● Examples:STAT+OK 3 1333LIST+OK1 4442 4443 445

.


POP3 commands: TOP

● TOP retrieves the first lines of a message● Useful if you want to view the summary of a mail box

(senders, recipients, subjects) without having to download full messages.

● Syntax: TOP <message no.> <lines>● If the number of lines is 0, it'll retrieve all the headers● Example:STAT+OK 3 1333TOP 2 0<all the headers of the second message>.


POP3 commands:RETR and DELE

● RETR gets a complete message. This is the command used when fetching all email from a POP3 account.

● DELE marks the message for deletion. It will be deleted from the server when the connection is closed.

● Examples:STAT+OK 3 1333RETR 2<the complete message>.DELE 2+OK


POP3 commands:RSET and QUIT

● RSET undeletes all messages set for deletion using DELE in the same session.

● QUIT closes the connection and removed all messages marked for deletion.

● Examples:RSET+OKQUIT+OKConnection closed by foreign host.


The IMAP protocol

● Internet Message Access Protocol.● Used to access email on a remote server.● Solves many of the problems of the POP3 protocol.● Very widespread (thought not as much as POP3).● Allows uploading email to a server and also multiple

folders in the same account.● IMAP goes over TCP using port 143 by default.


IMAP commands: tags

● All IMAP commands must be preceded by a tag.● This tag should be unique to each command.● The tag can be a string, a number, ...● Example:01 COMMAND1<response>02 COMMAND2<response>03 COMMAND3<response>


IMAP commands: LOGIN

● The LOGIN command authenticates the user and initiates the session.

● Syntax: <tag> LOGIN <user> <password>● Example:

* OK Welcome to Binc IMAP Copyright (C) 2002-2005 Andreas Aardal Hanssen at 2008-03-28 17:15:49 +01001 LOGIN user pass1 OK LOGIN completed


IMAP commands:LIST and STATUS

● The LIST command lists all available folders.2 LIST "" "*"* LIST (\Unmarked) "/" "INBOX"* LIST (\Unmarked) "/" "INBOX/Personal"2 OK LIST completed

● The STATUS command gives basic information about a folder

3 STATUS "INBOX" (MESSAGES)* STATUS "INBOX" (MESSAGES 6)3 OK STATUS completed4 STATUS "INBOX" (RECENT)* STATUS "INBOX" (RECENT 2)4 OK STATUS completed5 STATUS "INBOX" (UNSEEN)* STATUS "INBOX" (UNSEEN 5)5 OK STATUS completed


IMAP commands:SELECT and EXAMINE

● The SELECT command is used to select a folder.● We use SELECT when we want to download a

message.6 SELECT "INBOX/Personal"* 0 EXISTS* 0 RECENT* OK [UIDVALIDITY 1206721805]* OK [UIDNEXT 1] 1 is the next UID* FLAGS (\Answered \Flagged \Deleted \Recent \Seen \Draft)* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft)] Limited6 OK [READ-WRITE] SELECT completed

● EXAMINE is the readonly version of SELECT.


IMAP commands: FETCH

● Used to retrieve a message from the server.● The headers and the body can be retrieved separately:7 FETCH 1 body[header]<header of the message>8 FETCH 1 body[text]<body of the message>9 FETCH 1 body[]<full message>

● Sequences of messages are supported as well:10 FETCH 1:* body[header]<header of all selected messages>


IMAP commands:CREATE, DELETE, RENAME

● CREATE and DELETE are used to create and delete folders.

● RENAME is used to rename a folder.11 CREATE INBOX/Test11 OK CREATE completed12 RENAME INBOX/Test INBOX/New12 OK RENAME completed13 DELETE INBOX/New13 OK DELETE completed


IMAP commands: STORE● STORE is used to add and remove flags from a

message.● There are many types of flags, only some of them are

standard: \Seen, \Deleted, \Recent, ...● Example:

14 STORE 1 +flags \Deleted* 1 FETCH (FLAGS (Junk \Seen \Deleted))14 OK STORE completed15 STORE 1 -flags \Deleted* 1 FETCH (FLAGS (Junk \Seen))15 OK STORE completed16 STORE 1 flags \Deleted* 1 FETCH (FLAGS (\Deleted))16 OK STORE completed


IMAP commands:EXPUNGE and CLOSE

● EXPUNGE removes all messages marked as deleted in the current folder.

● CLOSE does the same but also unselects the folder.17 STORE 1 flags \Deleted* 1 FETCH (FLAGS (\Deleted))17 OK STORE completed18 EXPUNGE* 1 EXPUNGE* 4 EXISTS18 OK EXPUNGE completed19 CLOSE19 OK CLOSE completed


IMAP commands: COPY

● COPY is used to copy messages to another folder● There is no “Move” operation in IMAP: you just copy and

then delete the original messages.● Example:

20 CREATE INBOX/Test20 OK CREATE completed21 COPY 1:* INBOX/Test21 OK COPY completed22 STATUS INBOX/Test (MESSAGES)* STATUS “INBOX/Test” (MESSAGES 3)22 OK STATUS completed


IMAP commands: APPEND

● Used to add a message to a folder● Receives a folder, a list of flags and the size of the

message23 APPEND INBOX () {87}+ go ahead with 87 charactersFrom: [email protected]: [email protected]: Appending an e-mail

Yeah!23 OK APPEND completed


IMAP commands: LOGOUT

● This command closes a session and disconnects from the server

● Example:100 LOGOUT100 OK LOGOUT completed


Mailing lists

● System to send emails to multiple addresses.● Usually automated using dedicated software.● Some mailing lists managers:

● Majordomo● Smartlist● GNU Mailman


Smartlist

● Very small list manager.● Few requirements but not very friendly.● To test it in the QEMU image add these lines to /etc/exim4/exim4.conf.localmacros:

SYSTEM_ALIASES_PIPE_TRANSPORT=”address_pipe”SYSTEM_ALIASES_USER=”list”

● Then restart Exim:/etc/init.d/exim4 restart


Smarlist:Creating and deleting lists

● Go to /var/list and run this command:$ .bin/createlist listname

● Add the list aliases to /etc/aliases

● To delete a list:$ .bin/removelist listname

● Remove the aliases from /etc/aliases

● Some configuration options are available in the file /var/list/listname/rc.custom


Smartlist: subscribing and unsubscribing

● To subscribe, send an email to the address listname-request@domain with the word 'subscribe' in the subject.

● To unsubscribe, send an email to the same address with the word 'unsubscribe'.

● In both cases you will receive a confirmation message.● You can also edit /var/list/listname/dist by hand


GNU Mailman

● GNU Mailman is a free mailing list manager.● It has a webbased configuration interface.● Allows users to change their settings.● Creates a web archive with all messages.● Includes bounce handling and spam prevention.● Very easy to set up and use.


Installing GNU Mailman

● Run QEMU with the '-redir tcp:8080::80' option● Install the Debian package 'mailman'● Edit /etc/apache2/apache2.conf and addInclude /etc/mailman/apache.conf

● Restart apache (/etc/init.d/apache2 restart)● Edit /etc/mailman/mm_cfg.py, change '%s' for '%s:8080' in DEFAULT_URL_PATTERN and set DEFAULT_URL_HOST to 'localhost'

● Run 'newlist mailman'.● Run 'mmsitepass' and set a global password.● Start mailman: /etc/init.d/mailman start


● Run 'newlist listname' to create a new list● Open this URL in your browser:http://localhost:8080/cgi-bin/mailman/listinfo

● You'll see the existing mailing lists.● Select one and you'll go the list page. From there you

can subscribe, edit options, etc.● Now go to the list's admin interface to configure the

list.● To remove a list: 'rmlist [-a] listname'

Mailman: managing lists


E-Mail spam

● Spam is email that meets the following conditions:● It's sent massively.● It isn't authorized by the recipient.● It's independent of the recipient.

● The overwhelming (>80%) majority of the email sent every day is spam.

● One of the reasons is that it's cheap and the SMTP protocol is too unrestrictive.


Some anti-spam techniques

● Relay control● SMTPAUTH● POPBeforeSMTP● DNSBased Blackhole Lists (DNSBL)● Heuristics (example: SpamAssassin)● Bayesian filter (example: DSPAM)● Greylisting


Heuristic filters:how they work

● They identify features that appear only in spam messages:● Subject all uppercase● Links to external pages● HTMLonly body● Names of medicines, etc.


Heuristic filters:benefits and drawbacks

● Works OK with many spam messages.● The set of rules must be updated constantly.● The spammer can easily adapt their messages to fool the heuristics.

● False positives can appear.


Bayesian filters:how they work

● The antispam filter is trained with both legitimate email and spam.

● It computes statistics about the words and combinations of words that appear in both groups of messages.

● Each new message is matched against these rules to detect whether it's spam or not.


Bayesian filters:benefits and drawbacks

● Very accurate spam identification.● The number of false positives is very low.● It adapts well to new types of spam.● The user must train the system.● It's necesssary to feed the filter with several messages of each type for it to start working properly.


Greylisting:how it works

● The first time a message arrives to a server, it is rejected with a temporary error (“Server unavailable, try again later”). The IP address, sender and recipiend are recorded in a greylist.

● If the message appears again after some minutes, it is accepted and sent.

● If not, the server forgets about it after some hours and deletes the info from the greylist.


Greylisting:benefits and drawbacks

● Independent of the contents of the message.● Very effective: it's too costly for the spammer to send

again every failed message.● It forces the spammer to retain the same IP address

so it's easier to detect.● Very easy to configure and maintain.● Big drawback: it can delay (or even prevent) the

delivery of legitimate email.

Introduction To Email

Technology

Transcript of Introduction To Email