Edit Rename Changes History Upload Download Back to Top

Email's Exceedingly Newbie Notes

Notes written by Barry as he begins the journey of understanding what came before.

Daemon

That would be Exim4.

Ok - we're on version 4.50. That seems okay. Further, crab is set to not upgrade this to a later version. The latest release is 4.62 (perhaps not in Sarge), but the warnings at http://www.exim.org/ only say to avoid verions 4.43 and prior for security reasons. As of this writing Ubuntu Dapper has version 4.60.

Configuration

Well, consider /etc/init.d/exim4. It always runs update-exim4.conf before it starts the daemon.

update-exim4.conf is a script which takes its input from /etc/exim4/update-exim4.conf.conf. That little text file can be configured by hand or by the

dpkg-reconfigure exim4-config
process. It makes no difference.

The main thing in my mind about this file is the parameter in there which says whether or not to use a split configuration.

If you pick unsplit then update-exim4.conf will take what it finds in /etc/exim4/exim4.conf.template, do some variable substitution based on other assignments in update-exim4.conf.conf, and generate /var/lib/exim4/config.autogenerated.

If you pick split then the script uses the directories under /etc/exim4/conf.d instead of the template file to buid the autogenerated file. The directories are processed in the order main, acl, router, transport, retry, rewrite, auth. Then within these directories the files found are processed in lexical order. Only files with letters, number, underscores, and hyphens in their names are considered. So files replaced by debian package management with ".dpkg-old" at the end are ignored.

The split files are just concatenated together to form the autogenerated file (of course the variable substitution happens).

When Exim starts it will either take /etc/exim4/exim4.config if it exists, or the autogenerated file. So if you have an exim4.config file all this generated stuff is irrelevant.

A thorough understanding of the exim configuration structure can be gotten by reading

/usr/share/doc/exim4/README.Debian.gz
See the section entitled "What are these gazillion number of files in /etc/exim4/conf.d? How do I get rid of them? How does the configuration work? What kind of crack are you smoking?"

Further enlightenment comes from the man page for update-exim4.conf.

Now the package exim4-config includes /etc/exim4/exim4.conf.template and some structure under /etc/exim4/conf.d which likely produces the same config.autogenerated file. So you get the same stock behavior whether you pick split or unsplit.

Apparently update-exim4.conf.template will generate a new template file if the conf.d structure is modified. This might be useful if you decide to go with unsplit but some other package (say an anti-virus package) pops some file into the conf.d structure and you'd like to effortlessly incorporate it's configuration into your unsplit template file.

Deviation

So as of now how does our exim conf.d directory differ from the stock install provided by exim4-config?

Here are the files that are not from the original install of exim4-config:

acl/30_exim4-config_check_rcpt.dpkg-old
auth/30_exim4-config_examples.dpkg-old
main/02_exim4-config_options.dpkg-old

main/000_localmacros
main/04_mailman_config
main/05_postgres_config
main/15_sa-exim_plugin_path
main/20_clamav-daemon_socket_path

router/350_mailman_router
router/450_mms_user

transport/30_mailman_transport

Well except for the first three, these files are very exciting. They are what allow (or should allow) fancy integration of exim4 with other stuff (like mailman, spam assassin, postgres, clamav, and mms). Can't wait to delve further.

The first three, on the other hand, are worrying. They suggest that configuration files were administratively modified prior to a package upgrade. During the package upgrade, new configuration files were installed and the old modified ones set aside with the "dpkg-old" suffix. What is worrying is that these files were not subsequently deleted.

An easy example is the auth one. It seems that in the "cram_md5_server" was uncommented. The new version of the file (currently being used) also uncomments the "plain_server" and "login_server" sections. Anand reports that this change was required to allow relaying services to Daniel Noll.

The other two files are more radically altered.

Using dpkg-deb I extracted the files from the current archive of exim4-config. I then did a simple diff of the new versions of the three files in question against the stock versions. It turns out that all three are changed. So my assumption now is that the changes have been applied but the "dpkg-old" files have inadvertantly been left hanging around.

Further, a comparison of all the md5sums from the stock package revealed that

acl/40_exim4-config_check_data
has also been modified.

Diffs

main/02_exim4-config_options

73a74,75
> # following line added by patrick@lesslie.ath.cx to stop long timeouts
> rfc1413_hosts =

acl/30_exim4-config_check_rcpt

9a10,52
>   # greylistd(8) configuration follows.
>   # This statement has been added by "greylistd-setup-exim4",
>   # and can be removed by running "greylistd-setup-exim4 remove".
>   # Any changes you make here will then be lost.
>   # 
>   # Perform greylisting on incoming messages from remote hosts.
>   # We do NOT greylist messages with no envelope sender, because that
>   # would conflict with remote hosts doing callback verifications, and we
>   # might not be able to send mail to such hosts for a while (until the
>   # callback attempt is no longer greylisted, and then some).
>   #
>   # We also check the local whitelist to avoid greylisting mail from
>   # hosts that are expected to forward mail here (such as backup MX hosts,
>   # list servers, etc).
>   #
>   # Because the recipient address has not yet been verified, we do so
>   # now and skip this statement for non-existing recipients.  This is
>   # in order to allow for a 550 (reject) response below.  If the delivery
>   # happens over a remote transport (such as "smtp"), recipient callout
>   # verification is performed, with the original sender intact.
>   #
>   defer
>     message        = $sender_host_address is not yet authorized to deliver \
>                      mail from <$sender_address> to <$local_part@$domain>. \
>                      Please try later.
>     log_message    = greylisted.
>     !senders       = :
>     !hosts         = : +relay_from_hosts : \
>                      ${if exists {/etc/greylistd/whitelist-hosts}\
>                                  {/etc/greylistd/whitelist-hosts}{}} : \
>                      ${if exists {/var/lib/greylistd/whitelist-hosts}\
>                                  {/var/lib/greylistd/whitelist-hosts}{}}
>     !authenticated = *
>     !acl           = acl_whitelist_local_deny
>     domains        = +local_domains : +relay_to_domains
>     verify         = recipient/callout=20s,use_sender,defer_ok
>     condition      = ${readsocket{/var/run/greylistd/socket}\
>                                  {--grey \
>                                   ${mask:$sender_host_address/24} \
>                                   $sender_address \
>                                   $local_part@$domain}\
>                                  {5s}{}{false}}
> 

acl/40_exim4-config_check_data

5a6,41
>   # greylistd(8) configuration follows.
>   # This statement has been added by "greylistd-setup-exim4",
>   # and can be removed by running "greylistd-setup-exim4 remove".
>   # Any changes you make here will then be lost.
>   # 
>   # Perform greylisting on incoming messages with no envelope sender here.
>   # We did not subject these to greylisting after RCPT TO:, because that
>   # would interfere with remote hosts doing sender callout verifications.
>   #
>   # Because there is no sender address, we supply only two data items:
>   #  - The remote host address
>   #  - The recipient address (normally, bounces have only one recipient)
>   #
>   # We also check the local whitelist to avoid greylisting mail from
>   # hosts that are expected to forward mail here (such as backup MX hosts,
>   # list servers, etc).
>   #
>   defer
>     message        = $sender_host_address is not yet authorized to deliver \
>                      mail from <$sender_address> to <$recipients>. \
>                      Please try later.
>     log_message    = greylisted.
>     senders        = :
>     !hosts         = : +relay_from_hosts : \
>                      ${if exists {/etc/greylistd/whitelist-hosts}\
>                                  {/etc/greylistd/whitelist-hosts}{}} : \
>                      ${if exists {/var/lib/greylistd/whitelist-hosts}\
>                                  {/var/lib/greylistd/whitelist-hosts}{}}
>     !authenticated = *
>     !acl           = acl_whitelist_local_deny
>     condition      = ${readsocket{/var/run/greylistd/socket}\
>                                  {--grey \
>                                   ${mask:$sender_host_address/24} \
>                                   $recipients}\
>                                   {5s}{}{false}}
> 

auth/30_exim4-config_examples

26,50c26,50
< # plain_server:
< #   driver = plaintext
< #   public_name = PLAIN
< #   server_condition = "${if crypteq{$3}{${extract{1}{:}{${lookup{$2}lsearch{CONFDIR/passwd}{$value}{*:*}}}}}{1}{0}}"
< #   server_set_id = $2
< #   server_prompts = :
< #   .ifndef AUTH_SERVER_ALLOW_NOTLS_PASSWORDS
< #   server_advertise_condition = ${if eq{$tls_cipher}{}{}{*}}
< #   .endif
< #
< # login_server:
< #   driver = plaintext
< #   public_name = LOGIN
< #   server_prompts = "Username:: : Password::"
< #   server_condition = "${if crypteq{$2}{${extract{1}{:}{${lookup{$1}lsearch{CONFDIR/passwd}{$value}{*:*}}}}}{1}{0}}"
< #   server_set_id = $1
< #   .ifndef AUTH_SERVER_ALLOW_NOTLS_PASSWORDS
< #   server_advertise_condition = ${if eq{$tls_cipher}{}{}{*}}
< #   .endif
< #
< # cram_md5_server:
< #   driver = cram_md5
< #   public_name = CRAM-MD5
< #   server_secret = ${extract{2}{:}{${lookup{$1}lsearch{CONFDIR/passwd}{$value}fail}}}
< #   server_set_id = $1
---
> plain_server:
>   driver = plaintext
>   public_name = PLAIN
>   server_condition = ${if pam{$2:${sg{$3}{:}{::}}}{yes}{no}}
>   server_set_id = $2
>   server_prompts = :
>   .ifndef AUTH_SERVER_ALLOW_NOTLS_PASSWORDS
>   server_advertise_condition = ${if eq{$tls_cipher}{}{}{*}}
>   .endif
> 
> login_server:
>   driver = plaintext
>   public_name = LOGIN
>   server_prompts = "Username:: : Password::"
>   server_condition = ${if pam{$1:${sg{$2}{:}{::}}}{yes}{no}}
>   server_set_id = $1
>   .ifndef AUTH_SERVER_ALLOW_NOTLS_PASSWORDS
>   server_advertise_condition = ${if eq{$tls_cipher}{}{}{*}}
>   .endif
> 
> cram_md5_server:
>   driver = cram_md5
>   public_name = CRAM-MD5
>   server_secret = ${extract{2}{:}{${lookup{$1}lsearch{CONFDIR/passwd}{$value}fail}}}
>   server_set_id = $1

New Files

main/000_localmacros

Clearly the first part put in the runtime configuration file.

SYSTEM_ALIASES_FILE_TRANSPORT=address_file
CHECK_RCPT_VERIFY_SENDER = true
CHECK_DATA_VERIFY_HEADER_SYNTAX = true
CHECK_DATA_VERIFY_HEADER_SENDER = true
MMS_USER_CHECK = select email_string from mms_mail_auth_view \
                 where login_name = '${quote_pgsql:$local_part}';

main/04_mailman_config

# See /usr/share/doc/mailman/README.EXIM.gz

# Home dir for your Mailman installation -- aka Mailman's prefix
# directory.
MAILMAN_HOME=/var/lib/mailman
MAILMAN_WRAP=MAILMAN_HOME/mail/mailman

# User and group for Mailman, should match your --with-mail-gid
# switch to Mailman's configure script.
MAILMAN_USER=list
MAILMAN_GROUP=daemon

main/05_postgres_config

# this gives us access to postgres from here on in
# the hide means this parameter won't be shown if exim is asked to
# display it's configuration
# pgsql_servers = host/database/user/password

hide pgsql_servers = postgres.openskills.org/mms/username/password

Actually a real user name and password is found in the file.

main/15_sa-exim_plugin_path

# This will enable sa-exim, but it won't actually scan and possibly reject
# messsages before you enable this in sa-exim.conf (see SAEximRunCond)
#
# For a starter, you'd probably want to read the documentation at:
#   /usr/share/doc/sa-exim/README.Debian
# and
#   /usr/share/doc/sa-exim/README.gz
#
local_scan_path = /usr/lib/exim4/local_scan/sa-exim.so

main/20_clamav-daemon_socket_path

# This will enable clamav, but it won't actually scan and possibly reject
# messsages before further configuration
#
# For a starter, you'd probably want to read the documentation at:
#   /usr/share/doc/clamav-daemon/examples/exim4
#
av_scanner = clamd:/var/run/clamav/clamd.ctl

router/350_mailman_router

# Taken from /usr/share/doc/mailman/README.EXIM.gz
# Added for mailman aliases by patrick@lesslie.ath.cx 1/9/2005

mailman_router:
  driver = accept
  require_files = MAILMAN_HOME/lists/$local_part/config.pck
  local_part_suffix_optional
  local_part_suffix = -bounces : -bounces+* : \
                      -confirm+* : -join : -leave : \
                      -owner : -request : -admin
  transport = mailman_transport

router/450_mms_user

mms_user:
  debug_print = "R: mms_user for $local_part@$domain"
  driver = redirect
  domains = openskills.org
  no_expn
  allow_defer
  data = ${lookup pgsql{MMS_USER_CHECK}}

transport/30_mailman_transport

# See /usr/share/doc/mailman/README.EXIM.gz
# Added for mailman aliases by patrick@lesslie.ath.cx 1/9/2005

mailman_transport:
  driver = pipe
  command = MAILMAN_WRAP \
            '${if def:local_part_suffix \
                  {${sg{$local_part_suffix}{-(\\w+)(\\+.*)?}{\$1}}} \
                  {post}}' \
            $local_part
  current_directory = MAILMAN_HOME
  home_directory = MAILMAN_HOME
  user = MAILMAN_USER
  group = MAILMAN_GROUP

Analysis

Configuration

Well it appears that all changes are reflected in the split configuration. But it is not the case that all changes are reflected in the unsplit template. You'll see, for example, the greylistd modifications in both places. Most other changes seem only to be in the split files.

So the guess is that the grelistd install modified both configurations. Further, my guess is that update-exim4.conf.template is not being run to have the split modifications reflected in the current template. So only the split configuration is definitive for the current state of our server. I used the unsplit template as a vehicle for study of basic Exim configuration.

General

SYSTEM_ALIASES_FILE_TRANSPORT

Taking a look at main/000_localmacros we have:

SYSTEM_ALIASES_FILE_TRANSPORT=address_file

This variable (macro) assignment is used in router/400_exim4-config_system_aliases. This file sets up a redirect router (chapter 22 of exim docs) to deal with the /etc/aliases file. The result being that our redirect router processing the aliases file gets the option "file_transport = address_file".

Chapter 22 is pretty clear on this option. It deals with addresses being path names and sending messages to those names by appending them to the named file. Now our aliases file has a few such entries - press-archive, membership-archive. So to deal with these we are explicitly choosing a stock transport named "address_file" that can be found in transport/30_exim4-config_address_file.

Why did we have to explicitly define this macro and not (analogously)

SYSTEM_ALIASES_PIPE_TRANSPORT=address_pipe

to deal with all the mailing list related pipes in also found in the aliases file? In fact chapter 22 is clear that you have to specify the pipe_transport option if you want the redirect router to handle pipes. So it seems clear that the router instance in router/400_exim4-config_system_aliases is not handling the mailman related pipes. In fact, from my brief study of mailman and current understanding, the mailman related pipes in /etc/aliases are not being used at all!

CHECK_RCPT_VERIFY_SENDER

hmmm. This adds

deny
    message = Sender verification failed
    !acl = acl_whitelist_local_deny
    !verify = sender
  .endif

to acl/30_exim4-config_check_rcpt to the standard recipient ACL (the same one modified by greylistd - but not affecting the same "verb"). Every sender is verified with a test SMTP connection with an associated DNS lookup to find the MX record. Exim caches the answers.

Mailman docs have an explicit warning section on SMTP callbacks. Apparently if mailman is delivery mail to a subscriber but the address is no longer valid then the bounce message of the target SMTP server might have a bogus address. Therefore, the sender verification will fail. So mailman will never find out about the bounced mail and remove the obsolete subscriber (because of this ACL).

Perhaps that is fine given the benefit of reduced spam. But now my eyes are open about the consequences.

CHECK_DATA_VERIFY_HEADER_SYNTAX

This adds to acl/40_exim4-config_check_data:

deny
    message = Message headers fail syntax check
    !acl = acl_whitelist_local_deny
    !verify = header_syntax

This seems harmless. Not sure why it was thought to add it. Though it does address spam to some extent.

CHECK_DATA_VERIFY_HEADER_SENDER

This adds to acl/40_exim4-config_check_data:

deny
    deny
    message = No verifiable sender address in message headers
    !acl = acl_whitelist_local_deny
    !verify = header_sender

This seems harmless. Not sure why it was thought to add it. Though it does address spam to some extent.

rfc1413

In main/02_exim4_config_options there is the removal of the asterisk from rfc1413_hosts to avoid the ident callback to sending SMTP servers. This is a funny mofication since the comments and documentation are quite specific regarding disabling the ident callbacks. They both say to set the timeout to 0 to disable.

I have to agree with dropping the ident callback. It took me a few years to realize that my own firewall shouldn't just silently drop ident packets.

Grelistd

Nothing too difficult here. The greylist checks are inserted into two acls. One is run right after the RCPT (identifying the recipient) and the other is run right after the DATA. I feel the comments added by the greylistd install into the two acls is adequate to explain why the two are needed (relating to not interfering with SMTP callouts). I also felt that the man page for greylistd provided anything else needed to fill in the gaps.

Of course chapter 11 of the Exim manual helps to understand the ${readsocket... stuff.

Mailman

Regarding integration with Exim4 this is quite straightforward. The pipes in the alias file are not really enabled. So the standard alias file based redirect router is not used.

Rather router/350_mailman_router configures a simple "accept" router to hand off to the mailman_transport precisely when /var/lib/mailman/lists/$local_part/config.pck exists. So, for example, mail to smalltalk-central-admin@lists.openskills.org has a $local_part of "smalltalk-central" (thanks to the local_part_suffix stripping off the -admin). So MailMan will get the mail if /var/lib/mailman/lists/smalltalk-central/config.pck exists (which it does).

Now the router hands off to the mailman_transport found in transport/30_mailman_transport. This simply pipes to the executable /var/lib/mailman/mail/mailman.

MMS

We see

main/000_localmacros

MMS_USER_CHECK = select email_string from mms_mail_auth_view \
                 where login_name = '${quote_pgsql:$local_part}';

main/05_postgres_config

hide pgsql_servers = postgres.openskills.org/mms/username/password

"username" and "password" have some real values plugged in. That's why the "hide" is used.

router/450_mms_user

mms_user:
  debug_print = "R: mms_user for $local_part@$domain"
  driver = redirect
  domains = openskills.org
  no_expn
  allow_defer
  data = ${lookup pgsql{MMS_USER_CHECK}}

Certainly exim docs 9.20 and 9.21 should be consulted concerning general postgres setup.

So this seems quite easy. Access to postgres requires exim-daemon-heavy and the c library libpq3.

Spam Assassin

This is not the normal spam assassin discussed in the Exim docs. Rather it is the Spam Assassin at SMTP time. This is the sa-exim package. This just hooks into the local_scan functionality of exim by the setting found in main/15_sa-exim_plugin_path. The dynamic library specified just invokes spamc (which invokes spamassassin on a socket).

/etc/spamassassin/local.cf has some modifications. In fact the stock local.cf has no parameters specified so the ones found here are all added by Anand.

version_tag os
required_hits 4.00
report_safe 0
envelope_sender_header X-SA-Exim-Mail-From
bayes_auto_learn_threshold_spam 6.50
bayes_auto_learn_threshold_nonspam -20.00

/etc/default/spamassassin has some mods

8c8
< ENABLED=0
---
> ENABLED=1
17c17
< OPTIONS="--create-prefs --max-children 5 --helper-home-dir"
---
> OPTIONS="--create-prefs --max-children 3 --helper-home-dir --socketpath=/var/run/spamd.sock"
26c26
< #NICE="--nicelevel 15"
---
> NICE="--nicelevel 15"

Also you'll find some changes in /etc/exim4/sa-exim.conf

50c50
< #SAspamcSockPath: /var/run/spamd.sock
---
> SAspamcSockPath: /var/run/spamd.sock
64c64
< SAEximRunCond: 0
---
> #SAEximRunCond: 0

ClamAV

This is not being used.


Edit Rename Changes History Upload Download Back to Top