"http://www.w3.org/TR/html4/loose.dtd">

Postfix After-Queue Content Filter

Introduction

This document requires Postfix version 2.1 or later.

Normally, Postfix receives mail, stores it in the mail queue and then delivers it. With the external content filter described here, mail is filtered AFTER it is queued. This approach decouples mail receiving processes from mail filtering processes, and gives you maximal control over how many filtering processes you are willing to run in parallel.

The after-queue content filter is meant to be used as follows:

Network or
local users -> Postfix
queue -> Content
filter -> Postfix
queue -> Network or
local mailbox

This document describes implementations that use a single Postfix instance for everything: receiving, filtering and delivering mail. Applications that use two separate Postfix instances will be covered by a later version of this document.

The after-queue content filter is not to be confused with the approach that is described in the SMTPD_PROXY_README document, where incoming SMTP mail is filtered BEFORE it is stored into the Postfix queue.

This document describes two approaches to content filter all email, as well as several options filter mail selectively:

Principles of operation
Simple content filter example
Simple content filter limitations
Advanced content filter example
Advanced content filter performance
Filtering mail from outside users only
Different filters for different domains
FILTER actions in access or header/body tables

Principles of operation

An external content filter receives unfiltered mail from Postfix (as described further below) and does one of the following:

Re-inject the mail back into Postfix, perhaps after changing content and/or destination.
Reject the mail (by sending a suitable status code back to Postfix). Postfix will return the mail to the sender.

NOTE: in this time of mail worms and forged spam, it is a VERY BAD IDEA to send viruses back to the sender address, because the sender address is almost certainly not the originator. It is better to discard known viruses, and to quarantine material that is suspect so that a human can decide what to do with it.

Simple content filter example

The first example is simple to set up. Postfix receives unfiltered mail from the network with the smtpd(8) server, and delivers unfiltered mail to a content filter with the Postfix pipe(8) delivery agent. The content filter injects filtered mail back into Postfix with the Postfix sendmail(1) command, so that Postfix can deliver it to the final destination.

This means that mail submitted via the Postfix sendmail(1) command cannot be content filtered.

In the figure below, names followed by a number represent Postfix commands or daemon programs. See the OVERVIEW document for an introduction to the Postfix architecture.

Unfiltered

->

smtpd(8)

pickup(8) >- cleanup(8) -> qmgr(8)
Postfix
queue -< local(8)
smtp(8)
pipe(8) ->
->
Filtered
Filtered

^
| | v

maildrop
queue <- Postfix
postdrop(1) <- Postfix
sendmail(1) <- Content
filter

The content filter can be a simple shell script like this:

 1 #!/bin/sh
 2 
 3 # Simple shell-based filter. It is meant to be invoked as follows:
 4 #       /path/to/script -f sender recipients...
 5 
 6 # Localize these.
 7 INSPECT_DIR=/var/spool/filter
 8 SENDMAIL="/usr/sbin/sendmail -i"
 9 
10 # Exit codes from <sysexits.h>
11 EX_TEMPFAIL=75
12 EX_UNAVAILABLE=69
13 
14 # Clean up when done or when aborting.
15 trap "rm -f in.$$" 0 1 2 3 15
16 
17 # Start processing.
18 cd $INSPECT_DIR || {
19     echo $INSPECT_DIR does not exist; exit $EX_TEMPFAIL; }
20 
21 cat >in.$$ || { 
22     echo Cannot save mail to file; exit $EX_TEMPFAIL; }
23 
24 # Specify your content filter here.
25 # filter <in.$$ || {
26 #   echo Message content rejected; exit $EX_UNAVAILABLE; }
27 
28 $SENDMAIL "$@" <in.$$
29 
30 exit $?

Notes:

Line 21: The idea is to first capture the message to file and then run the content through a third-party content filter program.
Line 22: If the mail cannot be captured to file, mail delivery is deferred by terminating with exit status 75 (EX_TEMPFAIL). Postfix places the message in the deferred mail queue and tries again later.
Line 25: You will need to specify a real content filter program here that receives the content on standard input.
Line 26: If the content filter program finds a problem, the mail is bounced by terminating with exit status 69 (EX_UNAVAILABLE). Postfix will return the message to the sender as undeliverable.
Note: in this time of mail worms and spam, it is a BAD IDEA to send known viruses or spam back to the sender, because that address is likely to be forged. It is safer to discard known to be bad content and to quarantine suspicious content so that it can be inspected by a human being.
Line 28: If the content is OK, it is given as input to the Postfix sendmail command, and the exit status of the filter command is whatever exit status the Postfix sendmail command produces. Postfix will deliver the message as usual.
Line 30: Postfix returns the exit status of the Postfix sendmail command.

I suggest that you first run this script by hand until you are satisfied with the results. Run it with a real message (headers+body) as input:

% /path/to/script -f sender recipient... <message-file

Once you're satisfied with the content filtering script:

Create a dedicated local user account called "filter". This user handles all potentially dangerous mail content - that is why it should be a separate account. Do not use "nobody", and most certainly do not use "root" or "postfix".
Create a directory /var/spool/filter that is accessible only to the "filter" user. This is where the content filtering script is supposed to store its temporary files.

Configure Postfix to deliver mail to the content filter with the pipe(8) delivery agent.

/usr/local/etc/postfix/master.cf:
  # =============================================================
  # service type  private unpriv  chroot  wakeup  maxproc command
  #               (yes)   (yes)   (yes)   (never) (100)
  # =============================================================
  filter    unix  -       n       n       -       10      pipe
    flags=Rq user=filter argv=/path/to/script -f ${sender} -- ${recipient}

This runs up to 10 content filters in parallel. Instead of a limit of 10 concurrent processes, use whatever process limit is feasible for your machine. Content inspection software can gobble up a lot of system resources, so you don't want to have too much of it running at the same time.

To turn on content filtering for mail arriving via SMTP only, append "-o content_filter=filter:dummy" to the master.cf entry that defines the Postfix SMTP server:
```
/usr/local/etc/postfix/master.cf:
  # =============================================================
  # service type  private unpriv  chroot  wakeup  maxproc command
  #               (yes)   (yes)   (yes)   (never) (100)
  # =============================================================
  smtp      inet  ...other stuff here, do not change...   smtpd
        -o content_filter=filter:dummy
```
The "content_filter" line causes Postfix to add one content filter request record to each incoming mail message, with content "filter:dummy". This record overrides the normal mail routing and causes mail to be given to the content filter instead.

The content_filter configuration parameter accepts the same syntax as the right-hand side in a Postfix transport table.
Execute "postfix reload" to complete the change.

To turn off content filtering, edit the master.cf file, remove the "-o content_filter=filter:dummy" text from the entry that defines the Postfix SMTP server, and execute another "postfix reload".

With the shell script as shown above you will lose a factor of four in Postfix performance for transit mail that arrives and leaves via SMTP. You will lose another factor in transit performance for each additional temporary file that is created and deleted in the process of content filtering. The performance impact is less for mail that is submitted or delivered locally, because such deliveries are already slower than SMTP transit mail.

Simple content filter limitations

The problem with content filters like the one above is that they are not very robust. The reason is that the software does not talk a well-defined protocol with Postfix. If the filter shell script aborts because the shell runs into some memory allocation problem, the script will not produce a nice exit status as defined in the file /usr/include/sysexits.h. Instead of going to the deferred queue, mail will bounce. The same lack of robustness can happen when the content filtering software itself runs into a resource problem.

The simple content filter method is not suitable for content filter actions that are invoked via header_checks or body_checks patterns. These patterns will be applied again after mail is re-injected with the Postfix sendmail command, resulting in a mail filtering loop. The advanced content filtering method (see below) makes it possible to turn off header_checks or body_checks patterns for filtered mail.

Advanced content filter example

The second example is more complex, but can give better performance, and is less likely to bounce mail when the machine runs into some resource problem. This content filter receives unfiltered mail with SMTP on localhost port 10025, and sends filtered mail back into Postfix with SMTP on localhost port 10026.

For non-SMTP capable content filtering software, Bennett Todd's SMTP proxy implements a nice PERL/SMTP content filtering framework. See: http://bent.latency.net/smtpprox/.

In the figure below, names followed by a number represent Postfix commands or daemon programs. See the OVERVIEW document for an introduction to the Postfix architecture.

Unfiltered

Unfiltered ->

-> smtpd(8)

pickup(8) >- cleanup(8) -> qmgr(8)
Postfix
queue -< smtp(8)

local(8) ->

-> Filtered

Filtered

^
| | v

smtpd(8)
10026 smtp(8)

^
| | v

content filter 10025

The example given here filters all mail, including mail that arrives via SMTP and mail that is locally submitted via the Postfix sendmail command. See examples near the end of this document for how to exclude local users from filtering, or how to configure a destination dependent content filter.

You can expect to lose about a factor of two in Postfix performance for mail that arrives and leaves via SMTP, provided that the content filter creates no temporary files. Each temporary file created by the content filter adds another factor to the performance loss.

Advanced content filter: requesting that all mail is filtered

To enable the advanced content filter method for all mail, specify in main.cf:

/usr/local/etc/postfix/main.cf:
    content_filter = scan:localhost:10025
    receive_override_options = no_address_mappings

The "content_filter" line causes Postfix to add one content filter request record to each incoming mail message, with content "scan:localhost:10025". The content filter request records are added by the smtpd(8) and pickup(8) servers (and qmqpd(8) if you decide to enable this service).
Content filter requests are stored in queue files; this is how Postfix keeps track of what mail needs filtering. When a queue file contains a content filter request, the queue manager will deliver the mail to the specified content filter regardless of its final destination.
The "receive_override_options" line disables address manipulation before the content filter, so that the content filter sees the original mail addresses instead of the result of virtual alias expansion, canonical mapping, automatic bcc, address masquerading, etc.

To turn off content filtering, delete or comment out the two above main.cf lines. All other changes made for advanced content filtering have no effect when content filtering is turned off.

Advanced content filter: sending unfiltered mail to the content filter

In this example, "scan" is an instance of the Postfix SMTP client with slightly different configuration parameters. This is how one would set up the service in the Postfix master.cf file:

/usr/local/etc/postfix/master.cf:
    # =============================================================
    # service type  private unpriv  chroot  wakeup  maxproc command
    #               (yes)   (yes)   (yes)   (never) (100)
    # =============================================================
    scan      unix  -       -       n       -       10      smtp
        -o smtp_send_xforward_command=yes

This runs up to 10 content filters in parallel. Instead of a limit of 10 concurrent processes, use whatever process limit is feasible for your machine. Content inspection software can gobble up a lot of system resources, so you don't want to have too much of it running at the same time.
With "-o smtp_send_xforward_command=yes", the scan transport will try to forward the original client name and IP address to the after-filter smtpd process, so that filtered mail is logged with the real client name IP address. See smtp(8) and XFORWARD_README for more information.

Advanced content filter: running the content filter

The content filter can be set up with the Postfix spawn service, which is the Postfix equivalent of inetd. For example, to instantiate up to 10 content filtering processes on localhost port 10025:

/usr/local/etc/postfix/master.cf:
    # ===================================================================
    # service       type  private unpriv  chroot  wakeup  maxproc command
    #                     (yes)   (yes)   (yes)   (never) (100)
    # ===================================================================
    localhost:10025 inet  n       n       n       -       10      spawn
        user=filter argv=/path/to/filter localhost 10026

"filter" is a dedicated local user account. The user will never log in, and can be given a "*" password and non-existent shell and home directory. This user handles all potentially dangerous mail content - that is why it should be a separate account.

If you want to have your filter listening on port localhost:10025 instead of Postfix, then you must run your filter as a stand-alone program, and must not use the Postfix spawn service.

Advanced filter: injecting mail back into Postfix

The job of the content filter is to either bounce mail with a suitable diagnostic, or to feed the mail back into Postfix through a dedicated listener on port localhost 10026.

The simplest content filter just copies SMTP commands and data between its inputs and outputs. If it has a problem, all it has to do is to reply to an input of `.' from Postfix with `550 content rejected', and to disconnect without sending `.' on the connection that injects mail back into Postfix.

/usr/local/etc/postfix/master.cf:
    # ===================================================================
    # service       type  private unpriv  chroot  wakeup  maxproc command
    #                     (yes)   (yes)   (yes)   (never) (100)
    # ===================================================================
    localhost:10026 inet  n       -       n       -       10      smtpd
        -o content_filter= 
        -o receive_override_options=no_unknown_recipient_checks,no_header_body_checks
        -o smtpd_helo_restrictions=
        -o smtpd_client_restrictions=
        -o smtpd_sender_restrictions=
        -o smtpd_recipient_restrictions=permit_mynetworks,reject
        -o mynetworks=127.0.0.0/8
        -o smtpd_authorized_xforward_hosts=127.0.0.0/8

Note: do not use spaces around the "=" or "," characters.
Note: the SMTP server must not have a smaller process limit than the "filter" master.cf entry.
The "-o content_filter=" overrides main.cf settings, and requests no content filtering for mail from the content filter. This is required or else mail will stay in the content filtering loop.
The "-o receive_override_options" overrides main.cf settings. It is complementary to the options that are specified in main.cf:
- Disable attempts to find out if a recipient is unknown, and disable header/body checks. This work was already done before the content filter and repeating it would be wasteful.
- Enable virtual alias expansion, canonical mappings, address masquerading, and other address mappings.
These receive override options are either implemented by the SMTP server itself, or they are passed on to the cleanup server.
The "-o smtpd_xxx_restrictions" and "-o mynetworks=127.0.0.0/8" override main.cf settings. They turn off junk mail controls that would only waste time here.
With "-o smtpd_authorized_xforward_hosts=127.0.0.0/8", the scan transport will try to forward the original client name and IP address to the after-filter smtpd process, so that filtered mail is logged with the real client name and IP address. See XFORWARD_README and smtpd(8).

Advanced content filter performance

With the "sandwich" approach to content filtering described here, it is important to match the filter concurrency to the available CPU, memory and I/O resources. Too few content filter processes and mail accumulates in the active queue even with low traffic volume; too much concurrency and Postfix ends up deferring mail destined for the content filter because processes fail due to insufficient resources.

Currently, content filter performance tuning is a process of trial and error; analysis is handicapped because filtered and unfiltered messages share the same queue. As mentioned in the introduction of this document, content filtering with multiple Postfix instances will be covered in a future version.

Filtering mail from outside users only

The easiest approach is to configure ONE Postfix instance with multiple SMTP server IP addresses in master.cf:

Two SMTP server IP addresses for mail from inside users only, with content filtering turned off.

/usr/local/etc/postfix.master.cf:
    # ==================================================================
    # service      type  private unpriv  chroot  wakeup  maxproc command
    #                    (yes)   (yes)   (yes)   (never) (100)
    # ==================================================================
    1.2.3.4:smtp   inet  n       -       n       -       -       smtpd
        -o smtpd_client_restrictions=permit_mynetworks,reject
    127.0.0.1:smtp inet  n       -       n       -       -       smtpd
        -o smtpd_client_restrictions=permit_mynetworks,reject

One SMTP server address for mail from outside users with content filtering turned on.

/usr/local/etc/postfix.master.cf:
    # =================================================================
    # service     type  private unpriv  chroot  wakeup  maxproc command
    #                   (yes)   (yes)   (yes)   (never) (100)
    # =================================================================
    1.2.3.5:smtp  inet  n       -       n       -       -       smtpd
        -o content_filter=foo:bar 
        -o receive_override_options=no_address_mappings

After this, you can follow the same procedure as outlined in the "advanced" or "simple" content filtering examples above, except that you must not specify "content_filter" or "receive_override_options" in the main.cf file.

Different filters for different domains

If you are an MX service provider and want to apply different content filters for different domains, you can configure ONE Postfix instance with multiple SMTP server IP addresses in master.cf. Each address provides a different content filter service.

/usr/local/etc/postfix.master.cf:
    # =================================================================
    # service     type  private unpriv  chroot  wakeup  maxproc command
    #                   (yes)   (yes)   (yes)   (never) (100)
    # =================================================================
    # SMTP service for domains that are content filtered with foo:bar
    1.2.3.4:smtp  inet  n       -       n       -       -       smtpd
        -o content_filter=foo:bar 
        -o receive_override_options=no_address_mappings

    # SMTP service for domains that are content filtered with xxx:yyy
    1.2.3.5:smtp  inet  n       -       n       -       -       smtpd
        -o content_filter=xxx:yyy 
        -o receive_override_options=no_address_mappings

Set up MX records in the DNS that route each domain to the proper SMTP server instance.

FILTER actions in access or header/body tables

The above filtering configurations are static. Mail that follows a given path is either always filtered or it is never filtered. As of Postfix 2.0 you can also turn on content filtering on the fly.

To turn on content filtering with an access(5) table rule:

/usr/local/etc/postfix/access:
    whatever       FILTER foo:bar

To turn on content filtering with a header_checks(5) or body_checks(5) table pattern:

/usr/local/etc/postfix/header_checks:
    /whatever/     FILTER foo:bar

You can do this in smtpd access maps as well as the cleanup server's header/body_checks. This feature must be used with great care: you must disable all the UCE features in the after-filter smtpd and cleanup daemons or else you will have a content filtering loop.

Limitations:

FILTER actions from smtpd access maps and header/body_checks take precedence over filters specified with the main.cf content_filter parameter.
If a message triggers more than one filter action, only the last one takes effect.
The same content filter is applied to all the recipients of a given message.