Some senders think that ESPs are essential to good deliverability, as if they are a gateway to email delivery nirvana. Well, ESPs certainly understand and monitor deliverability, and they do help getting the technical pre-requisites right. But in the end it is the sender’s list and content what drives the deliverability, and infrastructure is not more than an important pre-requisite.

Quite the opposite, some think that sending in-house just requires a server with email software. As soon as you start sending more than a few thousand emails you need to know about best practices, provider policies, and deliverability management. This doesn’t have to be a hurdle, since there are resources and consultants to help, but it will either cost effort or money to address these matters.

So what are the advantages of using an ESP? I think the most important are the following:

• Low setup costs
• Easy provisioning
• High availability
• Deliverability management
• Email marketing support

And for in-house infrastructure, the following are important advantages:

• No CPM (per mail) fees
• Tight integration
• Dedicated system resources
• Full control over deliverability
• Data remains in-house

How this works out in the situation of a specific sender depends on the situation, and how one values certain aspects. With regards to the costs, volume is very important. The ESPs charge a rate per thousand (CPM) of about € 0,5 to € 3. The one-time costs of an in-house system typically vary between € 5K and € 30K and the monthly costs between € 500/mo and € 1500/mo.

If you ask me, senders that send less than one million per month are most often better of using an ESP. Above that, it can be interesting to send in-house. Because it is cheaper in the long run, because it integrates well with other in-house systems, or just because a self-owned system is of strategic importance.

Before 1-2-1 Marketing turned to us, they were already using PowerMTA to deliver transactional and marketing email from golf courses. The golf courses use a specific software solution for making tee time reservations and send email campaigns to their members. All email from the reservation software was submitted directly to the PowerMTA using SMTP.

To improve the deliverability we decided that bounce processing and complaint feedback loops needed to be implemented. Due to migration of email addresses and other causes bounce rates had risen to dangerous numbers. But adding bounce management in the golf course software running at over hundred sites was not an option. The solution was to suppress email to invalid addresses and complainers in the email relay service.

PowerMTA is able to process synchronous bounces, asynchonous bounces and complaint reports from feedback loops. Information from bounces and complaints is provided in a CSV based log file. To implement efficient suppression using a large list of addresses we choose to implement Exim in front of PowerMTA. This “back to back” configuration combines the strong filtering functionality of Exim with the strong delivery functionality of PowerMTA.

The new email relay solution uses address rewriting and header insertion to make sure that bounces and complaints can be processed properly. Information from bounces and complaints is periodically processed by scripts and transformed into a sender specific suppression database with a fast binary tree index. This resulted in a “worry free” solution for the users of the email relay service.

One solution to determine IPR are tools that monitor a list of mailboxes at various providers. The addresses of these mailboxes, the so called seed-list, is added to the mailing list of campaigns to be tested. When a new mailing is detected by the tool, it checks if the mailing arrived in it’s mailboxes and determines the disposition (inbox or junk). These tools then calculate the IPR by extrapolating the results of the seed-list.

But extrapolating the results of a seed-list may be far from accurate. Even the largest seed-list may be tiny compared to the size of the mailing list. To calculate the IPR the importance of tested domains in the mailing list must be taken into account. If the tool detects that Yahoo junks the mail for example, the impact on the IPR is much larger for a typical US sender than for a typical European sender.

Another problem with automatic mailbox monitoring is the fact that more providers are starting to filter based on personal behavior or preferences. Examples are Hotmail Sweep, Gmail Priority Inbox and Yahoo Automatic Organizer. This means that the results may be different between seed accounts and real user accounts, or even between the accounts of different users at the same provider.

There is a fairly simple and free method to get a picture about your IPR, based on all mailboxes in your mailing list. Even though this also may not give you an accurate IPR, it nicely complements the information you can get from a seed-list based tool. This method uses raw bounce, open and click data which is used to create a report similar to the one below.

Using this report you need to look for domains that have a bounce ratio which is much higher than the average or an open/click ratio which is much lower than the average. In the example report above it is clear that there is an issue with yahoo.com, which is 4.91% of the list, impacting the IPR with the same percentage.

Some email marketing solutions already provide a mailing report which can be split by domain. The ones that lack these reports should add them soon, as it provides valuable insight into the deliverability. Otherwise raw data exports must be used to group all data on domain level, and presented in a spreadsheet. For this, I have used ad-hoc Ruby or Perl scripts to read the exported files and output a sorted CSV file which can be read in Excel.

Postfix is a popular open source MTA thanks to it’s security and performance. I myself have used it in the early 2000s to power an email service provider. At one time I had 8 servers sending up to one million emails per day.

However, Postfix has a fundamental problem, just like the other open source MTAs. They have a design focus on filtering and distributing inbound email. That means they are very good at accepting mail from different sources and delivering them locally. Delivering large volumes of email requires another design approach.

If you are a sender concerned about deliverability you will notice that it is important to sign email with DKIM, segment mail streams over different IPs, and adapt delivery to reputation related errors. Commercial MTAs targeted for outbound delivery make these things a lot easier. They support many IPs, parallel queues and provide lots of delivery related configuration options.

Around 2004 I decided to switch from Postfix to PowerMTA from Port25. A redundant set of two servers was enough to replace the 8 Postfix servers. Still it was quite an investment, but it was all paid back in less administration and better quality. Instead of troubleshooting performance issues I could concentrate on other things, like deliverability management.

After I started to work as a consultant in 2007, I have also worked with StrongMail and Message Systems. These are also good products, each with distinctive advantages. What I like about PowerMTA is that it does one thing, and does it well, making it a manageable product with a great price/value ratio.

421 4.7.1 Intrusion prevention active for [X.X.X.X]
451 qq read error (#4.3.0)
452 Message for would exceed mailbox quota
530 Authentication required
550 5.1.1 Not our Customer
550 Mailbox unavailable or access denied –
550 Administrative prohibition
550 Envelope blocked – User Entry
550 no such address here
550 5.7.1 e-mail address access denied
550 not valid
550 failed_address_router router forced verify failure
553 sorry, that domain isn’t in my list of allowed rcpthosts (#5.7.1)
554 5.7.1 UBE Not Welcome Here!
554 5.3.0 rewrite: map parse not found

In order to process bounces in the right way they first need to be classified. After classification we can apply bounce rules to them. A good rule is that invalid email addresses are immediately disabled and excluded from further mailings. Other bounce types should be removed as soon as it is clear that the problem is permanent. Spam related bounces should be detected and require action from the sender to resolve them.

The famous two categories, “hard” and “soft”, for which many use different definitions, is vague concept and a generalization which should not be used to base any business logic on. All decent mail servers will retry the delivery when it encounters an error that starts with the code ’4xx’, indicating a temporary failure. So when the email bounces, it has been retried many times so there’s nothing “soft” about it.

If you think that the three digit code in the bounce can be used for classification, you are wrong. Even the enhanced status codes as described in RFC 3463 are often ambiguous or used wrongly. Moreover, the status classes described in this and other RFCs are not really useful from the perspective of email list management. For example a 5.7.1 “relay access denied” is not really a security issue, but often caused by an invalid email domain that resolves to a host.

So what is a good classification then? After thinking about for a very long time and analyzing many bounces I came up with the following:

• recipient related
  • user unknown
  • mailbox inactive
  • quota exceeded
• domain related
  • invalid domain
  • no mail host
  • relay/access denied
• spam related
  • sender blocked
  • content blocked
  • policy issue
• system related
  • system issue
  • protocol issue
  • connection issue

With these four main categories, it is more clear where the source of the problem is. Using the subcategories, you make your bounce rules aggressive or relaxed. For example by changing the behaviour on mailbox inactive or quota exceeded bounces.

With some programming I was able to parse and classify hundreds of thousands of bounces into the categories mentioned above. Using some ingenious text pattern matching I ended up having a list of almost 400 different patterns. More on that in a future post.

It is astonishing to see how even large ISPs obfuscate a simple “user unknown” with descriptions such as “unrouteable address”, “no such mail drop defined” or “unsupported mail destination”. This makes it almost impossible to automatically classify bounces without frequent adaptation to newly encountered descriptions. How can these ISPs expect senders to properly maintain their database that way?

421 PR(ct1) The mail server IP connecting to Windows Live Hotmail server has exceeded the connection limit allowed. If you are not an email/network admin please contact your E-mail/Internet Service Provider for help.  For e-mail delivery information, please go to http://postmaster.live.com

421 RP-001 The mail server IP connecting to Windows Live Hotmail server has exceeded the rate limit allowed. Reason for rate limitation is related to IP/domain reputation problems. If you are not an email/network admin please contact your E-mail/Internet Service Provider for help. Email/network admins, please visit http://postmaster.live.com for email delivery information and support

PowerMTA offers a number of parameters for tuning email delivery to specific domains. The max-connect-rate and max-msg-rate parameters can be used to prevent these errors from occurring. If you would not react to these errors and keep on pushing your mail, your reputation could become even worse. In the PowerMTA configuration, this looks like:

<domain hotmail.com>
  max-connect-rate 10/m
  max-msg-rate 1000/h
  # other settings
</domain>

Delivery settings in PowerMTA are made within the context of a recipient domain, such as hotmail.com. PowerMTA uses a separate mail queue for each unique combination of Virtual MTA and recipient domain. But there are many other domains related to Windows Live Hotmail, such as hotmail.co.uk, hotmail.de, live.com, live.nl, live.be, msn.com, etc. All these domains and many others are handled by the same mail servers, mx1.hotmail.com to mx4.hotmail.com.

Luckily PowerMTA offers domain-macro that can be used to make settings for a set of domains. For example:

domain-macro hotmail hotmail.com, live.nl, msn.com, hotmail.co.uk, live.com

<domain $hotmail>
  max-connect-rate 10/m
  max-msg-rate 1000/h
  # other settings
</domain>

This effectively creates separate queues for each domain using the same settings. But because there are more queues, the number of connections and the message rate is increased with each queue. This is far from ideal, since the mail servers of Hotmail look at all the traffic from a sender IP, whatever the recipient domain. What we want is a single queue in PowerMTA for all mail traffic going to Hotmail’s servers.

For this we have to resort to an undocumented feature of PowerMTA, the queue-to parameter. This allows us to place mails for a domain in the queue of another domain. The latter queue will send all mail to the mail servers that it’s name resolves to. We can use this to collect the mails for a list of domains in a single queue. See the configuration below:

domain-macro hotmail hotmail.com, live.nl, msn.com, hotmail.co.uk, live.com

<domain $hotmail>
  queue-to hotmail.queue
</domain>

<domain hotmail.queue>
  route hotmail.com
  max-connect-rate 10/m
  max-msg-rate 1000/h
  # other settings
</domain>

The route parameter is used to make the MX lookup more explicit, and allowing us to use a special, arbitrary name for the queue to make clear that it is a special queue for all mail to Hotmail.

Now we are able to control the mail traffic going to Hotmail’s servers much more accurately. However, there is one catch to this configuration, and that is that MX lookups of other domains than hotmail.com are hard-coded in your configuration. Chances may be slim, but if one of the domains listed in the domain-macro is moved to another set of MXs, all mails will bounce, because they are always routed to the MXs of hotmail.com. Also, because the list of domains that are queued to the special queue are hard-coded, there could be other domains that also resolve to Hotmail’s servers.

You can further enhance this configuration by using smtp-pattern-list to detect the errors mentioned above, and automatically put the queue into backoff mode running at a slower pace. How this is done can be found in the PowerMTA Users Guide and is beyond the scope of this post.

Update:

After analyzing the connection data during “exceeded the connection limit” errors, I concluded that it is not the connection rate that causes the errors. Instead I found a correlation between the errors and the number of concurrent connections. This suggests that max-smtp-out should be used instead of max-connect-rate to prevent these errors. A value of 1-5 concurrent connections is a good limit for volumes between 1000-10000 per hour.

The most interesting conclusions I found are listed below.

While six in ten email users (61%) say that they usually refrain from opening emails that they suspect may be spam, only 39% mention taking the extra step of flagging it as spam and 44% say they move it to a junk folder.

So most consumers do not typically open spam, however many email users  have opened one or more spam mails in the past:

Though six in ten say that they typically do not open messages they think are spam, 43% of email users have opened spam in the past, and many have put themselves at even greater risk: clicking on links (11%), opening attachments (8%), replying (4%) or forwarding (4%) these messages.

If you think that spam readers are computer illiterates that don’t know much about security treats you could be surprised:

Those who consider themselves experts or very experienced with Internet security – and who also tend to be younger – are more likely than those who feel inexperienced to have opened spam (52% vs. 38%).

Indeed, some experts like me do occasionally open spam mails to see where they come from, or why they made it to the inbox. See also this post at Word to the Wise. But why do others open spam, or even act on it?

While most users who have opened spam in the past say that it was because they didn’t realize it was spam (57%) or they did so by mistake (33%), others have opened emails that they expected were spam intentionally.

Some of these users are just plain “thrill seekers”:

Men – who also tend to take more risks when sorting through their inbox – are more likely than women to say they open spam purposefully, out of curiosity (21% vs. 14%) or out of interest in the email’s offerings (17% vs. 13%).

Apparently email security is like having safe sex, it requires strong self discipline to avoid the risks.

To register for the feedback loop of Yahoo, you are required to sign emails with DomainKeys or DKIM. DomainKeys requires that the domain used to sign the email (signing identity) matches the Sender: or From: domain (sending domain). This can be troublesome for Email Service Providers, since the DNS and the mail server needs to be configured for each individual From: domain, which often refers to the ESPs customer.

With DKIM, it is not required that the signing identity matches the From: domain. Thus it becomes possible to sign emails with different From: domains using the domain of the ESP. PowerMTA supports this ‘third-party signing’ scheme since release 12 using the dkim-identity parameter. This PowerMTA Recipe shows you how to setup DKIM for any From: domain using PowerMTA.

The dkim-identity parameter

The key to DKIM signing emails with an arbitrary sender domain is the dkim-identity parameter. This parameter allows you to specify the email address of the ‘signing identity’ which becomes the ‘i=’ field in the DKIM signature header.

The dkim-identity parameter is used in combination with the domain-key parameter. The domain-key parameter specifies the selector, the domain of the signing entity and the private key used for signing the mails. The domain becomes the ‘d=’ field in the DKIM signature header.

The DKIM standard requires that the domain part of the ‘i=’ field is the same or a subdomain of the domain in the ‘d=’ field. Thus the domain part of the email address in dkim-identity must match or be a subdomain of the domain in the domain-keys parameter.

Emails will be signed with DKIM by PowerMTA if:

1. dkim-sign is set to ‘yes’ or ‘true’ and,
2a. the domain in any domain-key matches the sender domain (Sender: or From:) or,
2b. the domain part of dkim-identity matches or is a subdomain of the domain in any domain-key

Example configuration

Following is an example configuration which signs all emails to yahoo.com, gmail.com and aol.com with DKIM. The ‘i=’ field is set to postmaster@esp123.com and the ‘d=’ field is set to esp123.com.

Please follow the instructions in the PowerMTA User’s Guide on how to create a private key and how to configure the public key in the DNS.

domain-key sel1, esp123.com, /etc/pmta/sel1.esp123.pem

<domain gmail.com>
  dkim-sign yes
</domain>

<domain yahoo.com>
  dkim-sign yes
</domain>

<domain aol.com>
  dkim-sign yes
</domain>

<domain *>
  dkim-identity postmaster@esp123.com
</domain>

Testing your configuration

You should check your PowerMTA configuration and DNS setup by sending a test email to an account at Gmail, Yahoo, AOL and any other recipient domain configured for DKIM. Retrieve the test email and select Show original (Gmail), Full Header (Yahoo) or View Message Source (AOL). If you see an authentication header with ‘dkim=pass’ or ‘dkim: pass’ your setup is working properly.