Sembee Blog of Exchange MVP Simon Butler

My favourite antispam tool Vamsoft ORF has had an update and now supports Exchange 2010, as well as Windows 2008 and Windows 2008 R2 IIS based SMTP.

While support was available for Exchange 2010 in the previous version, a patch was required, this has now been integrated.

The support for Windows 2008 and 2008 R2 is important because of the changes in IIS.
With Exchange 2003, Exchange used the SMTP engine from IIS. This meant that the product worked with and without Exchange.
With Exchange 2007 and 2010, Exchange has its own SMTP engine and you do not install the IIS SMTP engine on to the server at all. Vamsoft ORF worked with the Exchange SMTP engine, but not the IIS engine that was part of Windows 2008/2008 R2. This update corrects it.

What that means is that you can now use Windows 2008/2008 R2 as an SMTP gateway, as I have outlined in this article on amset.info: http://www.amset.info/exchange/gateway.asp

More information on this update is here: http://www.vamsoft.com/orfee_changelog.asp

For the price of $239 per server, this product is very cost effective.

Some of the background to my liking for Vamsoft ORF, particularly with the latest version can be found elsewhere on my blog here:

Truly Spectacular Results from Vamsoft ORF
http://blog.sembee.co.uk/archive/2009/11/16/112.aspx
Real Time Blacklisting
http://blog.sembee.co.uk/archive/2009/09/26/108.aspx

I have recently seen an issue with the POP3 connector which I haven't seen before, but will be very widespread. In this particular circumstance it caused the client's server to get blacklisted and have a server processing many thousands of messages which it shouldn't need to.

It is yet another reason why using the POP3 Connector is a bad idea. I have blogged on the POP3 connector being a bad option in the past: http://blog.sembee.co.uk/archive/2006/09/25/25.aspx .

This client was not only using a POP3 connector, but they were also using a catch all mailbox at the ISP - I have posted today why using a catch all is a bad idea here:  http://blog.sembee.co.uk/archive/2010/02/15/117.aspx (posting that item was inspired by this one).

The Problem

The actual problem was quite simple, and something that Exchange could have dealt with on its own if the server was setup for SMTP delivery. However it became a noticeable issue because of the way this particular server was configured.

The domain was subject to an NDR or directory harvest attack (I cannot tell which due to the nature of the SBS Connector) and ended up with large numbers of email messages in their queues.

What puzzled the client was that port 25 wasn't open to the internet, and they had followed my guides on recipient filtering and authenticated user relay so that the server was secure ( http://www.amset.info/exchange/spam-cleanup.asp ).
As I wrote in that article, messages can continue to appear in the queues for some hours after the initial clean-up due to the way Exchange displays the queues when there are a very large number of messages in the queues. However for this client, the messages continued to appear for weeks. Eventually, fed up with cleaning the queues daily, I was asked to look at the server.

What I found was that the messages in the queues were all from postmaster@ so had the classic hallmarks of an NDR or direct harvest attack, but the client was using the POP3 Connector.

Due to the way the POP3 connector works, messages that come in to the server through it are not subject to the recipient filter. The recipient filter works at the connection point, but the POP3 connector simply drops the in to the queue for delivery. This is the key point and the result was the same as a standard NDR attack through SMTP without recipient validation  - the messages that could be delivered were, and the messages with invalid external recipients, or where there was a delivery problem, hung around in the queues. As time went on, the server became blacklisted by most major ISPs for being a source of spam and back scatter.

Furthermore, the client also had the POP3 connector setup to send a copy of messages that could not be delivered to a valid user  in to a mailbox, so not only were the messages being delivered there (and the client had what they considered to be a major spam problem) but the NDRs were going out as well. The user concerned thought they were receiving large amounts of spam - when in actual fact they were receiving email that wasn't even addressed to them.

In short, it was a complete mess.

This will be a widespread problem

In many respects, the client was not to blame for this problem. This configuration is quite common, and would therefore affect everyone using the POP3 connector with a catch all mailbox. However you may not see the messages in the queues and therefore be unaware that your server is a source of spam or backscatter.

The most common configuration when SBS is used with a POP3 connector is to route email OUT through a smart host - usually the ISPs SMTP Server. If you are doing that in combination with a catch all mailbox then you wouldn't see the symptoms of this problem. When a smart host is used, Exchange is sending the email straight back out again and the smart host is responsible for the delivery of the email.

It was only because this client was using direct delivery rather than a smart host that the email messages were shown in the queue causing further investigation. The client had accepted large amounts of spam in the mailbox as something that happens - and asked me to look at that as another issue - not realising that it was all caused by the same thing.

If the server had been configured in the usual way for POP3 use, that is to use a smart host, then the first the client would have known there is a problem is when their ISP called to tell them - although many do not.

Furthermore the email messages also do not appear in message tracking logs as they do not pass through Exchange, but simply bounce off SMTP. The only messages that do appear in message tracking are those delivered to the user set to receive the messages that could not be delivered.
Therefore a server could be the source of back scatter and the administrators (whether in house or an external support company) would be completely oblivious to the issue.

I haven't been able to verify if the email messages showed in the volume reported by the SBS Reporting tool, because as with most SBS Servers I see, it wasn't turned on.

The Solution

Changing the client to SMTP delivery of email resulted in the spam level dropping immediately. In the 24 hours after the change, the number of messages the server dropped for non-valid recipients was measured in 1000s. The account which received a copy of the unmatched addresses from the POP3 connector saw the level of spam almost completely drop away - as most of the spam wasn't addressed to the user.

Conclusion

There is a very simple conclusion to this blog posting.
Don't use a catch all mailbox with the POP3 Connector. Ideally you shouldn't use the POP3 connector at all.

If you are using the POP3 connector and do not wish to move to SMTP delivery, then you should look at switching to user specific POP3 mailboxes instead of a catch all. While that is more tedious to setup, it does mean you are only downloading email that you may want, rather than lots of spam that you almost certainly do not, only for it to be rejected.

This is another post in my serious of articles on why you shouldn't use certain features in Exchange, even though they are there. As with the other articles, the article does NOT tell you how to enable the feature in question.
In this post I am going to outline why a "catch all" mailbox is a bad idea.
Many of the points in this article also apply to enabling the option to have a copy of any Non Delivery Reports delivered to someone else in the Exchange org.
This post applies to all email servers, not just Exchange though.

I actually completed this post some time ago, I just never got round to putting it on the blog. However I have recently seen a problem with a SBS Server where a catch all mailbox was used, which I am going to blog on separately, so thought this article should go up first.

The other articles in this series to date are:

Why you shouldn't use logos in signatures ( http://blog.sembee.co.uk/archive/2008/04/14/76.aspx )
Why you shouldn't enable the POP3 Server ( http://blog.sembee.co.uk/archive/2008/03/03/71.aspx )
Why you shouldn't use the POP3 connector ( http://blog.sembee.co.uk/archive/2006/09/25/28.aspx )
Why you shouldn't use a self generated SSL certificate ( http://blog.sembee.co.uk/archive/2006/03/05/9.aspx )
Why you shouldn't put Exchange 2003 in to a DMZ ( http://blog.sembee.co.uk/archive/2006/02/23/7.aspx )

Where does the request come from?

Newcomers to Exchange will often ask where is the "catch all" option, particularly if they are used to that option provided by their ISP with POP3 mailboxes, or are coming from the POP3 connector on Small Business Server.
It may also be asked by a manager, in the mistaken belief that they are missing out on important emails when someone mistypes the email address.

Of course Exchange doesn't support catch all mailboxes, which is why the question is asked.

Similarly, I have seen servers with the "Send copy of Non Delivery Report to..." option set on the SMTP virtual server so that someone gets a copy of the message which can be forwarded on to the relevant person. Of course that is just a COPY, the sender will already have received the NDR, and then may get confused when their message is replied to, despite getting an NDR saying it wasn't delivered.

Why are they a bad idea?

Catch all mailboxes have been a bad idea since the late 1990s, with the growth of worms on the internet that make up email addresses.
In short, a catch all mailbox means that every email address on your server is valid. Therefore the bots that create email addresses based on common name combinations will be able to successfully deliver their messages to your server.

As such, if you enable a catch all, then someone needs to monitor the mailbox constantly for the odd valid email message. Depending on the number of users, the number of messages that could by saved by the catch all may be one or two a day at most.

Meanwhile the person monitoring the mailbox will be deleting the vast majority of the messages, as they will be spam and virus infected messages.
The fact that messages have been delivered at all is also a security risk. If the message is opened or the attachment looked at because it seems legitimate, then the payload could be executed. However if the message had been dropped then it would not even get the chance.

By dropping messages to invalid recipients you will save on bandwidth as the messages do not have to be delivered and on processing power, as the messages do not have to be processed by your AV, Antispam and then Exchange.

Furthermore, if a spammer decides to launch an NDR attack, or simply sends a large amount of spam to your domain, then the messages will be delivered. You may find that you have a mailbox that Outlook cannot open because it has 150,000 messages in it.

I have posted on this blog in the past about VAMSOFT ORF which uses emails to non-valid addresses as a feature to block spam, to great effect. If you are receiving 10,000 messages a day to non-valid address then that would be a tremendous waste of bandwidth - I have a client who drops this kind of level a day.

What is the problem with "Send copy of Non Delivery Report to..." option?

With the "Send copy of Non Delivery Report" option, if you have that set, you are actually being a poor internet citizen. To receive a copy of the Non Delivery Report (NDR) you need to allow the message to be delivered. The server then attempts to send the original NDR back to the sender. However if the message is a virus or spam message (which is most likely) then the sender will be spoofed. Your server is then a source of "back scatter" which could lead to a poor spam score or even blacklisting.
During the last major email-borne virus attack, there were more back scatter NDRs going back and forth than infected messages.
 
Your server is also exposed to an NDR attack or NDR spam attempt, as the server will accept the message and then try and send it back to the "sender" who is the real target of the message.
I have more on NDR attacks on my spam clean up page: http://www.amset.info/exchange/spam-cleanup.asp

Then there is the internal security aspect. If someone senior makes a typo in a confidential email address, this could be seen by someone else, who possibly should not. The original sender will be unaware of this, because they will still get a copy of the NDR.

What are the alternatives?

If you have Exchange 2003 or higher on Windows 2003 SP1 or higher, then enable the recipient filter and tar pit option (instructions: http://www.amset.info/exchange/filter-unknown.asp ). Anyone who sends an email to the wrong address will get a failure immediately. If they are a legitimate sender then they will call or email someone else to get the correct email address.
On older versions of Exchange or if you can't set the tar pit, for example when Exchange 2003 is installed on Windows 2000 where the option isn't available, then setting recipient filtering can actually expose your server to attack, as it cannot defend itself from a directory harvest - therefore a third party tool is required such as Vamsoft's ORF to do recipient filtering and the tar pit.
For other email server products, you should check for recipient validation functionality. If it doesn't exist, but an LDAP lookup option is available, then something like VAMSOFT ORF can query an LDAP database for valid addresses, so could be used as an SMTP gateway. ( http://www.amset.info/exchange/gateway.asp )

If you are aware of a common misspelling, for example Steven and Stephen, then add the misspelling to the user's account as an additional email address. That will ensure that the common misspellings are delivered, without exposing your server.

I have mentioned before the results I have received from Vamsoft ORF in the past, most recently using they honey pot feature http://blog.sembee.co.uk/archive/2009/09/26/108.aspx.

However recently I deployed the product with another client and the results are truly spectacular.
The client has approximately 300 users, and they noticed the results almost immediately.

It was deployed as I have written in the above blog posting, so running in test only for a day or two to build up a white list to begin with then it went live.

The proof is in the numbers, so here is a screenshot of the statistics. At the time this was taken, the system had been running for almost 12 days.

For those of you not believing their eyes, that is 8.8 million messages were attempted to be delivered.
Roughly 700,000 messages a day.
Of which 60,000 were not spam, so around 5,000 a day or 16 per user on average.
The spam ratio hovers at between 99% and 100% (there is some rounding going on there as it is to the nearest full percentage point).

The logs have been watched very carefully for false positives. There have been none.

So lets just go through what is working with that client.

First is DHA protection. Direct Harvest Attack. This is simply a large number of email messages coming from the same IP address to multiple email addresses in a short space of time. For some reason this client receives a lot of messages to invalid recipients. The software blocks the host from sending more messages. It works hand in hand with the honey pot test and recipient validation.

Next is the Honey pot test. I have talked about that before (link above), but in brief it is blocking hosts sending to known non-valid recipients. This feature is simply the most effective thing I have seen against spam for a long time.

Third is recipient validation. Dropping email that is simply sent to users who do not exist. This is a straight query against the AD.

A DNS blacklist is being used - Spamhaus ZEN, but it is only blocking a small percentage of email.

What the screenshot doesn't show is that the built in Exchange 2007 Content Filtering is also enabled, but the number of messages being received in to the quarantine mailbox is a handful a day.

We are not using Greylisting, reverse DNS or the SPF tests.

In short - the three tests that are getting the most results are based on two factors - non-valid recipients and blocking hosts that are sending to them.

The messages are blocked at the point of delivery, therefore the amount of bandwidth used is negligible. The messages do not come in and have to be processed by Exchange, scanned by AV and anti spam software

Due to the volume of email and the number of queries, this system will most likely be moved to an SQL backed database and the load on the domain controller that is used is being watched carefully and  the hardware of the DC increased if required.

If you haven't had a chance to try Vamsoft ORF, then I suggest that you do. The impact can be almost immediate. It is priced per server and because it is based on host and recipients, no definition files to be updated.

Works with all versions of Exchange, including Exchange 2010.

Vamsoft ORF: http://www.shareit.com/product.html?productid=169362&affiliateid=200023740

A common question that keeps coming up on forums and similar sites is "What is the best anti spam solution?"

Unfortunately there is no single answer to this question.
I usually respond with something like "What is the best car, best house, best wife?"

The only answer is that the best product is the one that works for you.
On most forums, most of the posters will have experience with one or two products, so will post that product X has worked well for them.
Someone else may well post and say that product X sucked and product Y was the best solution. Then another person will say, don't bother with a product, outsource it to service Z.

On my home network I have had good experience with Vamsoft ORF, but when I tried it on another site it was unsuccessful. I also tried GFI Mail Essentials at home, found it's performance wasn't great for me. However at another client it has been very successful.

When it comes to looking at antispam solutions, the key metric should not be how much spam does it remove, but how much legitimate email it blocks. If the product is stopping email you want from being delivered, then you need to look at a different product.

I have personal experience with this with a client a few years ago.

The client was a large finance company. They did loans and mortgages through brokers, many of whom used AOL and similar accounts.  (It will surprise you how many of the very small businesses like one man band brokers still do).
They have a requirement for zero false positives - because a single false positive could mean the loss of many thousands of pounds of business.
We evaluated every product on the market, from open source to high end commercial and out sourced solutions. The requirement was very strict - and every product failed because they were all blocking one or two messages a week that were legitimate.
It actually reached the point where we started to put a plan together to hire IT contractors whose sole responsibility was to go through the quarantine email manually, as it was worked out that they would only have to save one email every six months to make it worth while to the company. However in the end, one of the out sourced providers built a custom solution for them so that the management could be handed off.
 
The point I am trying to get across is that asking people what works for them and then using that for a buying decision isn't really a good idea. It does not allow you to bypass the evaluation period. Everyone knows that users don't like spam and no doubt as the administrator of the server you will be under pressure to find a solution that works. However purchasing in haste may actually end up costing your company money.

Most of the major products have evaluation versions you can download. Install them and run them in report only mode. See what it would catch. If you decide to start block messages, then quarantine them first so you can check for false positives.

You could find that the product that someone posts saying "We tried product X and it didn't catch a thing" actually works very well for you.

Blacklisting.
For some email administrators Blacklists are the greatest weapon against spam. It cannot be denied that they can have a significant effect on the amount of email that your server has to process, and they do meet the primary objective of spam detection - dealing with the email at the point of delivery, therefore  reducing back scatter. They are also free, and once setup require little to no maintenance by the administrator.

However personally I dislike blacklists. I don't like the idea of someone else (either human or computer) deciding on what email I should receive, based on lists and reports that I have no control over.
Furthermore, from a business perspective, using a blacklist may cause potential clients to be rejected, as one of my specialism's is the cleanup of servers that have been abused and are likely to be blacklisted.

However, if I could blacklist IP addresses that I know are trying to send spam to me, in real time, where I have complete control over all aspects of the filter, then that could be something of use. A new feature in Vamsoft ORF has introduced exactly that, and has actually got to the point where I have turned off the antispam features in Exchange.

I have written about Vamsoft ORF before, using it for Greylisting (http://blog.sembee.co.uk/archive/2006/09/18/24.aspx) and as part of an SMTP gateway configuration (http://www.amset.info/exchange/gateway.asp).

With the latest version at the time of writing, 4.3, they have introduced a feature called Honey Pot. The simple way that this works is to block IP addresses that attempt to send email to addresses in the Honey Pot list.
In the Vamsoft setup guide it gives you some ideas on how to publish the honey pot addresses, however I found that I didn't need to publish anything.
Going through logs on my backup SMTP gateway, which does recipient validation through Vamsoft rather than Exchange, I noticed that the same non-valid addresses were being used time and time again. These were addresses that I had NEVER used, would never be likely to.

IMPORTANT: The use of addresses that have never been used is the key here. Adding addresses that were in use will provide you with false positives, because that could be legitimate email. If you decide to follow this practise then ensure that you only use addresses that have NEVER been used.

Therefore what I did was turn off recipient validation on my primary SMTP point of entry and configured  that function to be done by Vamsoft ORF. This allowed me to see the addresses that were being sent to on that server as well. I was then able to compile a list to use as my honey pot.
I review the logs frequently to see if new email addresses are being tried, which can be added to the list of honey pot addresses.

This means I am using three tests for spam - recipient validation (which should be something that every site does) greylisting, and honey pot.

The effect was significant. I have been using this setup for a number of weeks and the amount of spam I am seeing in my mailbox or caught by IMF (so got through the initial greylisting and honey pot) is almost zero. One or two messages a week. I have actually now turned off IMF on my Exchange servers.

Why is this being so effective?
The simple reason this is being so effective is that the spammer's list of email addresses will contain a mixture of valid and invalid addresses. As soon as the spammer's server attempts to send an email to a non-valid address that is on my honey pot list, it is blacklisted. Even if that IP address subsequently tries to send to a valid address it will be blocked.
Combined with greylisting, which sends away the initial connection, the even if a legitimate address is used first, the spam doesn't get though. The first attempt is greylisted, then if the list of email addresses contains one of the bogus ones, then it gets blocked. The server attempts to deliver again after greylisting and its connection is blocked.

I also think this is more effective than regular blacklisting because it is in real time and is based on email received by my servers.

I have combined this with an SQL backed database for Vamsoft ORF so that both of my SMTP gateways share the same information, meaning that a blacklisting that is set by one server, is also used by the other.

Finally, I have also combined this with custom NDR text, that points people to a special page on my web site. This page explains what is happening, and other ways to contact me. If required, I can then white list to allow the legitimate messages through and take the spam hit for a short time.

Vamsoft ORF: http://www.shareit.com/product.html?productid=169362&affiliateid=200023740

Having now completed a few Exchange 2007 deployments, upgrades and consulted on a few more, not one of them has featured an Edge server.

Which made me think, what is the point of Edge services for most users?

To use Edge you need to purchase another Exchange 2007 license which isn't cheap. What do you get for your money? Simply the ability to put a machine in a DMZ or similar network.
The edge server is just for SMTP traffic, but the most common concern I hear is for people worried about web traffic and therefore they want to put OWA in the DMZ. With Exchange 2003 this would have been a frontend server, although it is a bad idea to try and put an Exchange 2003 frontend server in to a DMZ.

The anti-spam agents that are installed on Edge can be installed on to another server by simply running a Powershell script, therefore the need for the Edge becomes less. All that it does is move where the spam filtering takes place - and if your main Exchange 2007 server is exposed to the internet then you haven't really lost anything, other than the warm fuzzy feeling that your Exchange server is not directly exposed to the internet.

If Edge was more like ISA, but for Exchange exclusively, so allowing you to have OWA in the DMZ with the small number of ports open similar to what Edge currently requires for SMTP traffic, then it would become something worth considering.

At the moment, if you want to protect SMTP traffic then you have more options if you do NOT use an Edge server. Instead install a standard Windows 2003 Server with IIS. That gives you options to use most third party products that offer a gateway facility.

I have built a few using a third party tool on top of IIS called Vamsoft ORF. This provides the basic option of recipient filtering via an LDAP lookup and can also do greylisting. There is an article on my other site that discusses building this type of server: http://www.amset.info/exchange/gateway.asp
With that product you can even integrate Antivirus software as an agent. Pick up a single copy of a server product different to what you are using internally and you have the multi layer protection that you should be aiming for.

Even after the purchase of Vamsoft ORF and another AV product, you are still easily within the cost of another Exchange 2007 license.

Furthermore by using Windows 2003 standard - i.e. 32 bit software - you could use an old server that you are removing from another role without having to purchase something new. It is a basic configuration, so if the server fails easily replacing it would be simple. You could even put the gateway functionality in to a virtual machine and keep a copy of it. If the physical hardware fails then simply copy the virtual machine on to the replacement hardware.

I have heard from many sources that grey listing can be an effective weapon for fighting spam, yet hadn't had an opportunity to try it out.

However one of my clients was being hammered hard with spam, with 700+ messages a day being filtered by Intelligent Message Filter, and lots of messages getting past "I Hate Spam" from Sunbelt Software. Therefore I thought they would be a good site to test the method with.

I installed Vamsoft's ORF (http://www.shareit.com/product.html?productid=169362&affiliateid=200023740) on to the gateway machine and left it to get on with it, enabling just the grey listing and the automatic white list* feature.

* The automatic white list is built by watching outbound email and recording the email address used. When the external recipient replies, the message comes straight back in as the server then knows that the email address is legitimate.

The effect was immediate and noticeable. I watched the logs of the application very carefully to ensure that no legitimate email was being blocked. The amount of spam that was blocked by the application was considerable. After a running a week, the application reported that over 85% of all email that was being received was spam.

That doesn't count messages that were dropped by the filter on unknown users (http://www.amset.info/exchange/filter-unknown.asp).

The process isn't 100% effective, IMF was still catching some messages - but this was down to 20 or 30 a day, a massive reduction in the pre-grey listing number.
Users were also reporting that a few items were reaching their inboxes, but nothing like the level they had been receiving.

I have since deployed the application on four other sites, including my own Exchange server and seen similar significant drops in the number of spam messages being received.

As with user filtering, this technique also saves the bandwidth, as the messages are not even delivered to your server, so don't have to be processed.

The Vamsoft product works with any IIS SMTP mail server, so if you have Exchange 2000 then you can use it as well. It also features Active Directory filtering, which Exchange 2003 has built in, allowing users of the older version of Exchange to benefit.

How Does It Work?

Grey Listing is very simple idea.

A server attempts to deliver the message to the server. If the server hasn't received an email from that sender before, then it rejects the message with a temporary failure.

The systems that spammers use don't care about failure messages. They aren't interested in the failure and will therefore not try again. Spammers want to drop and run, before any system blocks the IP address that they are sending their email messages from.

However a legitimate email server will try again. Most email servers will try again for up to 48 hours, so you will get the email message eventually.

Are there any risks?

Any anti spam technique comes with risks. Unless you have a human looking at every message, you are relying on the computer making the decision whether the message is spam or not.

This technique will introduce a delay for new email messages - I have seen the delay as short as 90 seconds up to 20 minutes or more. If your business cannot tolerate any delay in email message delivery then this technique is not for you.

I have also seen a few email messages fail to be delivered from some sites that generate large amounts of email - such as eBay and a few ISPs. This is because each message appears to come from a different IP address in their server cluster.
With eBay, white listing their domain isn't advised as that will also allow in phishing emails.

Conclusion

While spammers don't comply with the RFC on SMTP email delivery and try just the once to deliver their email messages, this technique will be an effective first strike weapon in the war on spam. It shouldn't be considered the only weapon, but combined with other techniques can make spam more manageable.