Microsoft Exchange and Remote Desktop Services Specialists

SEMblog

Microsoft Exchange Server and
Blackberry Enterprise Server news, views and fixes.

One morning you find that there is spam in the queues, your server has been blacklisted etc...

One of the worst experiences for an Exchange administrator is to come in one morning and find that either email is being blocked, the queues are long or the users are getting NDRs saying that the server is blacklisted.

This seems to result in confusion amongst administrators who then go looking for advice only to get conflicting answers on what the problem might be.
I am going to try and clear up some of that confusion which should help Exchange administrators find the source of the problem.

There are two main issues that Exchange administrators seem to see and fail to understand.

  1. There are a large numbers of messages in the queues.
  2. The IP address of the server has been blacklisted.

In both of these occasions many administrators seem to think that a client machine on their network has been compromised and is sending email through the Exchange server.

This is not the case.

To abuse an Exchange server in this way, a BOT writer would need to

  1. get the BOT inside the network
  2. infect the machine
  3. realise that it is on a corporate network where there is an Exchange server
  4. find the Exchange server
  5. send the message.

The above, is not going to happen - at least not at the moment. Too much like hard work. The first two are the most difficult - if the network security has been configured correctly and the users trained to recognise potential suspect emails or web sites.

Then sending the message requires either a MAPI interface or SMTP to be configured on the Exchange server to allow users to relay through the server. While this is default, if you do not have any users who need to relay through the server (Outlook, OWA and Windows Mobile/Blackberry BES users do not need to) then you should disable it.

Then for a successful infection and abuse, the above is also presuming that the user is an administrator and the network admin will not notice the infection!

What the BOT writer is really looking to do is infect clueless home users who are not keeping their machines patched, not using security software and are running as a local admin. Much higher chance of success there involving simpler techniques.

Therefore with the target in mind, the BOT will usually have its own SMTP engine and will be sending out email directly to the internet.


So what has happened?

If you have been blacklisted but the queues are clear, then a client machine has probably been compromised. This is often the case when you have a single IP address on the Internet which is shared among all machines on your LAN.

However to further complicate things - if you are using a smart host - such as your ISPs SMTP server - then your queues could be clear but the server is still being abused. However in that scenario it is likely that your server would not be blacklisted on public lists, but your ISP may have noticed and not be very happy with you. If messages are not being delivered to the smart host then phone your ISP and ask - or they may phone you. Often ISPs will block first and ask questions later.

Finding the Source - Compromised workstation

A quick and dirty method to find the compromised machine is to simply stop Exchange from sending any messages by freezing the outbound traffic, and then block port 25 on the firewall and wait. A compromised machine will quickly show on the logs when it cannot connect. You can then go and find the machine and deal with it.

Having up to date Antivirus is not enough. Once the BOT is on the machine, it is no longer your machine. The only way to ensure that it is clean is to wipe the machine. BOTs are very good at hanging around and they will update themselves regularly.

There is a complication on this as well - if you have been foolish enough to browse from the Exchange server then the server itself may have a BOT and be sending out messages. However those messages would still not show in the queues. If you don't browse from the Exchange server then that shouldn't be the cause of your problems.

Finding the Source - Large Number Of Messages in the Queues

If you have a large number of messages in the queues, then those will be coming from outside your network. That does not mean you are an open relay, there are other ways that the spammer can abuse your server.

The two most common are authenticated relaying and the NDR attack.
I have discussed these in more detail in my spam cleanup article on amset.info - http://www.amset.info/exchange/spam-cleanup.asp .

However in short, authenticated relay is where the spammer has attacked your SMTP port trying to break a password - usually the administrator account. Once broken, the account is used to relay email. Authenticated relaying is enabled by default.

An NDR attack is where messages are sent to your server to non-existent users on purpose. Either as a directory harvest attack (to see what users are valid) or to get your server to bounce the messages to the "sender". The sender is spoofed and is the actual target.
Exchange 2000 is unable to defend itself against these kinds of attack without third party support. Exchange 2003 and higher has features built in to deal with this kind of threat, however if you have Exchange 2003 on Windows 2000 then you should not use them as Windows 2000 is unable to defend itself against a directory harvest.


So what do you do?

When you first notice there is a problem, you need to verify whether it is the result of an attack or compromised machine, or the result of a configuration error or change. Do not presume one or the other.
Once you know which it is then you can look further.

If you are dealing with an ongoing problem then pull the plug on the internet connection. That will stop messages going out and if the spammer is abusing your server will stop the messages from piling up. This will give you some breathing space to clean up and see what is going on.

If your IP address has been blacklisted, then use your ISPs SMTP server to send email through.

Ideally you should have at least two IP addresses so that the Exchange server can have its own address. If a workstation is then abused it does not result in your email IP address getting blacklisted.

Remember, any SMTP server is a target for a spammer. They don't want to use their own resources, they want to use those that belong to someone else so that they can hide.

Forthcoming Speaking Engagement - UK Community Day - 8th and 9th April

Once again the UK IT Pro User Groups are getting together to have a community day at Microsoft in Reading. This time it will be spread over two days - 8th and 9th of April 2008.

I presented at one last year alongside Nathan Winters, but this time I have my own session. I will be presenting a session on behalf of the Microsoft Messaging & Mobility User Group (MMMUG - http://www.mmmug.co.uk/) on the first day on the subject of Client Access to Exchange 2007. This will include unified messaging with Outlook Voice Access, OWA, Windows Mobile and Outlook 2007. Particularly emphasis on what is new on Exchange 2007 SP1.
The idea is to show you the different ways that you can access your email, then look at the control the Exchange administrator has over those interfaces. This will be a hands on session, rather than something that is just a serious of Powerpoint slides.

As well as my own session I will be there all day sitting in the other sessions, the product group Q&A and the end Q&A at the end of the first day.

The event is free of charge, but you do need to register in advance. Full details of the agenda for the two days, the speakers and their sessions, plus registration details can be found on the web site: http://www.ukusergroups.co.uk/

Why You Shouldn't Enable the POP3 Server

This is another post in my series of articles on why you shouldn't use certain features in Exchange, even though they are there. As with the other articles, the article does NOT tell you how to enable the feature in question.

The other articles in this series are:

- why you shouldn't use the POP3 connector: http://blog.sembee.co.uk/archive/2006/09/25/28.aspx
- home grown versus commercial SSL certificates: http://blog.sembee.co.uk/archive/2006/03/05/9.aspx

For this article I am going to outline some of the reasons why you shouldn't enable the POP3 server on the Exchange server. This is different from the POP3 connector, which is used by Small Business Server to pull in email from an ISP. This is POP3 used to collect email from Exchange.

Why is POP3 enabled?

POP3 is not enabled by default and there are a number of reasons why it is enabled.

Some administrators enable it because that is all they know, and they want to use Outlook Express for email. This is familiar to them and their users.
Others enable it for remote access, because either they don't know about or cannot use RPC over HTTPS (aka Outlook Anywhere (Exchange 2007) or Outlook over the Internet (SBS)) http://www.amset.info/exchange/rpc-http.asp

It may also be enabled to allow other non Microsoft clients to access email.

However in most cases it is a request from a user, who may or may not be completely up front about why they want to use POP3. POP3 can be used/abused in so many ways that it is one of the reasons why the Exchange server admin should really think twice before enabling it.

Therefore the first thing an Exchange Server admin should do when they are asked to enable the feature is ask the question why. If the asker then goes coy, you know it may not be in the business' best interest to enable it.

Why Shouldn't You Enable the POP3 Server

There are any number of reasons why the POP3 server should not be enabled. These are the most common reasons why not:

  1. Username and Password Sent in the Clear.
    In the default configuration, POP3 sends the username and password across in the clear. That is a security risk. If you really do need to enable it, look at using SSL to secure it. 
  2. Risk of content loss.
    POP3 is designed to REMOVE the content from the server and store it locally. It is too easy for a user to download the content and remove it from their mailbox. While there is a setting to leave the email on the server, it cannot be controlled server side, so you are reliant on the user setting the client in the correct way.
    As the data is being stored locally, it cannot be backed up easily, therefore if the user loses their machine, it is stolen or suffers a hardware failure, then the email is lost as well.
    I have also seen it abused, as a way to get content out of the network - sales people in particular want the feature so they can store a copy of everything at home. A significant number of sales people are not loyal to their employer at all, and would prefer that their clients do everything through them, ideally on a personal email address.
  3. Loss of control of access.
    Once POP3 is enabled, it can be used by any number of things, PDAs, phones, Blackberry etc. The IT department may or may not know about those, and be unaware of them in the event that they cause a problem. 
  4. Storage and Regulatory Compliance Concerns.
    If you operate in an environment where you have to store email or be aware of the content of the messages, then that is a big argument not to enable POP3.
    If the client is configured for POP3 and email is sent from that client, then there is no way it can get back in the store unless it is imported. If the user is sending email with something obscure, then that isn't going to happen.
    The user could also be sending email out through another SMTP server, even sending email with their personal email address as the reply to address (again sales type people are notorious for doing this - then claiming it was an "accident").
  5. Feature Loss
    You also lose the GAL, calendaring and everything else that Exchange offers. If the email is being extracted then OWA becomes close to useless, no sync to Windows Mobile devices over the air.

If your must enable the POP3 Server

As with many things in the Microsoft world, everything is enabled until you turn it off. POP3 access is no exception. If you enable the server then all users will be able to use POP3. Even if you don't publish the information about how to configure it, if OWA is used and the port is open, the users will soon work it out - the information is all over the internet.

Therefore if you must enable it then you should secure it.

  1. Use admodify.net to disable the feature in bulk for Exchange 2003 users, Exchange Management Shell for Exchange 2007 users. Then enable it for the users who need it only.
  2. Use SSL and only open the POP3S port (995). That will slow down a causal user.
  3. If you can, use IMAP instead. That leaves the content on the server. It isn't perfect as there is still the chance that email is sent out via another SMTP server, but it is better than POP3.

Conclusion

Make sure that management are aware that it is being enabled and are aware of the risks that are involved. If they say no (which is ideal) then you can simply turn down future requests with that same message.

My personal opinion is that POP3 has no place in a corporate email environment and there is no need to enable it at all.
If you need to provide access to mobile devices, then purchase suitable devices that use either Windows Mobile or Blackberry.
If you need to provide access to non-Microsoft clients, use IMAP.
If you need to provide remote access for Outlook, then use RPC over HTTPS.

Share an Exchange 2007 Server (UK Only)

Would your company like to use Exchange 2007, but are finding the costs too high, you don't have the internal skill set or just want to outsource it?
However have you found that hosted Exchange is too limiting for your company needs or you want a more personal approach to the management of your server?

If so, then we may have the answer.

I have recently been talking to a few clients who would like Exchange 2007, but for various reasons cannot justify their own server. They have also expressed a desire for it to be managed by someone they can get to know, rather than a request going in to a helpdesk queue and being completed by an unknown person. 

Therefore what we have talked about is a number of companies getting together to share an Exchange server and the management costs. This server would have a limited number of users, and would be managed by myself. My company would acquire the hardware, arrange hosting at a data centre, setup the server and then manage it.

However to make it worthwhile on costs, time and other investment, we need a few more mailboxes. Ideally we are looking for around 200 mailboxes, we currently have expressions of interest for around 75 mailboxes.

The monthly cost that is currently being looked at is £15 per mailbox per month, with a £100 per client per month management fee and maybe a setup fee. Numbers are not exact as it depends on how many mailboxes we get. If we get 400 or more, then multiple servers could be used, which will bring down the expense as the cost of the domain controllers and additional network hardware will be shared between more users.
We would also need to have a 12 month commitment to the service so that financing etc of the software and hardware can be arranged with some idea of the income flow.

At this time it is planned that each mailbox would have 2gb of space, plus there would be public folder space as well.

If you are interested, then please let me know through the company email address of contact @ amset.co.uk with the number of mailboxes you may be looking to host and whether you would be interested in Blackberry support, and the number of devices. We must ask that you do not contact us if you are outside of the UK, unless you have a UK billing address and the majority of the users will be located in the UK.

Please note this isn't going to happen overnight, once the legal stuff has been dealt with, the hardware needs to be acquired and setup, so it could be early April or later (at the time of writing) before we are ready to go.

Support for the migration from your existing solution should be included - although it depends on what you are currently using.

I appreciate that much of the detail is not exact, at the moment we need to find out how many others could be interested before proceeding any further.
I have written a brief FAQ below which should answer some common questions, although if you do have any queries, please contact me on the above address and I will attempt to answer them and also update this page.

AT THE TIME OF WRITING THIS IS NOT A SOLUTION YOU CAN BUY FROM US TODAY.

PLEASE NOTE THAT FOR LEGAL AND INSURANCE REASONS THIS SERVICE WOULD ONLY BE AVAILABLE TO UK BASED COMPANIES.

FAQ

Q: Isn't this Hosted Exchange?
A: It is a form of hosted Exchange, and we will be using the Microsoft Hosting licensing system to license the software. However the idea is to offer a service that is more flexible than those offered by Hosted Exchange providers because there is no control panel. Furthermore you know who is managing the server, that they built it and are aware of how it is working. I see it as taking the best bits of Hosted Exchange and having your own server, and putting them together.

Q: What don't we get that we would get with Hosted Exchange/Our Own Server.
A: You don't actually loose a great deal.
From a hosted Exchange point of view, you will not get a control panel or access to any kind of administration interface. Anything you want done from an admin point of view will need to be asked for and I will make the change for you - just as it would if you had your own server - you would ask your network admin or support company.

Things missing from having your own server will include your choice of antivirus and antispam, as we will need to use a solution for all users as it protects the server. You also don't have access to the admin console yourself.

Q: Will we see the other clients in the GAL etc?
A: No. Address list segregation will be used to make it appear to be your own server. While this isn't a traditional hosted Exchange environment, I will be using the techniques from Microsoft on setting up a hosted Environment to provide a secure deployment for all users.

Q: Will it be secure?
A: Yes, this will be a deployment done to best practises. Commercial trusted SSL certificates will be used, behind firewalls with the relevant ports open. It will be just as good as a deployment in your own office.

Q: Will we have access to all features? OWA, Windows Mobile support?
A: Yes. Everything Exchange offers will be there, except for Unified Messaging - see below.

Q: Blackberry?
A: Maybe. There are other issues with Blackberry, such as support for Exchange 2007 SP1 and paying for the licenses of both the server and the CALs. If you are likely to be a user of Blackberry, then please indicate that along with how many devices.

Q: What about Unified Messaging?
A: To begin with there will be no Unified Messaging support.
However I am already looking at how UM could be used with remote server for another client. This could be possible if you already have VOIP technology in use or by hosting the media gateway at your own site. That may mean having a different type of Internet connection in to your own office, and maybe increased bandwidth costs for everyone involved.
There are also security concerns to be addressed, so use of UM may be possible long term, but not at the start.

Q: Contract, SLA etc.
A: Can't answer questions on those bits yet, as that needs to be worked out if we went ahead with this project. There will be some kind of contract and SLA, however those details would need to be resolved once the project starts. That would also include support details, how to make requests, track requests etc. The operational details are a long way away.

Q: What about if you are not available?
A: Finding someone who back up me, in case I am not available to look after the server for whatever reason. Whoever I choose to use will be of high quality - I have very high standards and you will know who it is.

Q: Backups?
A: There will be some kind of backup solution, exactly what I do not know at this time. Certainly Exchange options will be used where possible, and then some additional backup will be used to protect the data in the event of server failure.

Q: It is more expensive than x service provider.
A: That maybe so. However this is highly customised solution with support from a named individual. This is not a "pile it high sell it cheap" solution based on price. This is a quality solution. I would compare it to buying a car, such as a 1978 used Mini to a brand new Mini. Same name, both cars, but very different in what you would expect.

Q: So what do we get that we wouldn't if we had our own server?
A: The first thing is less worry. Someone else worries about the server, the data, whether it is working correctly, bandwidth and use.
Next, you have peace of mind that it is managed by an experienced Exchange consultant, which is not something you may well expect to have if you had your own server. No need to worry about someone who doesn't know what they are doing playing around the with the server.
The server will be located in a data centre, so it will be protected and available to you where ever you are. If you have a high number of users out of the office, it may well be a better performing solution than hosting your own server.
There could also be opportunities to enhance the solution buy purchasing additional software products on a per server basis. While the cost may not be economical for 20 users, for 200 it becomes something viable.

Improve Exchange 2007 MC Performance - Use Windows XP

If you have used Exchange 2007 for anything more than 2 minutes, you will quickly notice that the performance of the Exchange Management Console (EMC) is very poor - no matter how much memory or processing power you have on your server.

While moving content on the blog, I was reminded of a post where I linked to an announcement about the Exchange 2003 management tools being made available for 32 bit OS. (http://blog.sembee.co.uk/archive/2007/01/25/39.aspx). This reminded me that I had a virtual machine with the management tools installed. Having just updated my main Exchange 2007 server to SP1, I decided to update that virtual machine as well with the 32 bit service pack. Imagine my surprise when I started the tools to find that they started much quicker than on my server. Checking other Exchange 2007 servers (include a quad processor with 8gb of RAM) I found that it still loaded quicker on this low spec virtual machine.

The Exchange Management Shell also seems to load quicker on a Windows XP machine. However that does mean you have to specify the Exchange server that you are using when you run any PowerShell commands for Exchange. If you are on a single server site then the quickest way I have found to do this is to start the commands with "get-<server role> |" then the rest of the command as normal. For example "get-mailboxserver | get-mailboxstatistics"
Replace mailboxserver with the role that you are working on. 

To install the tools on to a Windows XP machine, download the 32 bit version of Exchange 2007 Service Pack 1.

You will also need to install the base IIS components (not SMTP though) and some additional downloads including MMC 3.0, PowerShell 1.0 and net framework 2.0 if you haven't already installed it.

After installing the net framework, run Microsoft Update to ensure that you have the required updates. If you don't, then the Exchange 2007 setup will prompt you to install the update before you can install the management tools.

When it comes to the actual installation, carry out a custom install and choose the Management Tools only.

If your Exchange server has plenty of resources, and you have the licenses to do so, then rather than installing the Management Tools on to your own workstation, you could use a Windows XP virtual machine which is stored on the Exchange server. This is particularly useful if you are using the best practises for permissions have separate Administration and User accounts. If you usually access your Exchange server using RDP, then enable remote desktop on the virtual machine Windows XP installation and then connect to it directly. As it is stored on the server, it will always be available to you.

Links to Downloads Required

Exchange 2007 SP1: http://www.microsoft.com/downloads/details.aspx?FamilyId=44C66AD6-F185-4A1D-A9AB-473C1188954C&displaylang=en
MMC 3.0: http://support.microsoft.com/default.aspx?kbid=907265
PowerShell 1.0: http://support.microsoft.com/default.aspx?kbid=926139
.net Framework 2.0: http://www.microsoft.com/downloads/details.aspx?familyid=0856eacb-4362-4b0d-8edd-aab15c5e04f5&displaylang=en