Microsoft Exchange and Remote Desktop Services Specialists

SEMblog

Microsoft Exchange Server and
Blackberry Enterprise Server news, views and fixes.

Monitoring Exchange Service Availability from the Internet

When it comes to monitoring Exchange, there are multiple choices, involving products that can monitor every aspect of the platform. 

However sometimes it is useful to have a quick overview of whether the Exchange services are available, and with the increasing dependence on mobile working, whether they are available from the outside world. 

Fortunately, Exchange 2013 and higher provides a built in mechanism that allows you to build a quick and easy overview of Exchange using a free monitoring service.

The External Service

For the purposes of this article, we are using Uptime Robot https://uptimerobot.com/ . This provides a five minute check for up to fifty checks, and also provides a publicly available status page. That is useful to distribute to end users to check whether Exchange is available or not.

The Exchange Mechanism

Introduced with Exchange 2013, each of the core Exchange web based services has a health check page. This can be checked with no authentication required and will confirm if the service is working or not. Primarily designed for use with load balancers, so they can check if the server is available, it is also useful for the purpose of monitoring. 

If you browse to the following URL, then you will see what I mean: https://host.example.com/owa/healthcheck.htm (where host.example.com is the host name used by your Exchange server). 

You should see something like this:


The full list of URLs that can be checked in that way are:

  • Outlook Anywhere (aka RPC over HTTP): /rpc/healthcheck.htm
  • MAPI/HTTP: /mapi/healthcheck.htm
  • Outlook Web App (aka Outlook on the web): /owa/healthcheck.htm
  • Exchange Control Panel: /ecp/healthcheck.htm
  • Exchange ActiveSync: /Microsoft-Server-ActiveSync/healthcheck.htm
  • Exchange Web Services: /ews/healthcheck.htm
  • Offline Address Book: /oab/healthcheck.htm
  • AutoDiscover: /Autodiscover/healthcheck.htm
Setting up the Checks

The type of monitor you want is "Keyword". Enter a friendly name for the monitor - this is what will be seen by anyone browsing to the public status page. 

For the URL, use your public URL, in the format above. 

For the keyword, use the name of the server as it appear when you browse to the server. In most cases this will be the server name in all upper case. That the name doesn't resolve externally doesn't matter. You need to change it to "Keyword NOT exists". 

You can then add an alert contact. An external email address would be a good option here, but there is also an App available. 


Repeat for all of the URLs listed above, until you get a list like this:


I have added in a check on the SMTP port as well. If you publish IMAP/POP3 then add those ports too. 

That is all there is to it. If you stop the IIS services then you will get a screen like this: 


Within the Uptime Robot site you can then create a public status site - which uses a CNAME on your domain to point to a site on their server - secured with an SSL certificate by default. 
This is a link to a screenshot of the status site for the above checks:

This URL can be included in any pre-populated bookmarks that are pushed to users - including on phones using your MDM product, so end users can check whether Exchange is available and whether it has been down earlier. 
Furthermore, as it is an externally hosted service it also can give you a quick overview from inside the office if things are working correctly. Just remember to add the same internal CNAME entry if you are using the same domain internally and externally via split DNS. 

What if you are behind a load balancer already?

In that case, give each server a unique URL (which is good practise anyway) and then configure Uptime Robot to connect to it directly. The IP addresses for the monitors are on their web site: 

A similar practise would allow you to monitor Exchange if behind a device in a DMZ. 

The SMTP Port Monitor doesn't confirm it is working? 

Indeed that is the case - Transport can often stop accepting emails, but will still have the port open. For proper testing of SMTP traffic, you need to use a round trip mail monitor. https://www.everycloudtech.com/free-mail-flow-monitor . That will alert you as the administrator directly if mail flow stops for whatever reason. Again use an external email account, rather than one on Exchange. 

Exchange 2016, Windows 2016 and Macs

Now starting to see more implementations of Windows 2016 and Exchange 2016, so odd issues are starting to come to light.

One that I have seen a few times is a problem with Macs connecting to EWS on the 2016 versions of Exchange and Windows.

Going through the logs, there are 401 errors (unauthorised), yet the same credentials work with OWA.

Further troubleshooting with the web server and a hint on an Apple forum suggested that Windows 2016 was using HTTPS 2 by default and that the Mac was having some problems working with that for authentication.
The fix was to downgrade to HTTP 1.1.

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\HTTP\Parameters

New DWORD values

EnableHttp2Cleartext
EnableHttp2Tls

Set both to a value of 0.

The first one is for HTTP, the second for HTTPS. I set both, even though in most cases only HTTPS is allowed to the server and being used.

Reboot the server (restarting the IIS services does not appear to work).

Here is the registry value content to create a reg file for easy installation:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HTTP\Parameters]
"EnableHttp2Tls"=dword:00000000
"EnableHttp2Cleartext"=dword:00000000

From an operational point of view, I cannot see any loss of functionality. OWA is designed to work with Windows 2012 R2 which still uses HTTP 1.1. In very large environments you might see a small performance hit because of the loss of the optimisations provided by HTTP 2, but most systems will not see anything.

Exchange 2016 DAG - Move Active Database Failed

Spent most of the week dealing with a flapping DAG database - flipping between two servers, which turned out to be a bad network cable.

 

Anyway, while trying to get the databases to activate correctly during troubleshooting I hit this lovely error (real server name/database changed).

 

Error: Mailbox Database 1

An Active Manager operation failed. Error: The database action failed. Error: Move for database 'Mailbox Database 1’ was suppressed because too many moves have happened recently. 3 moves have happened within 01:00:00. [Database: 'Mailbox Database 1', Server: exch1.example.com]

 

Basically tried to activate a database three times in a hour and Exchange stops it from happening again.

 

Off to PowerShell and skip the checks:

 

Move-ActiveMailboxDatabase -Identity "Mailbox Database 1" -SkipMoveSuppressionChecks -ActivateOnServer exch1.example.com


Fixed

Greylisting and Honeypot IP Whitelists - Vamsoft ORF

Many years ago I wrote about how I was getting good results with the anti-spam technique greylisting. It is still a technique I use with many clients. 

However with the increasing use of cloud based services, I have found that greylisting can delay legitimate traffic, because the email can be delivered from different IP addresses on each attempt. 
While Vamsoft does have an option to Accept delivery retries from the same 24 subnet, this is not always effective because the large providers have bigger IP address pools. 

Therefore I have started whitelisting the major providers within the Vamsoft product. 

Another feature I use with Vamsoft is their Honeypot function - the same lists for greylisting I am using with this feature as well - as a single bad address will cause a lot of senders to get blocked. 

Getting the Lists of IP Addresses

The first thing to do is get the IP addresses. I am putting in Office365, Google Apps, Mimecast and Amazon SES. If you have senders on other cloud providers then you should add those addresses as well. 

For Office365 and Mimecast, the list of IP addresses is on their web site. 


Google Apps

For Google apps, you need to do an NSLOOKUP to get the current list. 
First query their SPF record:
nslookup -q=TXT _spf.google.com 8.8.8.8

Then query each result, which at the time of writing was this:

nslookup -q=TXT _netblocks.google.com 8.8.8.8
nslookup -q=TXT _netblocks2.google.com 8.8.8.8
nslookup -q=TXT _netblocks3.google.com 8.8.8.8

Although at the time of writing, Netbblocks 2 is ipV6, which you may not need. 

Amazon SES: 
Similar to Google, query their DNS records:
nslookup -type=TXT amazonses.com | find "v=spf1"

Entering the Lists in to Vamsoft

Once you have the lists, you are ready to put them in to Vamsoft. The GUI I find is a little cumbersome for this task, and if you have lots of servers will take a long time. Therefore modify the configuration file instead. 

First, check whether you have any IP addresses in the white list - Blacklists --> Greylisting --> IP Exceptions. If you don't, add one, as this will create the relevant part of the configuration file and the format. 

Next, option an elevated command prompt and enter this:

notepad "c:\Program Files (x86)\ORF Fusion\orfent.ini"

Then look for the section 

[GreylistingHostExceptions]

The following the format, add the IP addresses like this:

101=V5"23.103.132.0/22","Office365"

The number at the start has to be unique. I usually start at 101 as it will ensure it doesn't conflict with any existing entries - usually creating it in a separate Notepad file and then copying the result in to the configuration file. 

Save the configuration file and close it. Finally start the Vamsoft Administration tool and check the list has your addresses in it. If it does, save the file, which will sort out the numbering for you correctly. For the Honeypot feature, repeat above, but instead put the same list of IP addresses in to the section headed
[HoneypotIpExceptions]

Check the lists regularly - as the providers will add additional IP addresses and you need to update them. 

Self-Contained Exchange Server - Mixing Cloud and On Premise

Over five years ago I wrote about a self-contained environment I built for a small business where  they had no office of their own. A new client recently contacted me and asked if I had done anything similar recently, but using more up to date technologies. With the growth of cloud tech, Office365 etc, things have moved on.

This particular request was to provide Exchange for a project which was quite sensitive and the client didn't want to put the data in to Office365, but was quite happy to put it in to a private cloud using a dedicated server. It needed to be completely self-contained. No problem, as that is how I build my labs, so it was just scaled up. This is what I proposed and was deployed at the beginning of September.

Hardware
Dedicated Server rented from a major host here in the UK (I can actually tell you where the server is located). Fairly standard specification, dual RAID 1 arrays, 32gb of RAM.

Software
Installed on to the physical server was VMWARE 6.x.

VM Guests
Into that VMWARE server I installed the following guests

  • Pfsense. This provided the firewall for the entire environment, and once the builds were complete, the VMWARE admin console was put behind this as well.
  • Windows 2012 R2 DC (8gb RAM) Fairly obvious one - separating the Exchange server and the domain controller.
  • Windows 2012 R2 Exchange 2016 (16GB RAM) This was the main Exchange server.
    • Exchange 2016 Latest version of Exchange, naturally.
    • GFI Mail Essentials Providing malware, spam and attachment filtering, plus automatic signatures.
    • SSL Certificate - from http://certificatesforexchange.com/
  • Observium monitoring appliance from Turnkey Linux (open source) This provides a good overview of the virtual machines. The host was also kind enough to setup a read only user on the IPMI interface of the server.
  • Windows 10 Pro workstation (4gb RAM) This had Office 2016 installed on it, along with some other tools to allow testing of the implementation from the server itself. It also provides a landing point should one of the end users need to access the server and doesn't have the tools available immediately.

The Windows servers also got the various monitoring tools I use with my Exchange clients. Backup to the cloud, using Exchange aware backup application was also provided.

Microsoft Office

Shortly after deployment, it became apparent that the clients were a complete mixture of Office versions, some of which didn't support the latest version of Exchange. Therefore I proposed, and was accepted, that we used Office365 Business subscription. This provided Microsoft Office for both the Windows machines used by the users, plus their tablets and phones. I integrated the domain I built with their new Office365 subscription providing a single username and password experience - the size of the deployment didn't justify a single sign on implementation. Should someone leave the project, we simply un-licence their Office installation.

Costs?

All numbers correct at the time of writing (September 2016) and are excluding VAT.

  • Hosted Server: £140 a month
  • Per user licences: £25 a month (covers Exchange, Windows Server etc)
  • Office365: £7 a month per user
  • Server management Fee: £350/month
  • Setup: £1500 (includes hosting company setup charge and my time). 
  • SSL Certificate: £35/year.

Conclusion

The client has a solution that they can scale up and down as the project progresses, which fulfils their requirements of being a self-contained standalone solution, without the cost of the hardware and software up front. It is also managed for them by Sembee Ltd with responsive monitoring.