Sembee Blog

An interesting little issue with a client's configuration caused a problem recently.

The problem only affected users off site using Outlook Anywhere. While they could get their email correctly, the availability service didn't. This stopped Out of the Office from working correctly unless OWA was used, or the end user was in the office.  

This particularly configuration uses a Client Access Server in a data centre, which proxies over a site to site VPN in to the main office where another CAS, plus the mailboxes are actually located. Therefore the issue had to be around a configuration difference between the two servers. 

Running 

get-clientaccesserver servername |fl 

on the server in the data centre and comparing it to the server in the main office, showed that the value for AutodiscoverSiteScope was populated with the AD site for the main office. This was because the server in the data centre had been built in that location initially and then moved. 

Removing that entry so it was blank resolved the issue:

Set-clientaccessserver servername -AutodiscoverSiteScope $null 

A five minute fix resolved an annoying problem for the end users. 

Exchange 2007, Exchange 2010, MS Exchange Server, Outlook Exchange 2007, Exchange 2010, MS Exchange Server, Outlook

If you were affected by the Blackberry Internet Service outage today (10th October 2001) and your Blackberry connects to an in-house email server running Exchange server (2003 or higher), then you really should be running a BES (Blackberry Enterprise Server) or BES Express (BESX).

A Blackberry connected to a BES/BESX gives you the full functionality of the Blackberry with true two way synchronisation of Email, Contacts, Calendar and Tasks. It is an extension of your Inbox. No need to maintain two sets of data that kind of synchronises. 

If you use BESX, then the software is free and you do not have to change your device subscription/tariff. For smaller installations the software can be installed on your server in  a few hours and give you complete control over the devices that connect. 

If you are in an industry where the email traffic is sensitive, the data exchange between your Blackberry and the BES/BESX cannot be intercepted as the encryption is managed by your server, not the one at RIM. This provides a more secure mobile email solution. 

Through my company Sembee Ltd, I can install and configure a BES Express for you for just £250 plus VAT if installed on to an existing server (other terms and conditions apply). That includes post installation configuration and guidance on maintenance, handset setup etc. 

For more information, contact me through the company web site at http://www.sembee.co.uk/ 

Amset IT Solutions Ltd. / Sembee Ltd., Blackberry, Exchange 2003, Exchange 2010, MS Exchange Server, Small Business Server Amset IT Solutions Ltd. / Sembee Ltd., Blackberry, Exchange 2010, Exchange 2007, Exchange 2003, MS Exchange Server, Small Business Server

During a recent migration from Exchange 2007 to 2010 I found I was unable to remove the public folder store from the Exchange 2007 server. 

It was returning the following error when using remove-publicfolderdatabase or using EMC on Exchange 2007. 

Remove-PublicFolderDatabase : Object is read only because it was created by a future version of Exchange: 0.10 (14.0.100.0). Current supported version is 0.1 (8.0.535.0).

Obviously the Exchange 2010 server had touched the database in some way, probably due to the Offline Address Book migration. 

The fix was quite simple - remove it using the Exchange 2010 Exchange Management Shell. Can't use the GUI as the Exchange 2007 public folders do not appear in there.

Get-PublicFolderDatabase -Server EXCH2007 | Remove-PublicFolderDatabase

Where "Exch2007" is the name of the Exchange 2007 server. 

After removing the database I refreshed the GUI and was then able to drop the Storage Group and complete the removal of Exchange 2007. 

Exchange 2007, Exchange 2010, MS Exchange Server Exchange 2007, Exchange 2010, MS Exchange Server

An increasing issue appears to be a certificate prompt being seen by Outlook 2007 and higher clients following the introduction of additional CAS servers, or new multiple role servers holding the CAS role. 

While this has been an issue for some time and well known to those running a multiple server environment, the increasing number of postings on forums about this problem has probably occurred as single Exchange 2007 servers start to get to end of life and people migrate to Exchange 2010. 

The cause of this is usually autodiscover. 

What is Happening

CAS Servers have a value called "AutoDiscoverServiceInternalUri". This is published in to the domain as a Service Connection Point (SCP) and is queried by Outlook 2007 and higher as part of the internal autodiscover process. It tells the client where to connect to for the account information. 

If you have multiple CAS servers then they will all be publishing this information to the domain, in effect overwriting each other. 

This command will show you the name and the value set on all Client Access Servers in the org:

Get-ClientAccessServer |select name,AutoDiscoverServiceInternalUri

The Resolution

There are two resolutions to this issue, depending on your setup, and future plans. 

1. The simple fix is to bring forward the introduction of the trusted SSL certificate and get it installed on to the new server. The value for "AutoDiscoverServiceInternalUri" should match one of the host names on the SSL certificate. Remember that most SSL providers will not allow multiple certificates with the same names on them to be issued, so you may have to get a new certificate issued to cover all servers with the CAS role. 
   
2. Set the value for AutoDiscoverServiceInternalUri to be the same on all CAS Servers. If this is a specific server name, rather than a generic name, then you will need to change that value on all servers if you remove that server from production. Alternatively you could ensure that autodiscover.example.com resolves internally on your network to the IP address of a CAS server, then set all CAS servers to use that value. Then when the servers are changed, all you need to do is update the DNS. If you have clients on your internal network which are not members of the domain, then you may well have already configured this. 

Multiple AD Sites

If you have your CAS servers in multiple AD sites, then you may well have to consider using site scope to control which server the clients will connect to. There are other things to consider if this is the best thing to do or not and this Technet article explains how to use Site Scope: http://technet.microsoft.com/en-us/library/aa997633(EXCHG.80).aspx

CAS Array

This is not related to the Exchange 2010 CAS Array function, and you shouldn't use the CAS array host name for this. The CAS array doesn't use HTTPS and also shouldn't be resolvable from outside. 

Exchange 2007, Exchange 2010, MS Exchange Server, Outlook

This is part two of a three part case study of a recent network rebuild I carried out. For part one - click here: http://blog.sembee.co.uk/post/Case-Study-2-Part-1-Network-Rebuild-Intro-and-Workstations.aspx 

Servers

Now to the interesting bit. 

The server design was in my head for months, and then got completely redesigned following the client wanting to go with my suggestion of replicating the data off site. 

What we had was two HP ML350s, an old IBM and a HP desktop as the BES server. 

What we ended up with is three DL380s, two on site, one in the datacentre. 

All three DL380s are running VMWARE vSphere 4.1. 

VM1 - Two Windows VMs - a DC and a SQL Database server and a Linux based firewall. 

VM2 - Three VMs - a DC, Exchange 2010 and an application server. 

VM3 (in the data centre) - a DC, Exchange 2010 and a SQL database, plus a Linux based firewall.

As we are going to replicate Exchange data using a Database Availability Group, we needed to use Windows 2008 Enterprise edition. As Enterprise edition allows multiple installations of Windows on one physical machine, I decided to split up the functions in to dedicated servers. 

Furthermore, with more and more software products using SQL, and the client using SQL for an internal task, a dedicated SQL server was used. 

All three servers lived on the same network for a week, before the third server went off to the data centre. 

Data Replication

For real time data replication of the file structure, the network uses the latest version of DFS, built in to Windows 2008 R2. This works very well. 

For replication of Exchange data, a DAG is used for mailbox data, and native Public Folder replication. 

For SQL, this is mainly in the form of a backup, which is replicated to the data centre server shortly afterwards. Nothing the client does requires live replication of the SQL data. 

Exchange

Being an Exchange MVP, the design of the Exchange part of the platform was quite important, and everything has worked as I expected. 

The server that lives in the data centre is the only one that is exposed to the internet. All email comes in and leaves through that server. This provides a number of key benefits. 

• In the event of a loss of the main office, all email is coming in to a server that is under our control. We don't have to worry about email bouncing or being lost. 
• The dependency on the ISP at the main office is also removed, which I discuss further in part 3 networking. 
• Spam filtering is being done on the faster bandwidth available in the data centre.
• I have also pointed OWA and Outlook Anywhere traffic at the data centre server, not only for speed reason but if we have to use a backup internet connection, the clients don't have to be touched. This means that all inter-server traffic goes over the WAN connection. 

An RPC Client Access array is configured for outlook.example.local which points at the local CAS server, but allowing for easy changes in the event of a full failure. 

We also updated the Blackberry Enterprise Server from a very old version 4.0 to a 5.02 Express server. This is installed on the application server, with its database on the SQL server. 

Other Bits

WSUS - there are two WSUS servers in place, with the workstations pointing at a server in their office, and the laptops pointing to a child WSUS on the Exchange server in the data centre. This means that the laptops can pull their updates straight from Microsoft, whereas the desktops pull theirs from the local WSUS server. This saves bandwidth. 

As we had to use Windows Server Enterprise edition, which allows the use of four virtual machines, the server in the data centre had a spare. Therefore I have built a web server. Installed SmarterStats on to the server, which can only be accessed from the internal network. This means the client was able to change their public web site hosting arrangement and save money there. 

SmarterStats also allows use of OWA to be tracked. 

For backups, we dumped tapes, and Backup Exec. Switched to two Iomega Network Attached drives, with the backup job controlled by Backup Assist. The drives are exchanged each day, but are being used for archive purposes only. For full scale recovery, the copy in the data centre would be used. Shadow Copies is also enabled to provide additional levels of security.

The VMWare platform is managed by a vCenter server installed on the application server, with monitoring provided by Veeam's monitoring application. 

Remote access to the site is available via Log Me In, Remote Desktop Gateway and VPN. There is also the option of accessing the network resources with their Blackberries. This came in very handy when I couldn't remember a password in the data centre and needed to look it up on the password database (SecretServer from Thycotic) which has a mobile interface. 

Server Conclusion

In effect, the client now has their own mixed cloud and on site implementation, just they aren't sharing anything with anyone else. Data is stored off site, in real time. Traffic from the internet comes in through a static location which is secure, and fast. The client almost has a complete business continuity plan for a lot less than they would ever dream of. 

Part Three - Network is here: http://blog.sembee.co.uk/post/Case-Study-2-Part-3-Network-Rebuild-Networking.aspx

Amset IT Solutions Ltd. / Sembee Ltd., Case Study, Exchange 2010, MS Exchange Server, Networking General Amset IT Solutions Ltd. / Sembee Ltd., Case Study, MS Exchange Server, Networking General, Exchange 2010

Very occasionally, you get to do a job which you really enjoy. Being able to put lots of things that you have learnt over time in a single client deployment and make a very satisfying job. 

At the end of 2010 I completed just such a deployment.  

I could go on for hours about this deployment, as there are so many little things that were done, which I haven't had the chance to do before, or just make it a much better network. As I have complete control over the network, and have done for some time, I can ensure it runs exactly as it should. 

Only 40 users, so enough to use networking kit with. 

First, some background. This particular client is my oldest client. I have had them since about week six of my company. 

Just over 5 years ago I rebuilt their network, replacing their servers with a new domain, and all workstations were rebuilt. This was the first time I could try the locked down workstation method, as they had no proprietary or awkward third party application that "required" admin rights to run correctly. All desktops, and the one laptop didn't leave the building. 

Windows 2003, Exchange 2003 at the back end, on three servers, two HP and a very old clunky IBM which died last year. 

Clients were Windows XP, Office 2003. 

However it was starting to show its age. Three hours to setup a new workstation was becoming a joke, and the cost of server maintenance was getting higher all the time. 

Therefore it was decided that it was time to change the lot, all in one hit. 

Yes, you read that correctly. On the Monday they had the above, by the end of the week it was all changed. 

The first question then is how we could get away with doing a big bang change like this. 

It wasn't the original plan. I was looking at maybe changing the servers this year, then the workstations next. Office 2010 had just been released when planning started. However there was a keenness to do more, introduce laptops for some mobile workers so it was decided to make the change all at once. 

Furthermore, because the workstations were locked down, and were a basic build (Windows XP, Office 2003, AV, and a terminal application), with all relevant data redirected to a server, the amount of work that the move required would be minimal. The key company application is a database system that runs on Unix (which fortunately I have nothing to do with). The workstations are basically an office document and web browsing station. 

Then in a planning meeting I just happened to mention that we could replicate all of their data off site in real time for a lot less than they thought. So replacing the two servers became three, with replication thrown in as well. 

So this and the next two blog postings are a quick overview of what was done. If you would like to see it in action, and want me to do the same for your company, please let me know (UK Only). 

I am going to divide the rest of this blog in to three - workstations (below) and servers and networking which will have separate posts.

Workstations

This is quite easy. 

During the last 12 months of the previous XP/2003 based network, all replacement workstations were bought with the upgrade in mind. Minimum of 2gb of RAM and Windows 7 licences where possible. 

However a number had to be replaced, plus for the first time an active laptop fleet was introduced. 

This initial preparation work though made the initial deployment much easier. 

Desktops were Windows 7 Pro, Office 2010, Adobe Acrobat Reader, AV. The flash player was installed fresh, plus the terminal application. Installing off a memory stick, I was turning each machine around in about 45 minutes. 

Laptops were Dell Latitude, software as above. However we also added built in 3g cards so the users could work anywhere. Part of the plan (which I am not involved in) is to provide a web based access to their core database and inventory system. 

I also suggested, and was taken up, that every user, from the CEO down, was given a mandatory training session. So each staff member did a half day on Windows 7 and Office 2010. We found a local trainer, who created a bespoke course for the client. I explained what I wanted them to know. 

It should be pointed at this point that a large number of staff in this client are rather mature - I think I am still one of the youngest in the building when I go to visit. A change from Windows XP to Windows 7 would be quite different. The training was not only to show them how to do things, but also to simply give them confidence that they wouldn't break it. 

Therefore they were trained how to change the wallpaper, jump lists, gadgets. A brief overview on internet security and the like. They were trained on their actual workstations, so after the training was complete, there was a frantic period of machine change rounds. This meant that when they returned to their desks, things that they had done during training were still there. I felt this was important for adoption of the new platform. 

The new laptop users were given a slightly different course, which gave them a grounding in looking after the laptop. For most of them, this was the first time with a laptop. 

The client operates a conveyor belt system with desktops. New desktops go to the power users, with the slower ones going down the food chain, before eventually being removed. Therefore we started training with the power users on new desktops, while their older machines were rebuilt for the next session, and so on. This meant that during the training sessions I was rebuilding machines the users had just left. It got rather frantic. 

I rebuilt 9 machines in one day at one point, and put in 11 hour days four days on the trot. 

The end result though is that the client now has a complete desktop and laptop fleet that is on the latest OS and Office version, locked down, with the benefits that brings from a management and security point of view. 

In Part Two, I shall go over the server configuration. http://blog.sembee.co.uk/post/Case-Study-2-Part-2-Network-Rebuild-Servers.aspx 

Amset IT Solutions Ltd. / Sembee Ltd., Case Study, Exchange 2010, MS Exchange Server, Networking General Amset IT Solutions Ltd. / Sembee Ltd., Case Study, Exchange 2010, MS Exchange Server

Back in June 2009, I blogged on an authenticated user attack on a client's server.
http://blog.sembee.co.uk/post/Usernames-Tried-During-Authenticated-User-Attack.aspx

As part of that blog post, I included the list of names that were attempted.

The same server was attacked again in the last few days, and the list of usernames attempted changed very slightly. I have included the list below.
So quaint that they were tried in alphabetical order as well.

This list, along with the list from the original attack should be a list of usernames and passwords that you should avoid using, simply to ensure that you don't expose more than is necessary to this kind of attack.

www
vm
visitor
user
testuser
test
sysadmin
sysadm
support
supervisor
sales
operator
office
marketing
mail
info
guest
fax
backup
anonymous
admin
adm
account

MS Exchange Server, Networking General MS Exchange Server, Networking General

With Exchange 2007 and older versions, one of the key elements that an Exchange administrator needed to keep an eye on, and caused confusion for newcomers to Exchange was the amount of white space in the database.
This is reported as free space in the event viewer via event ID 1221 during the night and is the result of content being removed from the database by the online defrag process.

I have written about this event ID and the white space elsewhere:
http://www.amset.info/exchange/event1221.asp

With Exchange 2010, the behaviour of the database has changed.
Instead of doing a online defrag during a fixed time window, it now does it constantly. This means that content that has passed the deleted item retention period, is removed from the database shortly afterwards, rather than waiting for the next online defrag window.

However because the process is running constantly, event ID 1221 isn't written to the event log. Therefore an administrator may not have a clue as to how much of the database is white space, and how much is actual content.

This question can be easily answered, using EMS, as the amount of free space in the database is available via get-mailboxdatabase -Status:

Get-MailboxDatabase -Status | Select Servername, Name, AvailableNewMailboxSpace

This command will show you the name of the Server the database is mounted on, the name of the database (which is unique across the Exchange org with Exchange 2010) and the mount of space available in the database for new content.
The result will be something along the lines of this:

ServerName                     Name                          AvailableNewMailboxSpace
----------                          ----                            ------------------------
SMB-A                             Mailbox Database         27.75 MB (29,097,984 bytes)

The command used -get-mailboxdatabase -status can provide quite a bit of information about the databases in your Exchange org, use the |fl command to see the full list.

Exchange 2010, MS Exchange Server Exchange 2010, MS Exchange Server

One of the new features with Exchange 2010 is the client access array. When configured correctly, this is quite a useful feature. In my view it is something that should be configured on all Exchange 2010 servers, even on a single server deployment.

Background

The full explanation of the CAS Array feature is available on Technet, but in short, the reason it was introduced was due to the changes in the way connections to the database are now handled. With Exchange 2007 and older, Outlook connected directly to the mailbox server (unless using Outlook Anywhere). With Exchange 2010 all clients now connect to the CAS servers. The CAS servers then manage the connection to the database.
With the Database Availability Group (DAG) meaning that an active mailbox could be moved between servers easily, connecting directly to the mailbox server wasn't really practical.

The simple way to think of a CAS array is like a virtual Exchange server. Clients see this virtual name instead of the actual name of either the CAS server or the mailbox server.

Why you should configure a CAS Array

If you are deploying multiple CAS servers, or a DAG, then a CAS array is pretty much mandatory. However if you are on a single server, or are separating the mailbox and the CAS role on to separate machines, then a CAS array is still of value.
If you have ever done a migration or disaster recovery, one of the key pain points has been getting Outlook to point to the new server in a timely manner. As long as the original server was alive, then Outlook will redirect to the correct server automatically. During a migration though, it may not be possible to get all clients to connect to the old server in a timely manner and the old server has to be removed.

However as the CAS array is simply a DNS entry and a small configuration in Exchange, it is completely under the control of the network administrator. A change to the DNS will make all Outlook clients point to another server.

If there is a possibility at any time in the future of additional Exchange servers being introduced, or the CAS role moved to its own server, the use of the CAS array from the start will become invaluable for easing that transition. All MAPI clients will use it, so as well as Outlook, this can also include things like Blackberry Enterprise Server.

CAS Array Configuration Notes

Ideally the CAS array should be configured before any mailboxes are moved to Exchange 2010. If you don't, then the clients that are moved will use the true name of the CAS server, and even after the CAS array has been configured, they will not change unless the mailbox is moved between servers or the Outlook profile is changed.
If CAS Array is therefore introduced retrospectively, it can produce mixed results if all clients haven't been updated with the new value some how.

You can use the CAS array with Network Load Balancing (NLB), but if the server  has all of the roles and is also a DAG member, then you must use an external load balancer. Using NLB on the same server as the DAG is not supported.

A CAS array cannot go across Active Directory sites. Therefore if you are doing a two host DAG, with the second (passive) host in a data centre or similar, and have separated the AD sites, you will need two CAS arrays. In the event of a full failover, you will need to change both the CAS Array value on the database and the DNS. While this is a manual intervention, it does mean the process remains under your control.

The CAS array host does not have to be in the SSL certificate, simply because Outlook doesn't make any http connections to that host name.
You should not use the same host name for other services, particularly anything that is being accessed externally (like OWA), but you can use the same IP address and therefore NLB virtual IP.
For example, you could use outlook.example.local as the CAS Array host, then mail.example.com for OWA, SMTP, Outlook Anywhere etc.
If your internal and external domain are the same, then ensure the internal name doesn’t resolve, externally so no wildcard in the domain etc. Failure to do so will result in a confused Outlook, and will probably mean Outlook Anywhere has performance issues, if it connects at all.

Finally, on the DNS entry for the CAS array, turn the TTL time down. This will ensure that if you do have to change the host name IP address, it is picked up quickly.

Background and Configuration of the CAS Array: http://technet.microsoft.com/en-us/library/ee332317.aspx

Exchange 2010, MS Exchange Server Exchange 2010, MS Exchange Server

The default behaviour of Outlook with regards to sent items continues to come up on forums as an issue.

By default, when you send an email using the From field via your Send As permissions, the item you have sent goes in to your own Sent Items folder. This is because you sent it, not the person whose mailbox it was sent from. This can be useful from a tracking point of view (who sent the email).

However it may also be useful for the item to be stored in the Sent Items folder of the Shared Mailbox so that other users or even the mailbox owner can see what was sent.

How you achieve this depends on the version of Outlook that you are running. The version of Exchange doesn't matter.

For Outlook 2003 and 2007, a registry change is required, following the installation of an update. If you are keeping the machines up to date, then further updates should not be required.
These registry changes are outlined in the following articles:

Outlook 2007
http://support.microsoft.com/kb/972148
Requires Outlook 2007 Hotfix: 970944
http://support.microsoft.com/kb/970944/

Outlook 2003
http://support.microsoft.com/kb/953804/
Requires Outlook 2003 Hotfix: 953803
http://support.microsoft.com/kb/953803/

For older versions of Outlook, you will need to look at third party tools. The only one that I am aware of are the tools from Ivasoft: http://www.ivasoft.biz/

For OWA, you will need to use a server side tool, again the third party tools from the above site are the only ones that I am aware of - and support for latest version of Exchange isn't available.

For Outlook 2010, no registry change is required, you just need to add the mailbox in a different way.
Instead of adding the mailbox as an additional mailbox through the Properties of the primary mailbox, add the additional mailbox as an additional Account. That means going through the new Account wizard again. This feature also allows you to have connections to another mailbox in another Exchange forest at the same time - I have used this to migrate public folders (see http://blog.sembee.co.uk/post/Cross-Forest-Public-Folder-Migration.aspx)

However if you are using Outlook 2010, you should also be aware of the issues  in this KB article: http://support.microsoft.com/kb/2297543 (Performance problems when you try to access folders in a secondary mailbox in Outlook 2010).

(Late posting because I forgot to press publish).

Exchange 2003, Exchange 2007, Exchange 2010, MS Exchange Server, Outlook Exchange 2003, Exchange 2007, Exchange 2010, MS Exchange Server, Outlook