Every year we try to produce some statistics of the types and number of mobile devices that are connecting to the Exchange servers for email and calendaring.

There have been 7064 unique mobile devices connected between 01/04/2012 and 15/08/2012.

Of the 7064 devices, 4697 belong to students and 2367 to staff.

It has been about a year since I last published some statistics on the different mobile devices that are connecting to the Exchange servers for email/calendaring. Those statistics can be found here.

There have been 5161 unique mobile devices that have accessed the Exchange service in the month from 09/05/11 to 08/06/11.

As you can see there has been a huge increase in the number of mobile devices accessing the Exchange service and can surely be seen as an indication towards the reliance on mobile technology as a way of accessing University services.

A surprising statistic is that 4299 of the 5161 have used the service within the last 24 hours!

Microsoft have long been threatening to remove Public Folders from Exchange and have been deprecating their use with every iteration of the product. Typically, here at Newcastle University, users have requested Public Folders to keep calendars of meeting rooms. Although the Public Folders are easy to set up and manage, they don’t really work too well. Checking availability and the logistics of organising the time with the Public Folder calendar alongside personal calendars is often a complicated and laborious affair.

Using Outlook 2007 and Exchange 2007, the creation of dedicated Resource Mailboxes became a much simpler process and more user friendly. To fall in line with Microsoft’s deprecation of Public Folders, we are keen to have people to move their Public Folder calendars into Resource Mailboxes. The set-up and maintenance of the Resource Mailboxes is best suited to a School Computing Officer or if not available a dedicated super-user.

The resource mailbox is very similar to a standard mailbox however does include some extra options to allow for automated responses and resource dedicated configuration.

Although the Resource Mailboxes still work with Outlook 2003, the checking of availability and the manner in which to find the resource mailboxes is slightly more complicated.

I’ve recently drawn up some documentation for Computing Officers with regards to the configuration of Resource Mailboxes and also for end-users for how you would use them in daily operation.

Resource Mailbox Configuration

Using Resource Mailboxes

We’ve been using Resource Mailboxes internally within ISS for a little while now and have also introduced the service for the Student Interaction element of King’s Gate and parts of the Robinson Library. It is to note that this system is not a competitor for Syllabus Plus and our in-house timetabling services, but to be used as a supplement so that users can organise small meetings within their school/service.

Last week, we were unfortunate to have a major problem with our Exchange 2007 service. The server process (store.exe) that runs the mailboxes was crashing every 5 minutes or so. As the process that was failing is fundamental to the service working, it was important to try and diagnose the issue as quickly as possible.

The most difficult part of this diagnosis was that the error was so generic and wasn’t providing any relevant information as to what to look at. Exchange is a complicated and awkward piece of software at the best of times, so the problem was compounded by unsatisfactory logging of what was happening.

We were able to determine that the problem happened on both nodes of a cluster, so it looked likely that it was related to the database or even at a more granular level.

When dismounting databases to try and narrow down the issue, we noticed that when one particular database was dismounted that the problem went away. Sadly this meant a significant amount of downtime for the mailboxes on this affected database, particularly as we needed to obtain diagnostic information for Microsoft to investigate the problem, as it’s a fault that isn’t documented anywhere.

We narrowed down the problem to when a particular message (that was queued on one of our Hub Transport Servers) was trying to be delivered that the mailbox server crashed. We deleted the item from the queue and everything started to work OK. It wasn’t long before the problem reoccurred. We could then correlate (using the wonderful tool that is Powershell) that the second message that was causing a problem was scheduled to be delivered to the same mailbox as the first.

This indicated that the problem was common to one mailbox. We isolated that mailbox away from our production server to provide some stability to the thousands of other users that have mailboxes residing there. Once that mailbox was moved, the other mailboxes were fine, so it seemed as if the problem was really narrowed down.

We could reproduce the crash by replaying the problem message into the test system, so we were now at the stage where we could try and determine what it was about these two messages that caused the problem.

The problem seemed to be caused by some fault in a user’s rules. We had fortunately found the needle in the haystack and at the same time we were able to hopefully provide enough diagnostic information to Microsoft so they can thoroughly investigate why a problem with one user’s rules was enough to crash the entire server. That really is a big failing of Exchange.

One issue we noted was the user who had the problem mailbox was exclusively using Outlook 2003. If the user had moved to Outlook 2007, the problem would have been somewhat alleviated. Outlook 2007 alters the rules format.

It was an incredibly stressful couple of days and underlined the fact that email is a business critical system. We are still waiting to hear back from Microsoft, but should the problem reoccur, we should be much quicker in being able to diagnose the fault.

Apologies for the dry nature of this blog post, but I thought it important to give some background on why Outlook and more importantly Exchange often suffer performance issues. One of the main causes of poor performance is a user having high item counts in ‘Critical Path’ Outlook Folders and how Exchange deals with that.

Microsoft’s recommendations for Exchange 2007 are as follows:

“Ideally, it is best to keep the Inbox and Sent Items folders less than 20,000 items, and the Contacts and Calendar item counts less than 5,000. Even when maintaining item counts that are at or under the recommended maximum values, there are some operations which may still take noticeable time (usually this is approximately one minute). These operations include new sort orders and selecting folders for the first time. First time views of a folder can take even more time to generate the view. High item counts in critical path folders have an adverse effect on server performance because they are the most frequently accessed folders in a user’s mailbox. Other folders, especially custom folders that are created by end-users, can handle having larger numbers of items without having an adverse effect on the user experience because they are accessed less frequently. Be aware that, although the performance effect of having higher item counts in less frequently accessed folders is reduced, high item counts in these folders can still present intermittent performance issues as the number of folders like this increases, and the number of active users on the server increases.”
Source: http://technet.microsoft.com/en-us/library/cc535025%28EXCHG.80%29.aspx

It is important to note that not only the user with high items counts will suffer a performance hit, but also all other users on the server.

Microsoft are keen to address this problem with Exchange 2010, they are changing their recommended maximum limits to 100,000 items. This is mainly due to big improvements in disk I/O performance and larger memory caches.

Other ways to address the performance issues are to use Cached Exchange Mode. This passes the performance required to the client computer, from the server. All of the views, searches, sorts etc. take place on the Cached Mode’s OST file. Of course, this is not the answer for everybody and should the mailbox have a massive amount of items and the OST file also be massive, then the user will still experience performance issues.

Housekeeping is the real answer and users should get into the habit of deleting old mail that they do not need to keep. Filing email from ‘critical path folders’ (Calendar, Contacts, Inbox and Sent Items) into user defined folders is also a big help.

Exchange Server and Outlook come with an inbuilt address book (Global Address Book – GAL) that includes a list of every mailbox on the University’s Exchange email system. There are still other email systems at work at the University and at present the users on those system do not appear in the GAL.

As this is the case, it is important that the CAMA system is used as the de facto standard for finding e-mail address or telephone numbers. You can not guarantee that the person you are trying to reach has their mailbox residing on the Exchange system.

CAMA Interface:

The Global Address List is still not without merit, but it should be used carefully. As we have so many staff and students appearing in the list, there is quite a high chance that users have the same name. The GAL should contain information detailing a user’s school / service / course to help determine a specific individual.

Please note that the results have been blurred for the purpose of the screenshot.

1. Start typing the name of the individual that you are trying to find. The GAL will always try to navigate that list based on the user Display Name. This typically follows the Firstname, Lastname format, however due to legacy mail systems and number of users, there are users whose Display Name and email address starts only with their initials.

2. Use the scroll bar to find out more details. Most mailboxes have School / Service / Course information included to help distinguish users with the same name.

3. If you are experiencing difficult finding a user, try changing the search capacity. Changing from Name Only to More Columns allows the search field to look across multiple fields in a user’s account. This will let you search for all users with the same surname or same department for example.

4. Finally using the Advance Find function allows to specifically tailor your search for specific fields.

Outlook offers the function to remember previously typed email address. It stores these addresses in a Nickname cache (.nk2 file). As useful as this function can be, unfortunately Outlook also remembers addresses that you enter incorrectly.

When next typing the address, the erroneous address is offered as a choice. Without checking, it is easy to select that address and repeat the problem. Fortunately it is a simple procedure to remove the problem addresses.

When creating a new message, start typing the first few letters of the previously entered address.

Please note that the addresses have been blurred for the purpose of the screenshot.

The previously entered addresses appear. You can then use the cursor keys to navigate that list. Once the incorrect address is highlighted, press the Delete key on the keyboard. That address will be removed from the list.

NB. The NK2 file that controls what appears in the autocomplete list is associated with an Outlook profile. Should you change Outlook profiles, then a different/new NK2 file will be used.