Infrastructure issues (part 2)

Back in March we had performance issues with our firewalls. One of the things that our vendor raised was what they saw as an unusually high number of DNS queries to external servers. We were seeing around 2-3000 requests/second from our caching DNS servers to the outside world.

A bit more monitoring (encouraged by other sites reporting significantly lower rates than us) identified a couple of sources of unusual load:

1. The solution we use for filtering incoming mail sends DNS queries to all servers listed in resolv.conf in parallel. That doesn’t give any benefit in our environment so we changed things so that it only uses the caching DNS server on localhost.

2. We were seeing high rates of reverse lookups for IP addresses in ranges belonging to Google  (and others) which are getting SERVFAIL responses. These are uncacheable so always result in queries to external servers. To test this theory I installed dummy empty reverse zones on the caching name servers and the queries immediately dried up. The fake empty zones meant that the local servers would return a cacheable NXDOMAIN rather than SERVFAIL.

An example of a query that results in SERVFAIL is www.google.my. [should be www.google.com.my]). That was being requested half a dozen times a second through one of our DNS servers. www.google.my just caught my eye – there are probably many others generating a similar rate.

Asking colleagues at other institutions via the ucisa-ig list and on ServerFault reinforced the hypothesis that (a) the main DNS servers were doing the right thing and (b) this was a local config problem (because no-one else was seeing this).

Turned on request logging on the BIND DNS servers and used the usual grep/awk/sort pipeline to summarise – that showed that most requests were coming from the Windows domain controllers.

Armed with this information we looked at the config on the Windows servers again and the cause was obvious. It was a very long-standing misconfiguration of the DNS server on the domain controllers – they were set to forward not only to a pair of caching servers running Bind (as I thought) but also all the other domain controllers which would in turn forward the request to the same set of servers. I’m surprised that this hadn’t been worse/shown up before since as long as the domain returns SERVFAIL the requests just keep circulating round.

The graph below shows the rate of requests that gave a SERVFAIL response – note the sharp decrease in March when we made the change to the DNS config on the AD servers. [in a fit of tidiness I deleted the original image file and now don’t have the stats to recreate it – the replacement doesn’t cover the same period]

dns4

I can see why this might have seemed like a sensible configuration at the time – it looks (at one level) similar to the idea of a set of squid proxies asking their peers it they already have a resource cached). Queries that didn’t result in SERVFAIL were fine (so the obvious tests wouldn’t show any problems).

Postscript: I realised this morning that we’d almost certainly seen symptoms of this problem early last July – graph below shows the very sharp increase in requests followed by the sharp decrease when we installed some fake empty zones. This high level of requests was provoked by an unknown client on campus looking up random hosts in three domains which were all returning SERVFAIL. Sadly we didn’t identify the DC misconfiguration at the time.

dns4-july2014

Shift & Right Click!

Thought I would blog on something that I only learnt last year but has been a great time saver for me.

Holding down ‘Shift’ when right clicking in explorer gives some extra very handy options including ‘Open a Command Window here’ and ‘Copy as Path.’

Also, several files types have other contact sensitive options for instance Office files can ‘Open as read only’.

CAP

Free e-book: Introducing Windows Server 2008 R2

All you need is a Windows Live ID.

Free e-book offer from Microsoft Press: Introducing Windows Server 2008 R2
Learn about the features of Windows Server 2008 R2 in the areas of virtualization, management, the web application platform, scalability and reliability, and interoperability with Windows 7. Sign in to download Introducing Windows Server 2008 R2, written by industry experts Charlie Russel and Craig Zacker along with the Windows Server team at Microsoft.

http://www.microsoft.com/…dowsserver.aspx

Troubleshooting Terminal Services (RDS) Client issues

This problem keeps coming up every now and then so I thought it would be good to document it. If you get licencing error messages when you connect to a Terminal Services session on a remote server then this might fix the problem (you might also get this error if you use the Remote Application Service (RAS) here at Newcastle).

Open regedit

Create a backup of the MSLicensing registry key and its subkeys on the client, and then remove the original key and subkeys by doing the following:

1. On the client, navigate to the following registry subkey: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSLicensing.
2. Click MSLicensing.
3. On the Registry menu, click Export Registry File.
4. In the File name box, type mslicensingbackup, and then click Save.
5. If you need to restore this registry key in the future, double-click mslicensingbackup.reg.
6. On the Edit menu, click Delete, and then click Yes to confirm the deletion of the MSLicensing registry subkey.
7. Close Registry Editor, and then restart the computer.
When the client is restarted, the missing registry key is rebuilt.

Its probably also worth checking to make sure you have the RDP 7 client installed, see http://blogs.msdn.com/rds/archive/2009/10/28/announcing-the-availability-of-remote-desktop-connection-7-0-for-windows-xp-sp3-windows-vista-sp1-and-windows-vista-sp2.aspx (we rolled this out on campus at the end of 2009)

Connect From Anywhere using the Terminal Services Gateway

Posted by popular demand on behalf of Adele…

The TS Gateway service allows you to connect to your work PC from home or other off-campus locations, even when your work PC is on an internal University network (i.e. 10.x.x.x IP address). Used in conjunction with Wake On Lan. This gives you 24 hour access to your on-campus PC.

To use the service you must ensure that you have the latest Remote Desktop Client installed on the PC from which you are connecting back into work. If you are running Windows Vista or Windows 7, you should already have what you need. If you are running Windows XP or earlier, you may need to visit Microsoft.com and download a later RDP client.

Instructions

Prerequisite: the work PC must be set-up to allow Remote Desktop Connections and you will need to ensure that the ID that you are using is in the Remote Desktop Users group on the PC.

Launch Remote Desktop Client (you’ll find it by browsing to Accessories or just click Start…on Vista or Windows 7 (or Start.. Run if on XP) and type in mstsc and press Enter)

Click on Options as shown below:

Remote Desktop Connection options

Click on Advanced and then Settings as shown below:

Remote Desktop Connection Settings

Complete the TS Gateway settings precisely as shown below:

tsgateway settings

Click OK, and go back to the General tab. Enter the name of your work PC plus .ncl.ac.uk:

Enter the name of your work PC plus ncl.ac.uk

 

Click Connect. Enter an id that has rights to log on remotely to the PC. For example:

Enter credentials

Click OK. (You can use a local ID, but you’ll need to qualify it by using machinename\ rather than campus\

Setting up a Vista or Windows 7 PC for remote access

Click Start…

Right-click Computer and then select Properties.

Click on Advanced system settings and, if prompted, supply the credentials of an account that has admin rights to the PC. Click on the Remote tab and Select Users:

Setting up RDC - remote tab

Add the accounts for any user that you want to be able to remotely access the PC:

Add users to RDC permissions

Then click OK… OK. All done.

You should test the settings from another on-campus machine before attempting to connect from off-campus.

The procedure is more or less the same for Windows XP but you will need to be logged on with admin rights before starting.

When using the above service, it is strongly recommended that you ensure your home PC is fully up-to-date with Windows Updates and is running good antivirus/antispyware software. Be sure to adhere to the University’s Computing Rules of Use at all times, and take care to protect sensitive and important data from unauthorised access as you would when working directly on-campus.

Windows 7 and Windows Server 2008 R2 Event – 8th July

For Windows sys admins, the biggest contacts on your approach radar right now are Windows 7 and Windows Server 2008 R2, due to release later in the year (but be complete and released to manufacture next month). This free evening event, right here on the campus, couldn’t be much more timely then.

Rik Hepworth, the IT Director at Black Marble, will cover some of the great new features of the new operating systems, including BranchCache, XP Mode and what I personally think is the number one feature, DirectAccess.

This is bound to be a popular event, so sign up early over at the VBUG site.