wsus and cloning vm’s

I am currently working on WSUS (windows software update services) here at work, for the most part I’m following this excellent article at Ars. After screwing around with this for much longer than I should have, I was having issues with about half the servers not showing up in the WSUS console. Many things could have been the culprit and I checked them all.. group policy, dns, firewall rules, etc. What was the most frustrating was that I could see the clients touch the WSUS server by looking in the IIS logs, and there were no errors whatsoever, but half the servers wouldn’t show up.

Eventually I realized that it was exactly half of the servers and a light bulb went off. In our environment we have a bunch of web and app servers that are all virtual, and when we build them, we get the first node working right, and then clone and rename the vm to be the redundant node in the farm.This lead me to do some searching and I found this link. Admittedly, this is an old problem, but the first time I have run across it.

The following is a repost of the pertinent bits that have caused my trouble and are the resolution for it.

5. Imaged clients with a duplicate client ID will only appear once in the WSUS Admin Console. Each AU client must have a unique id which is created for each individual install. When imaging systems it is recommended always to use SysPrep. The WSUS admin console will only display one client for each unique ID. If you have multiple clients created from one image which are sharing the same ID, only one will appear in the WSUS admin console. All clients will check in and download updates, but only one will appear and display status in the WSUS admin console. In cases where clients are not checking in, and they were created from images without running SysPrep, the following steps will reset the existing duplicative client IDs.

a. Run regedit and go to HKEY_LOCAL_MACHINESOFTWAREMicrosoftWindowsCurrentVersionWindowsUpdate

b. Delete the PingID, SUSClientID and the AccountDomainSID values

c. Stop and start the Wuauserv Service

d. From the command prompt run: wuauclt /resetauthorization /detectnow

or-

From the command line, once you are sure the AU client is properly configured and not disabled, you could run a batch file (which might look something like this sample) and get the same results:

rem Fixes problem with client machines not showing up on the server due to imaging method

reg delete HKLMSOFTWAREMicrosoftWindowsCurrentVersionWindowsUpdate /v AccountDomainSid /f

reg delete HKLMSOFTWAREMicrosoftWindowsCurrentVersionWindowsUpdate /v PingID /f

reg delete HKLMSOFTWAREMicrosoftWindowsCurrentVersionWindowsUpdate /v SusClientId /f

cls

@echo Triggering detection after resetting WSUS client identity

net stop wuauserv

net start wuauserv

wuauclt /resetauthorization /detectnow

IIS – which app pool is which?

With IIS creating w3wp.exe’s for all of your apps (for which you have created a non default app pool) it is nice, but it’s impossible to tell exactly which is which. Microsoft includes a handy tool to do this with it’s in: %systemroot%system32iisapp.vbs (You may need to run it with cscript.exe if that’s not your default for .vbs).

Sample output:

C:WINDOWSsystem32>iisapp
W3WP.exe PID: 2028   AppPoolId: DefaultAppPool
W3WP.exe PID: 20224   AppPoolId: Autotransactiondocs
W3WP.exe PID: 17480   AppPoolId: IRAppPool

C:WINDOWSsystem32>

Oh, this is for Windows Server 2003 (and r2) only.. does not work with 2008 variants.

.net coding advice = do not enable debug in production!

This seems like a simple thing, really, but every single place I have worked that developed .net code has had this problem, and for some reason, devs don’t seem to get it. (It has been an uphill battle everywhere I have found it with resistance from devs. Why??)

Usually the symptoms start out with scaling issues in production, you’ll see things like threads running out, connections not being closed, and general poor application performance. You can track these down and eventually you will find that in the web.config the setting “debug=enabled” is in there. It is this way because it is the default when you create a .net application. YOU DO NOT LEAVE IT LIKE THIS IN PRODUCTION. When you do, bad things happen, such as timeouts all being set to infinite. Here’s an article with a complete list and some more info.

Depending on whether or not your organization pushes the .configs with the applications (this is a religious discussion in itself.. I recommend not pushing the configs with every build for reasons like this), then when you do your next code push, the process repeats because of this value being set improperly again.  The way to fix this for good is that in the production environment, you make a change to the machine.config to enable “retail” mode, which disallows debug from being enabled, regardless of what is in the application’s web.config.

<configuration>

<system.web>

<deployment retail=”true”/>

</system.web>

</configuration>

In my opinion, setting this in the machine.config should be part of the default build/installation/configuration of any production webserver.

detecting database connection issues (not closing) in .net

This one is really easy, but I am surprised at the number of people who are unable to troubleshoot it. If you think you are having sql connections not being closed properly, just open up perfmon, go to “.net data provider for sqlserver”, and add “numberofreclaimedconnections”. Basically what this counter shows is the number of connections that were not closed by the application, but were closed by the .net garbage collection process. Microsoft’s explanation is on this page. The same counter works for the oracle client as well (except it’s called “oracle” and not “sql”).

The symptoms of this would be a whole lot of idle connections to sql (or oracle) from the webserver, or errors about not being able to obtain a connection, running out of connections in the pool, etc. You can also see lots of connections in the time_wait status if you run a netstat –an.

this is just dumb (SMTP manager on 2008 server)

I had to install the SMTP service on a 2008 server today for the first time, and Microsoft does not make it easy.

  1. The SMTP install is under features and not under roles.
  2. When you do install SMTP, where do you manage it, you wonder. Thanks to google and this link, I now know that you have to use the IIS6 manager. What??!!

ssl diagnostics for iis7

I have spent the better part of the day trying to figure out why a client cert that’s send to our servers just plain out doesn’t work. We get a 403.17 when we shouldn’t. It’s maddening because all we can get out of IIS is the 403.17, and since we do not own the client cert, there’s no easy way for us to test. So we have to trust that our b2b partner has things set up right (we don’t think they do) and then wait 30-35 minutes in between requests for them to re-test. It’s maddening.

This post contains some good troubleshooting info and steps.

Here is a link to some info and a utility for mapping client certs, something that Microsoft inexplicably left out of IIS7. (Why?!)

Another tool I found during this whole process was the IIS7 ssl checker utility. It didn’t (hasn’t) helped our specific situation, but it is something that looks useful that I will save for future problems that will definitely come up.