strange .net runtime error and the fix

We had done a POC for Foglight by Quest back in March and ended up not purchasing the product because we already had SCOM in place. After the uninstall and ever since, we have had some issues.

  • errors logged whenever an app pool starts/restarts containing this description
  • .NET Runtime version 2.0.50727.3603 – Failed to CoCreate profiler.
  • .net health monitoring would log an app pool start but not log an app pool stop or shutdown
  • I had the suspicion that this was related to random app pool restarts during the day on one of our webservices
  • other debuggers would not attach to an app pool and work (like avicode that comes with SCOM)
  • I looked for how to fix this since March, of course I was too stubborn to call Quest, so that’s my fault. Much googling didn’t resolve much so when I finally did figure this out, I wanted to post what I found. It turns out that debugging is set in an environment variable. Seems like it’s a session variable because it’s set in the parameters of a service, so that way it runs with the service in the user context of the service. This is something I had never run across before so it seemed kinda odd.

    In the end we had to remove the reg key “Environment” (and the contents of the key) from the two locations. This key is what sets the debugger to enabled and tells it which debugger to use.

    • [HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesW3SVC]
    • [HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesIISADMIN]

    Once these have been deleted, all you have to do is IISRESET the server and the error is gone and random app pool restarts have ceased. As of this writing I have not tried to reinstall the SCOM AVIcode to the webservice, but I’m certain that it will work now.

    scom missing performance counters

    I recently had to make some changes to my scom test environment server. It’s a single virtual machine with scom, sql, ssrs, the works on it. Despite what one would think, it has performed well with about 30 agents. Recently I added about that many more and it pushed up the ram to the 4gb max that I had given it. When I added more I realized I had used 2008 x32 for the OS (oops) and was not able to see the extra ram. After some research I found that I could do an upgrade from 2008 standard to enterprise, which is what I did. Afterwards, scom wouldn’t run because the SDK service would not start. This is the event log message.

    Log Name:      Operations Manager
    Source:        OpsMgr SDK Service
    Date:          1/7/2011 2:25:48 PM
    Event ID:      26380
    Task Category: None
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:     

    Description:
    The System Center Operations Manager SDK Service failed due to an unhandled exception. 
    The service will attempt to restart.
    Exception:

    System.TypeInitializationException: The type initializer for ‘Microsoft.EnterpriseManagement.Mom.ServiceDataLayer.SdkDataAccess’ threw an exception. —> System.InvalidOperationException: The requested Performance Counter is not a custom counter, it has to be initialized as ReadOnly.
       at System.Diagnostics.PerformanceCounter.Initialize()
       at System.Diagnostics.PerformanceCounter.set_RawValue(Int64 value)
       at Microsoft.EnterpriseManagement.Mom.Internal.PerformanceCounterCategoryDescriptor.SingleInstancePerformanceCounterCategoryManager.CreateCounter(String categoryName, String counterName)
       at Microsoft.EnterpriseManagement.Mom.Internal.PerformanceCounterCategoryDescriptor.SingleInstancePerformanceCounterCategoryManager.CreateCounters(PerformanceCounterCategoryDescriptor categoryInfo)
       at Microsoft.EnterpriseManagement.Mom.Internal.PerformanceCounterCategoryDescriptor.SingleInstancePerformanceCounterCategoryManager..ctor(String instanceName, PerformanceCounterCategoryDescriptor categoryInfo)
       at Microsoft.EnterpriseManagement.Mom.Internal.SingleInstancePerformanceCounterCategoryDescriptor.CreateManager()
       at Microsoft.EnterpriseManagement.Mom.ServiceDataLayer.SdkPerfCounterManager..ctor()
       at Microsoft.EnterpriseManagement.Mom.ServiceDataLayer.SdkDataAccess..cctor()
       — End of inner exception stack trace —
       at Microsoft.EnterpriseManagement.Mom.ServiceDataLayer.SdkDataAccess.Initialize()
       at Microsoft.EnterpriseManagement.Mom.Sdk.Service.SdkSubService.AsyncStartCallback(Object notUsed)
       at System.Threading._ThreadPoolWaitCallback.WaitCallback_Context(Object state)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
       at System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(_ThreadPoolWaitCallback tpWaitCallBack)
       at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(Object state)
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="OpsMgr SDK Service" />
        <EventID Qualifiers="49152">26380</EventID>
        <Level>2</Level>
        <Task>0</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2011-01-07T19:25:48.000Z" />
        <EventRecordID>10025424</EventRecordID>
        <Channel>Operations Manager</Channel>
        <Computer>INFMGT52.accessgeneral.com</Computer>
        <Security />
      </System>
      <EventData>
        <Data>System.TypeInitializationException: The type initializer for ‘Microsoft.EnterpriseManagement.Mom.ServiceDataLayer.SdkDataAccess’ threw an exception. —&gt; System.InvalidOperationException: The requested Performance Counter is not a custom counter, it has to be initialized as ReadOnly.
       at System.Diagnostics.PerformanceCounter.Initialize()
       at System.Diagnostics.PerformanceCounter.set_RawValue(Int64 value)
       at Microsoft.EnterpriseManagement.Mom.Internal.PerformanceCounterCategoryDescriptor.SingleInstancePerformanceCounterCategoryManager.CreateCounter(String categoryName, String counterName)
       at Microsoft.EnterpriseManagement.Mom.Internal.PerformanceCounterCategoryDescriptor.SingleInstancePerformanceCounterCategoryManager.CreateCounters(PerformanceCounterCategoryDescriptor categoryInfo)
       at Microsoft.EnterpriseManagement.Mom.Internal.PerformanceCounterCategoryDescriptor.SingleInstancePerformanceCounterCategoryManager..ctor(String instanceName, PerformanceCounterCategoryDescriptor categoryInfo)
       at Microsoft.EnterpriseManagement.Mom.Internal.SingleInstancePerformanceCounterCategoryDescriptor.CreateManager()
       at Microsoft.EnterpriseManagement.Mom.ServiceDataLayer.SdkPerfCounterManager..ctor()
       at Microsoft.EnterpriseManagement.Mom.ServiceDataLayer.SdkDataAccess..cctor()
       — End of inner exception stack trace —
       at Microsoft.EnterpriseManagement.Mom.ServiceDataLayer.SdkDataAccess.Initialize()
       at Microsoft.EnterpriseManagement.Mom.Sdk.Service.SdkSubService.AsyncStartCallback(Object notUsed)
       at System.Threading._ThreadPoolWaitCallback.WaitCallback_Context(Object state)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
       at System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(_ThreadPoolWaitCallback tpWaitCallBack)
       at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(Object state)</Data>
      </EventData>
    </Event>

    After some research I found this article from Microsoft that contains instructions on how to fix this problem.

    Dashboarding using the SCCM Dashboard solution accelerator example

    I will turn this into a more useful blog post when I have a few minutes but for now it is just a collection of links about dashboarding using the System Center Configuration Manager Solution Accelerator. The short story is that this runs on top of Sharepoint 3.0, which is free, meaning you can run this for free. It’s not specific to SCCM so you can dashboard any sql data you want! Cool!

    http://technet.microsoft.com/en-us/library/ff369719.aspx

    http://systemscentre.blogspot.com/search/label/Dashboard

    http://garyhay.blogspot.com/2010/07/sccm-dashboard-queries-1.html

    http://blogs.msdn.com/b/shitanshu/archive/2010/04/08/part-1-how-microsoft-it-using-configuration-manager-dashboard.aspx

    http://blogs.msdn.com/b/shitanshu/archive/2010/04/11/part-2-using-configuration-manager-dashboard-for-software-update-deployment-readiness-patch-tuesday-checklist.aspx

    http://blogs.msdn.com/b/shitanshu/archive/2010/05/02/part-3-sql-queries-used-for-creating-custom-configuration-manager-dashboard-in-microsoft-it.aspx

    http://blogs.msdn.com/b/shitanshu/archive/2010/10/09/part-4-sql-queries-for-creating-configuration-manager-client-health-and-problem-management-dashboard.aspx

    http://64.4.11.252/en-us/edge/system-center-configuration-manager-2007-dashboard-part-2-configuration.aspx?query=1

    cleaning up the default management pack

    SCOM has what I feel is a major bug in that it will allow you to save items (monitors, rules, overrides, etc.) in the default MP. Doing this is bad for a lot of reasons, and not only does SCOM allow you to do this, but it is the default option as well. In my case it turned out that an occasional lack of attention allowed me to do this and then removing MP’s later becomes a huge pain in the rear. Anyway I found this good article on how to clean up the mess.

    SCOM: powershell run space failed to start

    I have been getting these messages since day 1 and tried various things that didn’t work to resolve.

    Below I am pasting an example rule with full text so that if someone is searching they will find it. This is one specific alert, I was having an issue with all non microsoft powershell scripted discoveries. For me this was 99% from the XSNMP SCOM Management Pack. To be clear the MP was not the cause of the problem, only the one that tried to run PS (and not work) the most.

    PM

    Log Name:
    Operations Manager

    Source:
    Health Service Modules

    Event Number:
    22400

    Level:
    1

    Logging Computer:

    User:
    N/A

    Description:

    Failed to run the PowerShell script due to exception below, this workflow will be unloaded. System.NullReferenceException: Object reference not set to an instance of an object. at System.Environment.GetEnvironmentVariable(String variable, EnvironmentVariableTarget target) at System.Management.Automation.ModuleIntrinsics.GetExpandedEnvironmentVariable(String name, EnvironmentVariableTarget target) at System.Management.Automation.ModuleIntrinsics.SetModulePath() at System.Management.Automation.ExecutionContext.InitializeCommon(AutomationEngine engine, PSHost hostInterface) at System.Management.Automation.AutomationEngine..ctor(PSHost hostInterface, RunspaceConfiguration runspaceConfiguration, InitialSessionState iss) at System.Management.Automation.Runspaces.LocalRunspace.DoOpenHelper() at System.Management.Automation.Runspaces.RunspaceBase.CoreOpen(Boolean syncCall) at Microsoft.EnterpriseManagement.Modules.PowerShell.RunspaceController.RunScript(String scriptName, String scriptBody, Dictionary`2 parameters, PowerShellOutputType outputType, Int32 serializationDepth, IModuleDebug iModuleDebug) at Microsoft.EnterpriseManagement.Modules.PowerShell.PowerShellProbeActionModule.RunScript(RunspaceController runspaceController) Script Name: MemoryPctUtil.ps1 One or more workflows were affected by this. Workflow name: xSNMP.Cisco.Rule.CollectMemoryPoolUtil Instance name: I/O Instance ID: {X} Management group: X

    Event Data:

    < DataItem type =" System.XmlData " time =" 2010-12-03T19:15:30.1742570-05:00 " sourceHealthServiceId =" X" >

    < EventData >

    < Data > X </ Data >

    < Data > xSNMP.Cisco.Rule.CollectMemoryPoolUtil </ Data >

    < Data > I/O </ Data >

    < Data > {X} </ Data >

    < Data > MemoryPctUtil.ps1 </ Data >

    < Data > 300 </ Data >

    < Data > System.NullReferenceException: Object reference not set to an instance of an object. at System.Environment.GetEnvironmentVariable(String variable, EnvironmentVariableTarget target) at System.Management.Automation.ModuleIntrinsics.GetExpandedEnvironmentVariable(String name, EnvironmentVariableTarget target) at System.Management.Automation.ModuleIntrinsics.SetModulePath() at System.Management.Automation.ExecutionContext.InitializeCommon(AutomationEngine engine, PSHost hostInterface) at System.Management.Automation.AutomationEngine..ctor(PSHost hostInterface, RunspaceConfiguration runspaceConfiguration, InitialSessionState iss) at System.Management.Automation.Runspaces.LocalRunspace.DoOpenHelper() at System.Management.Automation.Runspaces.RunspaceBase.CoreOpen(Boolean syncCall) at Microsoft.EnterpriseManagement.Modules.PowerShell.RunspaceController.RunScript(String scriptName, String scriptBody, Dictionary`2 parameters, PowerShellOutputType outputType, Int32 serializationDepth, IModuleDebug iModuleDebug) at Microsoft.EnterpriseManagement.Modules.PowerShell.PowerShellProbeActionModule.RunScript(RunspaceController runspaceController) </ Data >

    < Data />

    </ EventData >

    </ DataItem >

     

    Eventually the scripts would time out like this:

    Time window start
    12/7/2010 11:57:38 AM

    Time window end
    12/7/2010 12:02:37 PM

    Time First
    12/7/2010 11:57:38 AM

    Time Last
    12/7/2010 11:57:41 AM

    Count
    44

    Context

    Date and Time:
    12/7/2010 11:57:41 AM

    Log Name:
    Operations Manager

    Source:
    Health Service Modules

    Event Number:
    22411

    Level:
    1

    Logging Computer:
    X

    User:
    N/A

    Description:

    The PowerShell script will be dropped because the it has been waiting in the queue for more than 10 minutes. Script Name: DiscoverInterfaceName.ps1 One or more workflows were affected by this. Workflow name: xSNMP.Discovery.InterfaceName Instance name: GigabitEthernet2/21 Instance ID: {X} Management group: X

    Event Data:

    < DataItem type =" System.XmlData " time =" 2010-12-07T11:57:41.4054873-05:00 " sourceHealthServiceId =" X" >

    < EventData >

    < Data > X </ Data >

    < Data > xSNMP.Discovery.InterfaceName </ Data >

    < Data > GigabitEthernet2/21 </ Data >

    < Data > {X} </ Data >

    < Data > DiscoverInterfaceName.ps1 </ Data >

    < Data > 300 </ Data >

    < Data > 10 </ Data >

    < Data />

    </ EventData >

    </ DataItem >

     

    The issue turned out to be  a permissions issue with the health service since it was trying to run these powershell scripts as local system. This article is the one that finally jogged my memory appropriately. Thank you to the author!

    2008 R2 DC’s and SCOM

    We migrated our domain to 2008 AD from 2003 AD last week and found a couple of issues via SCOM.

    • DNSSec Zone TrustAnchors  -  for this one you’ll get an alert that looks like this: Zone TrustAnchors on DNS Server dns.name is not responding to queries.
    • DNS 2008 Server External Address Resolution Alert – for this you’ll just get a failure of the alert. To be fair the same alert for 2003 has the same problem, but we had fixed that a long time ago, thus the info had been forgotten.

    That’s it!

    scom web application monitoring part 2 – presenting the data – service levels and the dashboard

    This is the 2nd post in a short series on monitoring web applications with SCOM. Part 1 is here.

    One of the biggest issues I have with SCOM is the sheer amount of data… it is so easy to grab a parameter here, a value here, and you throw that in with all of the stuff the management packs will give you already and suddenly you have a lot to choose from and picking and presenting that data becomes the difficult thing. Do yourself a favor and don’t show management the SCOM console, it looks more complicated than it is and I don’t think it presents that well except to technical folks.

    Creating dashboards is limited, there needs to be some more work here from Microsoft. For example, like I mentioned in my previous post, you cannot save what a performance view is supposed to look like, meaning which (or all) counters are checked. I understand why Microsoft did this for the default performance view per user, but IMO once you create a dashboard view, that becomes impractical and there should be a way to make the selections a part of the view.

    The dashboard also has the problem of not looking too great via the web console. It’s limited and looks kinda fugly. As a result we have tried using the actual SCOM client that we installed as a citrix app so that we can display it on the flat screen via the wyse terminal. This has the problem of not being able to default a view without a lot of work, and we keep running into issues where you need the detail pane here but not here, and you need to be able to select your views on the left hand side sometimes, but you don’t want the “action” pane visible, and you end up with something that looks like a hack.

    Microsoft seems to have realized this and has since created a “solution accelerator” called the service level dashboard. I’m not going to go into what it takes to install this because there are already a ton of sites out there already that have the info. It isn’t the easiest thing to get installed because it requires a sharepoint installation which it customizes and bastardizes quite a bit, and it also needs access to the operations manager database, data warehouse, pretty much everything involving SCOM. In my case it was easier to put the actual sharepoint install on my SCOM server, which I did, and ended up having to figure out why sharepoint stepped all over my SCOM website. This wasn’t rocket science but it took some effort. If I was doing it over again, I would go ahead and install sharepoint before I installed SCOM, or find a home somewhere else that isn’t on the SCOM RMS.

    Once you go through the motions of getting sharepoint and the service level dashboard installed, we can get to work.

    I ran out of time today so it looks like this will be a 3 part post.

    scom web application monitoring – making it useful – part 1

    I could go on for days about SCOM and the URL monitoring and how it needs to be improved. Honestly.. it kinda sucks. So here I will attempt to describe what I think is wrong with it and how I work around it. The items in bold below are what I feel like are failures in the way this was designed.

    Also I am not writing this as strictly a “how to monitor a web app” post, there are already plenty of those. This is just about the changes required to make this useful. Here is a good article with the basics on setting up a web application monitor in SCOM.

    • Requirements

    To begin with, you will need to figure out what you need to monitor. In many cases it is simple enough to pull up the main page of a website and as long as it comes up, is in a reasonable timeframe, and is giving an HTTP status code of 200, you’re OK. This sort of monitoring is useful, but you can do so much more in order to get a lot more out of it. What I like to do is get the devs to code you up something special through some sort of bribery or blackmail. In our case what they did was define 5 business processes, for example “make a payment” and create a page that does the back end work of making that transaction but also the other end of the work which is cleaning up after itself. What you will get in the end isn’t exactly user experience, but it’s a good way to track the ongoing performance of a process relative to itself, and it’s a very good up/down indicator. Since we have dev environments as well, I have those on a development scom server, and I have the below web monitoring in place there as well in the first production like environment. This allows our QA folks to compare state and response time and see if the environment is working before they release code or start a test, but also they can see the impact of the new code by comparing response times from before and after the code release.

    • Once you have your URL’s, it’s time to get to work.

    Create a web application monitor and give it your URL. The problem with those default settings is that by default you are only logging the transaction response time and not alerting on it. From an alert standpoint, there is no timeout for your web request, matter of fact, the only thing SCOM will tell you out of the box is just if it was eventually able to pull up a URL as long as it doesn’t have an HTTP response code > 400. This default setting is not useful!

    To fix this, what you want to do is add response time criteria like this.

    image

    Because of a problem with the service level dashboard that I will explain later, I only put one HTTP request in each web application monitor. This brings me to a little UI weirdness here because you can also set response times in the “configure settings” for the specific URL pull like this.

     image

    I always leave this performance criteria blank because I can see the other one easier and get more out of it. This one here just seems redundant.

    • Seeing the data

    Now once you gather some data you will want to, well, see what’s going on. In order to do this, create a new performance view in the monitoring console and scope it to “collected by specific rules”, and then you get to go manually pick your rules. This is where Microsoft fails again, because the list of rules is not searchable and they all have arbitrary names. For web requests I figured out they are called “Performance Collection: Transaction response time total for Name of web app monitor”. like this screenshot.

    image

    Now that you have done that, you will be able to see a nice blank performance chart with some stuff to check.

    image

    Now when we pick one, we get a pretty graph like this.

    image

    This brings me to my next issue with all of this.. it’s that the performance chart settings are user specific.. meaning I cannot create a view of any sort that contains performance information and have the counters checked already. No matter which ones I put in, and it doesn’t matter if you are using a performance view or even a dashboard view that contains a performance view, those have to be selected every time. This is a pain!

    This also means that if you wanted to say, get fancy with a URL to a specific view, you cannot just create one of these and have folks click the link and end up at a pretty performance chart with the counters already checked. The fact that you cannot do this is a serious limitation with SCOM, IMO.

    • setting up alert parameters (what you cannot change)

    You will likely have to play with the values a bit in order to get them not to false alert. And this brings me to my next problem with SCOM web monitoring, it’s that you cannot change anything about how it samples other than where it is from (what host) and how often it samples. What I would love to do is be able to say “only alert when two consecutive thresholds are exceeded”, but that’s not an option. We get a lot of failures at night during our backup window that cause a single transaction to go out of SLA, and we get alerts based on that. As a result, we have to set our thresholds for response time to the highest level it could possibly be so that we aren’t false alerted every night, but this makes it so high that the alerting becomes less useful during the daytime. As of now I do not have a workaround for this.

    • stopping duplicate alerts

    When you do get your first alert you will see that two are sent.. one for the URL pull and one for the aggregate monitor on the web application monitor. This doesn’t really make sense to me why this would be set up this way at all, so let’s fix it.

    Start by right clicking on one of the alerts and open the health explorer for it. Expand it out and you will see something like this.

    image

    Each of the red lines has an alert set up for it, and the lower one for the actual request rolls up into the web application one. In my mind the web application one is redundant, so I am going to disable it. Right click, choose “monitor properties”, go to alerting, and uncheck it.

    image

    Now you will receive one alert instead of two.

    • useful alert details

    Of course the text of the alerts isn’t useful at all out of the box (it doesn’t tell you if the URL failed for time, SSL, http response, or anything). I am using this article as a basis for fixing this, but I don’t have it totally worked out yet. This will continue to require some further tweaking.

    This post ended up being longer than I intended (there’s a lot to fix) so I am going to break it up into two parts and get the service level dashboard stuff into a 2nd post.