Advice - Performance Counters, WMI and VB.NET

Freeware PerfCollector tool and code that I wrote allows both Performance Counter and WMI query style querying is available in another post.

Recently I was working on a project that needed to find which business methods were taking the longest. As required in all enterprise handbooks the project had developed a whole bunch of custom performance counters on the business code. It all works perfectly. Dumping out values for the number of times a business method has been called, dumping out the time the last call took, average times, recent average times etc.

Problem is that whenever I asked someone to help monitoring the business methods performance in a live environment they blindly waved at the performance counters and tell me to use them. How? Having performance counters in your code doesn’t mean anyone knows how to use them.

Fairly quickly I noticed that Windows PerfMon isn’t going to help. PerfMon logging allows you to choose the counter and then choose the instances (in our case instances are each of the business method names). Two major problems with this approach are a) PerfMon doesn’t dynamically add instances once you have setup logging, so in my case I’m stuffed as new business methods that didn’t exist when I first setup the counter log would never be logged and b) sampling every 1 second collects a huge amount of data and I don’t really want to parse files that have the same performance counter details repeated even though the values are changing – nor the room to store all this data.

Ideally what I’m looking for is IIS style log files for long running business calls. 1 line per long running business method identifying; method name, time and the length of call. I really wanted to catch long running business code and know which method to attack.

To solve this I knew I’d need to write code (couldn’t find much on the Internet). I looked at using VB.NET to access the System.Diagnostics.PerformanceCounter objects. I wrote code to log out the value of counters that changed (that met a threshold I set) and to re-enumerate all the instances to look for new counters. This gave me a trace on the business methods. Logging out performance counters in Visual Baisc dotNet looks like :-

Dim myCounter As System.Diagnostics.PerformanceCounter = New System.Diagnostics.PerformanceCounter
myCounter.CategoryName = “Process”
myCounter.InstanceName = “lsass”
myCounter.CounterName = “% User Time”
Dim lValue As Long = CLng(myCounter.RawValue)

I then realised that as the number of instances (business methods in my case) increased then the load on the server will increase (from my tool enumerating), tool thus defeating its purpose of finding long running queries. Maybe there is a simple way to capture performance counters that are changing? I remembered that WMI exposed performance counters. With further research I conducted I found that WMI can provide callbacks to code when a query you register in WMI finds a record(s).

WMI must be one of the most complex pieces of Microsoft technology I’ve worked with. E.g. root\cimv2 – huh? Why cim? why v2? Then if you want to look at instances you need to run wbemtest. Not wmitest? I’d love to write a dummies guide to WMI, but I suspect it just wouldn’t fly as fairly quickly it would turn into complex reference material. Below is a brain dump of what I’ve found in my 3 days of research – starting point only and I’m sure I’ve made some silly assumptions, but if it can help any of you let me know.

What can I do with WMI - WMI is available on newer Microsoft operating systems (W2K, WXP, W2K3 etc) and it allows you to a) interrogate a system configuration e.g. Disk, Processor, Memory etc b) interrogate all performance counters c) files on a file system and d) has differencing facilities so you can know when something is added, modified or deleted (and I’m sure many more if you can delve its depths)

How can I look at WMI stuff, specifically Performance Counters – Simple, follow these 9 simple steps ;-). Goto Start->Run and then type wbemtest. Next select Connect… and type in the top textbox (un-labled) root\cimv2. Select Connect. Now select Enum Classes… type win32_perfrawdata then select Ok. Now select the specific object (aka Performance Object), we’ll select Win32_PerfRawData_PerfProc_Process. From here you have 2 choices.
a) If you want to look at counters then in the properties list (the 2nd listbox) scroll down to the actual counters.
b) Or to view a value of a counter select Instances to view instances and then double click on an instance to view what counters are available and then select the counter to view the value.

My custom performance counter doesn’t exist – I’m sure there is a more technical reason, but use Start->Run and then type wmiadap.exe /f. Reboot and hopefully all is good. Reboot is definitely required as before a reboot you may be able to see the performance counters in wbemtest, but you won’t get instances created and thus no values.

How can I read performance counters in code – Few ways, but for my purpose I want to know when a counter changes. Firstly you need a decent WMI query like the one below.

SELECT
*
FROM __InstanceModificationEvent
WITHIN 1
WHERE
TargetInstance ISA 'Win32_PerfRawData_CustomCounter_CustomCounter'
AND TargetInstance.Count > PreviousInstance.Count
AND NOT TargetInstance.Name = '_Total'
AND TargetInstance.LastTime > 1500

The __InstanceModificationEvent part of this query is a special in-memory table that stores changes from all WMI events. The Performance Counters luckily are part of this differencing. The WHERE clause allow you to narrow down exactly what change you want to know about. The WITHIN clause is how often sampling occurs (1 second in this case). So what happens if 2 or more ‘changes’ happen in the WITHIN time. Well you get only 1 – I suspect the last one. Either poll more frequently (e.g 0.5. The lower this number the more stress you place on your PC), or ask Microsoft to support ‘GROUP WITHIN’ and aggregate Events – this isn’t supported. You also need to use code like this in VB.NET

Dim wqlQuery As String = GenerateWMIQuery()
' Setup WMI Query and Watcher
Dim query As System.Management.WqlEventQuery = New System.Management.WqlEventQuery(wqlQuery)
Dim mWatcher As System.Management.ManagementEventWatcher = New System.Management.ManagementEventWatcher
mWatcher.Query = query

' Set up a listener for events
AddHandler mWatcher.EventArrived, AddressOf WMIEventRaised
' Start listening
mWatcher.Start()
Do
' Loop every 10 seconds.
System.Threading.Thread.Sleep(10000)
Loop

Then catch via

Public Shared Sub WMIEventRaised(ByVal sender As Object, ByVal e As System.Management.EventArrivedEventArgs)

Dim ev As System.Management.ManagementBaseObject = e.NewEvent
Dim lValue As Long = CType(ev("TargetInstance"), System.Management.ManagementBaseObject)("PercentUserTime")
Dim sInstance As String = CType(ev("TargetInstance"), System.Management.ManagementBaseObject)("Name")

End Sub

How often should I sample and what is the effect on performance – No idea. Seriously depends on your hardware. My suggestion is make the WMI query work for you. Ensure that you are only querying for exactly what you need, then you should be able to query every 5 seconds to pick up the event you need. Sure you might miss something, but if performance is so bad you’ll catch it again. I’m querying every 5 seconds and the WMI performance looks fine as does the .NET code.

PerfCollector tool and code that I wrote allows both Performance Counter and WMI query style querying.