Process Tracking in the Microsoft Network Monitor 3.2 beta

Network protocol analyzers like WireShark (formerly Ethereal) and Microsoft Network Monitor are very powerful tools for troubleshooting and analysis work. Anyone who uses them quickly gets accustomed to the convenience and can’t imagine working without them. I’ve done my fair share of analysis and debugging with these sniffers and therefore follow their development with special interest.

I was therefore intrigued the other day when I noticed the announcement of Microsoft’s release of the Network Monitor 3.2 beta. Two particular items in the feature list caught my attention. One is the addition of a network capture and frame parsing API (NmAPI) to Network Monitor 3.x. Windows versions prior to Windows Vista featured the Network Monitor 2.x capture API. However, this API was removed from Windows Vista and the new Network Monitor 3.x targeting Vista failed to provide an alternative until now.

The other, more exciting, item on the feature list was Process Tracking. The announcement post claimed the new beta could group frames under their sending or receiving process in the Conversations view, showing process name and PID. I contend that anyone with some diagnostic experience would appreciate the immense importance and power of a process-oriented view of network frames.

I immediately retrieved the beta release from Microsoft Connect and began evaluating this new feature. I started a network capture and used Internet Explorer and Ping to send frames to the network. Soon enough, I realized that while Internet Explorer was successfully identified, the new Network Monitor failed to recognize Ping’s ICMP frames and instead opted to group them in the “<Unknown>” process group.

After reviewing the release notes, I learned that the tracking feature as documented only groups TCP and UDP sessions by process and frames containing other protocols are not supported.

At this point I began considering how would the Network Monitor developers go about implementing the Process Tracking feature and why would frame process association be restricted exclusively to TCP and UDP.

Under Windows XP, Network Monitor 3 uses the legacy Network Monitor 2 driver, nmnt.sys, included with the OS, to retrieve network frames from NDIS. Under Windows Vista, a new driver, nm3.sys, is used instead. I shall discuss the new Vista driver from this point forward.

nm3.sys is a NDIS 6 filter driver. The NDIS 6.0 architecture is new to Windows Vista, which explains why Microsoft opted to develop a new, separate filter driver. nm3.sys signs up for examining network frames by registering as a NDIS filter with the NdisFRegisterFilterDriver API. The caller specifies a lengthy set of filter callback functions in the NDIS_FILTER_DRIVER_CHARACTERISTICS structure provided in the call.

Among the callback functions registered by a NDIS 6 filter, of special interest are the FilterSendNetBufferLists and FilterReceiveNetBufferLists callbacks, invoked when a NET_BUFFER_LIST is to be sent or received. These callbacks receive a a list of network buffers containing the frames in question, but no process information is provided directly. To investigate whether process information can be retrieved indirectly, we need to track nm3’s callback implementations. Unfortunately, Microsoft failed to release public symbols (PDBs) to the Microsoft symbol store for the 3.2 beta, so the nm3.sys driver included with 3.1, for which symbols are available, shall be examined instead:

0:000> uf nm3!DriverEntry
00019006 8bff mov edi,edi
00019008 55 push ebp
00019009 8bec mov ebp,esp
0001900b 56 push esi
0001900c ff750c push dword ptr [ebp+0Ch]
0001900f ff7508 push dword ptr [ebp+8]
00019012 e8a19dffff call nm3!NmInitializeGlobals (00012db8 )

00019017 ff7508 push dword ptr [ebp+8]
0001901a e8dd84ffff call nm3!NmRegisterFilter (000114fc)
0001901f 8bf0 mov esi,eax
00019021 85f6 test esi,esi
00019023 7517 jne nm3!DriverEntry+0x36 (0001903c)
00019025 e8808effff call nm3!NmRegisterDevice (00011eaa)
0001902a 8bf0 mov esi,eax
0001902c 85f6 test esi,esi
0001902e 7411 je nm3!DriverEntry+0x3b (00019041)
00019030 ff3550700100 push dword ptr [nm3!g_FilterDriverHandle (00017050)]
00019036 ff1514600100 call dword ptr [nm3!_imp__NdisFDeregisterFilterDriver (00016014)]
0001903c e8b998ffff call nm3!NmFreeDriverResources (000128fa)
00019041 8bc6 mov eax,esi
00019043 5e pop esi
00019044 5d pop ebp
00019045 c20800 ret 8

During initialization, nm3 registers as a NDIS filter. Let’s see the specifics:

0:000> uf nm3!NmRegisterFilter
000114fc 8bff mov edi,edi
000114fe 55 push ebp
000114ff 8bec mov ebp,esp
00011501 81ec80000000 sub esp,80h
00011507 53 push ebx
00011508 56 push esi
00011509 8b3540610100 mov esi,dword ptr [nm3!_imp__RtlInitUnicodeString (00016140)]
0001150f 685e5b0100 push offset nm3! ?? ::FNODOBFM::`string' (00015b5e)
00011514 8d45e8 lea eax,[ebp-18h]
00011517 50 push eax
00011518 ffd6 call esi
0001151a 681c5b0100 push offset nm3! ?? ::FNODOBFM::`string' (00015b1c)
0001151f 8d45f8 lea eax,[ebp-8]
00011522 50 push eax
00011523 ffd6 call esi
00011525 68ce5a0100 push offset nm3! ?? ::FNODOBFM::`string' (00015ace)
0001152a 8d45f0 lea eax,[ebp-10h]
0001152d 50 push eax
0001152e ffd6 call esi
00011530 6a68 push 68h
00011532 33db xor ebx,ebx
00011534 8d4580 lea eax,[ebp-80h]
00011537 53 push ebx
00011538 50 push eax
00011539 e8e6410000 call nm3!memset (00015724)
0001153e 8b45f8 mov eax,dword ptr [ebp-8]
00011541 89458c mov dword ptr [ebp-74h],eax
00011544 8b45fc mov eax,dword ptr [ebp-4]
00011547 894590 mov dword ptr [ebp-70h],eax
0001154a 8b45f0 mov eax,dword ptr [ebp-10h]
0001154d 894594 mov dword ptr [ebp-6Ch],eax
00011550 8b45f4 mov eax,dword ptr [ebp-0Ch]
00011553 894598 mov dword ptr [ebp-68h],eax
00011556 8b45e8 mov eax,dword ptr [ebp-18h]
00011559 83c40c add esp,0Ch
0001155c 89459c mov dword ptr [ebp-64h],eax
0001155f 8b45ec mov eax,dword ptr [ebp-14h]
00011562 6850700100 push offset nm3!g_FilterDriverHandle (00017050)
00011567 8945a0 mov dword ptr [ebp-60h],eax
0001156a 8b4508 mov eax,dword ptr [ebp+8]
0001156d 8d4d80 lea ecx,[ebp-80h]
00011570 51 push ecx
00011571 c740343e290100 mov dword ptr [eax+34h],offset nm3!NetmonUnload (0001293e)
00011578 ff35a0700100 push dword ptr [nm3!g_FilterDriverObject (000170a0)]
0001157e c645808b mov byte ptr [ebp-80h],8Bh
00011582 50 push eax
00011583 66c745826800 mov word ptr [ebp-7Eh],68h
00011589 c6458101 mov byte ptr [ebp-7Fh],1
0001158d c6458406 mov byte ptr [ebp-7Ch],6
00011591 885d85 mov byte ptr [ebp-7Bh],bl
00011594 c6458601 mov byte ptr [ebp-7Ah],1
00011598 885d87 mov byte ptr [ebp-79h],bl
0001159b 895d88 mov dword ptr [ebp-78h],ebx
0001159e c745ac203d0100 mov dword ptr [ebp-54h],offset nm3!NetmonFilterAttach (00013d20)
000115a5 c745b094390100 mov dword ptr [ebp-50h],offset nm3!NetmonFilterDetach (00013994)
000115ac c745b474100100 mov dword ptr [ebp-4Ch],offset nm3!NetmonFilterRestart (00011074)
000115b3 c745b834100100 mov dword ptr [ebp-48h],offset nm3!NetmonFilterPause (00011034)
000115ba c745d0ec130100 mov dword ptr [ebp-30h],offset nm3!NetmonOidRequest (000113ec)
000115c1 c745a824110100 mov dword ptr [ebp-58h],offset nm3!NetmonFilterSetModuleOptions (00011124)
000115c8 c745a406100100 mov dword ptr [ebp-5Ch],offset nm3!NetmonSetOptions (00011006)
000115cf c745c8324c0100 mov dword ptr [ebp-38h],offset nm3!NetmonReceiveNetBufferLists (00014c32)
000115d6 c745dc92440100 mov dword ptr [ebp-24h],offset nm3!NetmonDevicePnPEventNotify (00014492)
000115dd c745e0b0440100 mov dword ptr [ebp-20h],offset nm3!NetmonNetPnPEvent (000144b0)
000115e4 895dcc mov dword ptr [ebp-34h],ebx
000115e7 c745e406110100 mov dword ptr [ebp-1Ch],offset nm3!NetmonFilterStatus (00011106)
000115ee c745d42e130100 mov dword ptr [ebp-2Ch],offset nm3!NetmonOidRequestComplete (0001132e)
000115f5 895dd8 mov dword ptr [ebp-28h],ebx
000115f8 c745bcfe4d0100 mov dword ptr [ebp-44h],offset nm3!NetmonSendNetBufferLists (00014dfe)
000115ff 895dc0 mov dword ptr [ebp-40h],ebx
00011602 895dc4 mov dword ptr [ebp-3Ch],ebx
00011605 ff152c600100 call dword ptr [nm3!_imp__NdisFRegisterFilterDriver (0001602c)]
0001160b 5e pop esi
0001160c 5b pop ebx
0001160d c9 leave
0001160e c20400 ret 4

The callback implementations are NetmonSendNetBufferLists and NetmonReceiveNetBufferLists. We can place breakpoints on these callbacks while a network capture is in progress and examine the stack. Let’s look at what things look like for an outgoing ECHO request sent by the PING command:

1: kd> kb 2000
ChildEBP RetAddr Args to Child
9db555a0 85cbc585 851b4be8 856cb458 00000000 nm3!NetmonSendNetBufferLists
9db555c0 85cbc5a8 856cb458 856cb458 00000000 ndis!ndisFilterSendNetBufferLists+0x8b
9db555d8 8c60545f 851ba808 856cb458 00000000 ndis!NdisFSendNetBufferLists+0x18
9db55654 85cbc638 851b2d60 856cb458 00000000 pacer!PcFilterSendNetBufferLists+0x233
9db55670 85d8764a 856cb458 856cb458 00000000 ndis!ndisSendNBLToFilter+0x87
9db55694 85e8a1ee 851b4750 856cb458 00000000 ndis!NdisSendNetBufferLists+0x4f
9db556dc 85e89dcc 84b412b8 00000000 00000000 tcpip!FlSendPackets+0x399
9db5571c 85e899db 85eeec68 00000000 00000000 tcpip!IppFragmentPackets+0x201
9db55754 85e8b7cb 85eeec68 9db55870 616c7049 tcpip!IppDispatchSendPacketHelper+0x252
9db557f4 85e8ac3f 00b55870 85eeec68 00000000 tcpip!IppPacketizeDatagrams+0x8fd
9db55954 85e8c75d 00000000 856cb400 85eeec68 tcpip!IppSendDatagramsCommon+0x5f9
9db55974 85e57d83 85eeec68 9db559c0 83682128 tcpip!IppSendDatagrams+0x2a
9db5599c 85e58a3a 00000000 00000000 856cb458 tcpip!IppSendControl+0xfe
9db55b20 85e58234 00000000 000003a5 85ee96a4 tcpip!Ipv4SetEchoRequestCreate+0x718
9db55b64 85df8a29 9db55b7c 00000000 85683038 tcpip!Ipv4SetAllEchoRequestParameters+0xf2
9db55ba4 8c681551 00000006 8370218c 00000000 NETIO!NsiSetAllParametersEx+0xbd
9db55bf0 8c681eb8 00000000 8568eaa0 8568ead8 nsiproxy!NsippSetAllParameters+0x1b1
9db55c14 8c681f91 83702101 00000000 856838f8 nsiproxy!NsippDispatchDeviceControl+0x88
9db55c2c 8184b1ad 85142448 83702170 83702170 nsiproxy!NsippDispatch+0x33
9db55c44 819f7f64 856838f8 83702170 837021e0 nt!IofCallDriver+0x63
9db55c64 81a02940 85142448 856838f8 0016f400 nt!IopSynchronousServiceTail+0x1d9
9db55d00 81a346cf 85142448 83702170 00000000 nt!IopXxxControlFile+0x6b7
9db55d34 8185c9aa 000000f8 00000138 00000000 nt!NtDeviceIoControlFile+0x2a
9db55d34 77159a94 000000f8 00000138 00000000 nt!KiFastCallEntry+0x12a
0016f3d4 77158444 773514b9 000000f8 00000138 ntdll!KiFastSystemCallRet
0016f3d8 773514b9 000000f8 00000138 00000000 ntdll!ZwDeviceIoControlFile+0xc
0016f41c 77351b48 00120013 0016f450 00000028 NSI!NsiIoctl+0x5d
0016f440 77351b1b 0016f450 0024dc1c 00000000 NSI!NsiSetAllParametersEx+0x23
0016f478 753591f2 00000001 00000006 753533e4 NSI!NsiSetAllParameters+0x53
0016f528 00fe24df 0023fb60 00000000 00000000 IPHLPAPI!IcmpSendEcho2Ex+0x1d5
0016fa7c 00fe2a23 00000002 008422d0 00841578 PING!main+0xacb
0016fac0 75c54911 7ffdf000 0016fb0c 7713e4b6 PING!_initterm_e+0x163
0016facc 7713e4b6 7ffdf000 770df6a4 00000000 kernel32!BaseThreadInitThunk+0xe
0016fb0c 7713e489 00fe2b5d 7ffdf000 00000000 ntdll!__RtlUserThreadStart+0x23
0016fb24 00000000 00fe2b5d 7ffdf000 00000000 ntdll!_RtlUserThreadStart+0x1b

On the top of the kernel-mode stack, we see the filter callback for outgoing network frames. The frames were dispatched to the filter by NDIS (NdisFSendNetBufferLists). User space sent the network frame by issuing a IOCTL to the network protocol stack. Notice the cut-off between the user-mode and kernel-mode stack at KiFastSystemCallRet. PING uses the IP Helper API to send the outgoing ECHO frame.

We can conclude from this stack trace that at least for some cases, the user-space thread context active when a NDIS filter callback for outgoing frames is invoked is in fact associated with the process that sent the respective network frame. Theoretically, a filter could use IoGetCurrentProcess, extract information (name, PID, etc.) and provide it to the user-space network capture program.

However, there remains the possibility that due to network frame buffering or other reasons, the originating process will not be the one active when the filter callback is invoked, but rather we’d be in arbitrary thread context. Let’s accept that and consider the situation on the receive path. Let’s consider a stack trace for an invocation of the receive callback:

0: kd> kb 2000
ChildEBP RetAddr Args to Child
8069dea8 85d79aba 851b4be8 854c67b0 00000000 nm3!NetmonReceiveNetBufferLists
8069dec4 85cba54a 85687438 854c67b0 00000000 ndis!ndisMIndicateReceiveNetBufferListsInternal+0x27
8069dee0 8636f71f 85687438 854c67b0 00000000 ndis!NdisMIndicateReceiveNetBufferLists+0x20
8069df28 8636e77e 00000000 8636e6fe 00000001 E1G60I32!RxProcessReceiveInterrupts+0xdd
8069df40 85d7911c 01f0d160 00000000 00000000 E1G60I32!E1000HandleInterrupt+0x80
8069df64 85cba468 84bfd5fc 00000000 00000000 ndis!ndisMiniportDpc+0x81
8069df88 8186fab0 84bfd5fc 85687438 00000000 ndis!ndisInterruptDpc+0xc4
8069dff4 8186dfa5 9db55470 00000000 00000000 nt!KiRetireDpcList+0x147
8069dff8 9db55470 00000000 00000000 00000000 nt!KiDispatchInterrupt+0x45
WARNING: Frame IP not in any known module. Following frames may be wrong.
8186dfa5 00000000 0000001b 00c7850f bb830000 0x9db55470

The stack trace for the receive path illustrates that the receive callback is invoked directly during interrupt processing by the network interface card. The NIC’s driver DPC notifies NDIS that new network frames are available by calling NdisMIndicateReceiveNetBufferLists, which invokes the filter callback.

This makes sense. When network frames are received, they are first processed by the driver, then by the filter and only then would they be dispatched to the user-space process listening on a matching socket, etc.

With the receive path being unsuitable for deducing process association and the send path’s reliability being impaired by buffering and other behavior that can lead to arbitrary thread context, Network Monitor opts for a totally different approach for process tracking, not involving the NDIS filter driver at all.

Examining the modified filter callbacks in the 3.2 beta version of the nm3 driver proved that it did not attempt to collect any process information. Process Tracking must be implemented elsewhere. A simple examination of the Network Monitor program, netmon.exe, revealed that its import address table (IAT) references the well-known APIs GetExtendedTcpTable and GetExtendedUdpTable.

These APIs were introduced with the Windows XP Service Pack 2 release. You can see them in action by running the beloved “netstat” command with the new “-b” switch, that shows which process is associated with an open socket. Using “-b -v” shows all the modules involved in a socket connection. The “-o” switch provides more concise information in the form of the relevant PID.

What the new Network Monitor does, is, in effect, the moral equivalent of running “netstat -b” whenever a TCP or UDP frame is captured. The local endpoint (IP and port) is matched against the table returned by the GetExtendedXxpTable APIs.

There are several shortcomings to this technique. One is that it only works with TCP and UDP. Another is that information may not be available for very short-lived connections, since by the time the extended tables are queried, the session may already be long gone. One process could potentially send out UDP datagrams using a raw socket without establishing a port binding and be confused with another process that used a regular socket and bound itself to the same source port.

In conclusion, the addition of Process Tracking to Network Monitor, while welcome, is not the holy grail of process network monitoring by any means. Since the NDIS filter architecture does not lend itself to process-oriented monitoring, a solution from that end is probably not available.

As implied by this discussion thread, the way to go here, may be a Windows Filtering Platform Callout Driver on Vista or a TDI upper filter on downlevel systems. WFP provides process information internally and an upper TDI filter can apparently rely on the user-space thread context be non-arbitrary. The question remains whether such solutions would be overkill or not for a network protocol analyzer.


4 thoughts on “Process Tracking in the Microsoft Network Monitor 3.2 beta

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s