Remote Procedure Call debugging

Recently, I discussed how one would go about finding the other end of an LPC (Local inter-Process Communication, rather than Local Procedure Call, apparently) port. LPC is used directly through the native API for some Windows components such as LSA, but is more frequently used by third parties in the form of the “ncalrpc” RPC transport. When dealing with those cases, or cases where the higher level RPC runtime is used in general (e.g., with the named pipes or TCP transports), we must turn to a whole other family of techniques.

While in the case of LPC analysis we turned to the aid of the kernel debugger, in the case of RPC we can utilize built-in instrumentation found in the Windows RPC runtime library. Since RPC debugging may come to involve a variety of distributed scenarios, rather than opting for a plain registry setting enabling instrumentation, Microsoft chose to provide control through the group policy facility.

Enabling debugging aid by the runtime is prerequisite to any useful analysis work. Follow the instructions in the MSDN page “Enabling RPC State Information” and restart the system. Usually you’ll be able to make do with the “Server” setting.

For illustration purposes, we shall consider the HELLO RPC sample available with the Microsoft Windows SDK. The HELLO sample includes an IDL file specifying a trivial illustrative interface providing the HelloProc remote call that passes a string to the server side and the Shutdown remote call that instructs the server to shut down. Let’s run the HELLO server process.

In order to diagnose a product using RPC we must figure out the server endpoint of interest. Our primary tool will be the “dbgrpc” utility distributed with the Debugging Tools for Windows. With RPC state information enabled, we begin by enumerating RPC endpoints:

C:\Program Files\Debugging Tools for Windows>dbgrpc -e
Searching for endpoint info ...
PID CELL ID ST PROTSEQ ENDPOINT
-------------------------------------------------------------
0274 0000.0001 01 LRPC IUserProfile
0274 0000.0003 01 LRPC sclogonrpc
0274 0000.0005 01 NMP \PIPE\InitShutdown
0274 0000.0007 01 NMP \PIPE\SfcApi
0274 0000.000a 01 NMP \pipe\winlogonrpc
0274 0000.000e 01 LRPC OLEFEB89B1D900E460783A2A6ABA
02a0 0000.0001 01 LRPC ntsvcs
02a0 0000.0003 01 NMP \pipe\ntsvcs
02a0 0000.0006 01 NMP \PIPE\scerpc
02ac 0000.0001 01 NMP \PIPE\lsass
02ac 0000.0003 01 LRPC audit
02ac 0000.0005 01 LRPC securityevent
02ac 0000.0007 01 LRPC protected_storage
02ac 0000.0009 01 NMP \PIPE\protected_storage
034c 0000.0001 01 LRPC actkernel
034c 0000.0005 01 LRPC IcaApi
034c 0000.0007 01 NMP \pipe\Ctx_WinStation_API_ser
03a4 0000.0001 01 LRPC epmapper
03a4 0000.0003 01 TCP 135
03a4 0000.000a 01 NMP \pipe\epmapper
0414 0000.0001 01 LRPC dhcpcsvc
0414 0000.0003 01 LRPC wzcsvc
0414 0000.0005 01 LRPC OLEA390A47C8A6F4EA78EA712E62
0414 0000.0009 01 NMP \PIPE\atsvc
0414 0000.000e 01 LRPC AudioSrv
0414 0000.0010 01 NMP \PIPE\wkssvc
0414 0000.0011 01 NMP \pipe\keysvc
0414 0000.0012 01 LRPC keysvc
0414 0000.0014 01 LRPC SECLOGON
0414 0000.0016 01 NMP \pipe\trkwks
0414 0000.0017 01 LRPC trkwks
0414 0000.001a 01 NMP \PIPE\srvsvc
0414 0000.001d 01 LRPC srrpc
0414 0000.001f 01 LRPC senssvc
0414 0000.0021 01 NMP \PIPE\W32TIME
04ec 0000.0001 01 LRPC DNSResolver
0548 0000.0001 01 NMP \PIPE\DAV RPC SERVICE
0548 0000.0003 01 NMP \PIPE\winreg
0548 0000.0004 01 LRPC LRPC00000548.00000001
05e4 0000.0001 01 NMP \pipe\spoolss
05e4 0000.0003 01 LRPC spoolss
05e4 0000.0006 01 LRPC OLE8BC761BE0AFF4D9CA9603B53B
0684 0000.0001 01 LRPC OLE872E70B024824F8894A85E384
00ac 0000.0001 01 LRPC OLEAA4283CA4B51483E95665C439
0204 0000.0001 01 LRPC OLEDBAAFA32AEBF41AD808B50A1B
0594 0000.0001 01 LRPC OLEA0D6A971EC424B7DB839E9308
0314 0000.0001 01 LRPC hello

Endpoint enumeration gives you an idea of available RPC services in a server system. Since the HELLO server process was the last one launched, it is conveniently found at the bottom of the output.

Without repeating too much of the RPC debugging primer in the Windbg documentation, I’ll just point out the important fact that RPC state information is organized into “cells” in each process. Through the use of a simple endpoint enumeration command, we’ve already concluded that the HELLO server process is PID 0x314. Not an impressive feat for a process we just launched, but consider that this could easily be a third-party RPC server started as a service or on demand in an unknown executable.

Most of the time, we can associate the endpoint name with the application of interest since a descriptive string is being used. However, in other cases, we may know the server application of interest, but the endpoint name is unknown, random or auto-generated. When there’s just one endpoint, we can just find the process of interest in the dbgrpc endpoint enumeration output. In any case, we can examine the call used by the server application to the RPC runtime to determine which endpoint name is in use:


0:000> bp rpcrt4!RpcServerUseProtseqEpA
0:000> g
Breakpoint 0 hit
eax=00452000 ebx=7ffd5000 ecx=00452008 edx=00000014 esi=00d5f55c edi=7c911970
eip=77e97a0b esp=0012ff3c ebp=0012ff6c iopl=0 nv up ei pl nz na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000206
RPCRT4!RpcServerUseProtseqEpA:
77e97a0b 8bff mov edi,edi
0:000> kb
ChildEBP RetAddr Args to Child
0012ff38 00401046 00452000 00000014 00452008 RPCRT4!RpcServerUseProtseqEpA
0012ff6c 00401e37 00000001 003330a0 00333120 hellos!main+0x46 [e:\projects\hello\hellos.c @ 21]
0012ffb8 00401d0f 0012fff0 7c816ff7 7c911970 hellos!__tmainCRTStartup+0x117 [f:\dd\vctools\crt_bld\self_x86\crt\src\crt0.c @ 266]
0012ffc0 7c816ff7 7c911970 00d5f55c 7ffd5000 hellos!mainCRTStartup+0xf [f:\dd\vctools\crt_bld\self_x86\crt\src\crt0.c @ 182]
0012fff0 00000000 00401d00 00000000 78746341 kernel32!BaseProcessStart+0x23

We note that the third argument to RpcServerUseProtseqEp specifies the server endpoint name:

0:000> da 00452008
00452008 "hello"

Note that more complex varieties of RPC servers may use alternative approaches for endpoint name selection that do not utilize the aforementioned API.

When debugging a remote call, finding the server-side in process resolution may prove to be insufficient. Fortunately, we can continue and extract thread information. Consider an endpoint list entry for a running HELLO server:

0314 0000.0001 01 LRPC hello

Let’s examine thread information for this RPC server process:

C:\Program Files\Debugging Tools for Windows>dbgrpc -t -P 314
Searching for thread info ...
PID  CELL ID   ST TID       ENDPOINT LASTTIME
---------------------------------------------
0314 0000.0002 03 000000f0 0000.0001 003ffcad

We can see that a thread associated with cell ID 2 is associated with the endpoint at cell ID 1. If this were a server process serving multiple endpoints, we’d be able to filter the threads of interest by ignoring those associated with other endpoints.

We can use the thread ID returned by dbgrpc to find the thread in the debugger:

C:\Program Files\Debugging Tools for Windows>cdb -p 0x314
Microsoft (R) Windows Debugger Version 6.9.0003.113 X86
Copyright (c) Microsoft Corporation. All rights reserved.
*** wait with pending attach
Symbol search path is: SRV*C:\websymbols*\\.host\Shared Folders\SymStore*http://
msdl.microsoft.com/download/symbols
Executable search path is:
ModLoad: 00400000 00455000 C:\Documents and Settings\AdminUser\Desktop\hellos.
exe
ModLoad: 7c900000 7c9b0000 C:\WINDOWS\system32\ntdll.dll
ModLoad: 7c800000 7c8f5000 C:\WINDOWS\system32\kernel32.dll
ModLoad: 77e70000 77f01000 C:\WINDOWS\system32\RPCRT4.dll
ModLoad: 77dd0000 77e6b000 C:\WINDOWS\system32\ADVAPI32.dll
(314.674): Break instruction exception - code 80000003 (first chance)
eax=7ffde000 ebx=00000001 ecx=00000002 edx=00000003 esi=00000004 edi=00000005
eip=7c901230 esp=0036ffcc ebp=0036fff4 iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=0038 gs=0000 efl=00000246
ntdll!DbgBreakPoint:
7c901230 cc int 3
0:002> ~
0 Id: 314.534 Suspend: 1 Teb: 7ffdd000 Unfrozen
1 Id: 314.f0 Suspend: 1 Teb: 7ffdc000 Unfrozen
. 2 Id: 314.674 Suspend: 1 Teb: 7ffdb000 Unfrozen
0:002> ~1 s
eax=00350020 ebx=00000000 ecx=00144530 edx=ffffffff esi=00144878 edi=00144a80
eip=7c90eb94 esp=0055fe18 ebp=0055ff80 iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
ntdll!KiFastSystemCallRet:
7c90eb94 c3 ret
0:001> kb
ChildEBP RetAddr Args to Child
0055fe14 7c90e399 77e765d3 000007c8 0055ff74 ntdll!KiFastSystemCallRet
0055fe18 77e765d3 000007c8 0055ff74 00000000 ntdll!NtReplyWaitReceivePortEx+0xc
0055ff80 77e76c9f 0055ffa8 77e76ac1 00144878 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCa
lls+0x12a
0055ff88 77e76ac1 00144878 7c90ee18 0012faf8 RPCRT4!RecvLotsaCallsWrapper+0xd
0055ffa8 77e76c87 00144218 0055ffec 7c80b6a3 RPCRT4!BaseCachedThreadRoutine+0x79
0055ffb4 7c80b6a3 00144a80 7c90ee18 0012faf8 RPCRT4!ThreadStartRoutine+0x1a
0055ffec 00000000 77e76c6d 00144a80 00000000 kernel32!BaseThreadStart+0x37
0:001>

Now, let’s add a breakpoint in the server-side implementation of the HelloProc remote call, run the HELLO client and see the context:

0:001> bp hellos!HelloProc
0:001> g
Breakpoint 0 hit
eax=004010f0 ebx=0055fd0c ecx=00000000 edx=00144c00 esi=0055f908 edi=0055f8e4
eip=004010f0 esp=0055f8e4 ebp=0055f8f8 iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
hellos!HelloProc:
004010f0 55 push ebp
0:001> k
ChildEBP RetAddr
0055f8e0 77e799dc hellos!HelloProc
0055f8f8 77ef321a RPCRT4!Invoke+0x30
0055fcf4 77ef36ee RPCRT4!NdrStubCall2+0x297
0055fd10 77e794a5 RPCRT4!NdrServerCall2+0x19
0055fd44 77e7940a RPCRT4!DispatchToStubInC+0x38
0055fd98 77e79336 RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0x113
0055fdbc 77e7be3c RPCRT4!RPC_INTERFACE::DispatchToStub+0x84
0055fdf8 77e7bc99 RPCRT4!LRPC_SCALL::DealWithRequestMessage+0x2db
0055fe1c 77e7bbdd RPCRT4!LRPC_ADDRESS::DealWithLRPCRequest+0x16d
0055ff80 77e76c9f RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x310
0055ff88 77e76ac1 RPCRT4!RecvLotsaCallsWrapper+0xd
0055ffa8 77e76c87 RPCRT4!BaseCachedThreadRoutine+0x79
0055ffb4 7c80b6a3 RPCRT4!ThreadStartRoutine+0x1a
0055ffec 00000000 kernel32!BaseThreadStart+0x37
0:001>

As expected, thread 1 is the one servicing the remote procedure call received at the endpoint. So even if we didn’t know the specific function being called on the server side, we could have followed the worker thread’s execution flow into the indirect call in NdrStubCall2 until arriving at the function of interest.

Another RPC behavior we can notice at this point is the spawning of an additional worker thread by the RPC runtime, since the current one is busy servicing the HelloProc call. While HelloProc is broken into, we note the dbgrpc thread list:

C:\Program Files\Debugging Tools for Windows>dbgrpc -t -P 0x314
Searching for thread info ...
PID CELL ID ST TID ENDPOINT LASTTIME
---------------------------------------------
0314 0000.0002 01 000000f0 0000.0001 0045c6f9
0314 0000.0003 03 00000218 0000.0001 0045c6f9

Notice how two threads are now associated with our endpoint. We can examine the new thread in the debugger:

0:001> ~
0 Id: 314.534 Suspend: 1 Teb: 7ffdd000 Unfrozen
. 1 Id: 314.f0 Suspend: 1 Teb: 7ffdc000 Unfrozen
2 Id: 314.218 Suspend: 1 Teb: 7ffdb000 Unfrozen
0:001> ~2 k
ChildEBP RetAddr
0065fe14 7c90e399 ntdll!KiFastSystemCallRet
0065fe18 77e765d3 ntdll!NtReplyWaitReceivePortEx+0xc
0065ff80 77e76c9f RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x12a
0065ff88 77e76ac1 RPCRT4!RecvLotsaCallsWrapper+0xd
0065ffa8 77e76c87 RPCRT4!BaseCachedThreadRoutine+0x79
0065ffb4 7c80b6a3 RPCRT4!ThreadStartRoutine+0x1a
0065ffec 00000000 kernel32!BaseThreadStart+0x37

The stack trace is consistent with another RPC worker thread on the endpoint. It’s nice of the RPC runtime to provide these thread management services for us.

In a situation where a process has multiple RPC worker threads servicing an endpoint, it can be difficult to figure out which worker thread will pick up the call, unlike in the degenerate case discussed above. In the more complicated cases, we can utilize server call (“SCALL”) information provided by dbgrpc. With the server process at a break and the client process having performed a remote call, we enumerate the server’s calls:

C:\Program Files\Debugging Tools for Windows>dbgrpc -c -P 314
Searching for call info ...
PID CELL ID ST PNO IFSTART THRDCELL CALLFLAG CALLID LASTTIME CONN/CLN
----------------------------------------------------------------------------
0314 0000.0004 02 000 7a98c250 0000.0002 00000009 00000000 0045c6f9 05d8.00d0

This is pretty awesome. The listing notes that the SCALL has cell identifier 0.4. We can get a more verbose information view repeating the above:

C:\Program Files\Debugging Tools for Windows>dbgrpc -l -P 314 -L 0.4
Getting cell info ...
Call
Status: Dispatched
Procedure Number: 0
Interface UUID start (first DWORD only): 7A98C250
Call ID: 0x0 (0)
Servicing thread identifier: 0x0.2
Call Flags: cached, LRPC
Last update time (in seconds since boot):4572.921 (0x11DC.399)
Caller (PID/TID) is: 5d8.d0 (1496.208)

While we used endpoint enumeration and thread cell enumeration to find the server side, we can use SCALL enumeration to find our clients. Let’s see what’s going on at process 0x5d8 in thread d0:

C:\Program Files\Debugging Tools for Windows>cdb -p 0x5d8
Microsoft (R) Windows Debugger Version 6.9.0003.113 X86
Copyright (c) Microsoft Corporation. All rights reserved.
*** wait with pending attach
Symbol search path is: SRV*C:\websymbols*\\.host\Shared Folders\SymStore*http://
msdl.microsoft.com/download/symbols
Executable search path is:
ModLoad: 00400000 00455000 C:\Documents and Settings\AdminUser\Desktop\helloc.
exe
ModLoad: 7c900000 7c9b0000 C:\WINDOWS\system32\ntdll.dll
ModLoad: 7c800000 7c8f5000 C:\WINDOWS\system32\kernel32.dll
ModLoad: 77e70000 77f01000 C:\WINDOWS\system32\RPCRT4.dll
ModLoad: 77dd0000 77e6b000 C:\WINDOWS\system32\ADVAPI32.dll
(5d8.3a0): Break instruction exception - code 80000003 (first chance)
eax=7ffd7000 ebx=00000001 ecx=00000002 edx=00000003 esi=00000004 edi=00000005
eip=7c901230 esp=0035ffcc ebp=0035fff4 iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=0038 gs=0000 efl=00000246
ntdll!DbgBreakPoint:
7c901230 cc int 3
0:001> ~
0 Id: 5d8.d0 Suspend: 1 Teb: 7ffdf000 Unfrozen
. 1 Id: 5d8.3a0 Suspend: 1 Teb: 7ffde000 Unfrozen
0:001> ~0 s
eax=77ea19bb ebx=00145618 ecx=00144a78 edx=00000000 esi=0012fb68 edi=0012fb3c
eip=7c90eb94 esp=0012fab4 ebp=0012fb00 iopl=0 nv up ei pl nz na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000206
ntdll!KiFastSystemCallRet:
7c90eb94 c3 ret
0:000> kb
ChildEBP RetAddr Args to Child
0012fab0 7c90e3ed 77e7ca99 000007c4 00145820 ntdll!KiFastSystemCallRet
0012fab4 77e7ca99 000007c4 00145820 00145820 ntdll!ZwRequestWaitReplyPort+0xc
0012fb00 77e7a326 00145858 0012fb20 77e7a357 RPCRT4!LRPC_CCALL::SendReceive+0x22
8
0012fb0c 77e7a357 0012fb3c 00444290 0012ff18 RPCRT4!I_RpcSendReceive+0x24
0012fb20 77ef3675 0012fb68 00145871 08efa12c RPCRT4!NdrSendReceive+0x2b
*** WARNING: Unable to verify checksum for C:\Documents and Settings\AdminUser\D
esktop\helloc.exe
0012fefc 004011b6 00444290 00444246 0012ff18 RPCRT4!NdrClientCall2+0x222
0012ff10 004010c5 00452010 058dc64f 08efa12c helloc!HelloProc+0x16
0012ff6c 004020d7 00000001 00332fe0 00333050 helloc!main+0xc5
0012ffb8 00401faf 0012fff0 7c816ff7 08efa12c helloc!__tmainCRTStartup+0x117
0012ffc0 7c816ff7 08efa12c 01c8c807 7ffd7000 helloc!mainCRTStartup+0xf
0012fff0 00000000 00401fa0 00000000 78746341 kernel32!BaseProcessStart+0x23
0:000>

We can clearly see the HelloProc client-side stub invoking NdrClientCall2 to perform the remote procedure call to our server process.

Note that the SCALL information also includes beginning of the RPC interface GUID (IfStart) and the slot (ProcNum) in the interface being invoked (think of the RPC interface as a C++ vtable) – this can be useful if we are looking for the server side implementation of an unknown interface and multiple interfaces are being exported by the server process.

You can figure out more techniques for using dbgrpc and the Windbg RPC debugging extension by going over Windbg’s RPC debugging documentation. I found the need for the above primer since the documentation is not exactly organized in tutorial form and can be daunting for the uninitiated.

There is another RPC debugging trick up our sleeve. I shall make an exception of my usual habit and discuss the “other” debugger, Visual Studio’s. The Visual Studio Debugger has an extremely powerful feature, unfortunately missing from Windbg, for RPC and COM debugging. Take a look at the documentation for the Native Debugging options dialog for where to turn it on. It is available as far back as Visual C++ 6.0, though you probably want to use a modern version of the Visual C++ debugger that would be able to use modern PDB symbol files (VC++ 6.0 chokes on XP SP2’s newer PDBs, etc.)

With this debugger feature enabled, you just perform a usual Step Into on the client side call during the debugging session, and instead of being lead into the low-level marshaling code generated by MIDL for the interface in question, another session of the debugger is automagically attached to the server-side process and the server-side thread is broken into at the call site of the server-side function implementation (O… M… G…) – pretty neat, don’t you think? COM folks, take notice – this stuff even works with full-fledged COM objects.

Unfortunately, Microsoft had to blow it by severely crippling this amazing debugger feature in Windows Vista. As if more excuses to dislike it were required, the debugger will no longer automatically locate the server process and attach to it on that OS. You’ll have to preattach the debugger to the server process by hand and only then will the server call be broken into when appropriate. On Vista, you can use the dbgrpc techniques discussed above to figure out which server process you should attach the Visual C++ debugger to. I also noticed the lack of the wonderful auto-attach behavior in one of my debugging sessions on a XP x64 system, although this is not mentioned in the Visual Studio documentation. What a waste!

Now on to RPC debugging across machine boundaries. Obviously, the RPC runtime will not provide us with process and thread identifiers if the call has crossed a machine boundary. For exploring this scenario, we shall modify the HELLO sample to use the named pipes transport (ncacn_np) to the remote HELLO server.

With CCALL information enabled (i.e., “Full” rather than just “Server” state information) we can see where outgoing RPC calls are headed.  Unfortunately, on one hand as soon as the server side responds the call is completed and the CCALL entry is gone. On the other hand, if we set up a breakpoint on the client-side stub (e.g., helloc!HelloProc) the RPC runtime doesn’t even know yet a remote call is about to be made.

If we know which server the outgoing call is headed to, we can break the server-side and thus make the client-side call block while waiting for the server to respond. In this state, we can examine CCALL information. First let’s set up the break on the server:

C:\Program Files\Debugging Tools for Windows>cdb E:\Projects\hello\hellos.exe
Microsoft (R) Windows Debugger Version 6.9.0003.113 X86
Copyright (c) Microsoft Corporation. All rights reserved.
CommandLine: E:\Projects\hello\hellos.exe
Symbol search path is: C:\WINDOWS\Symbols;SRV*E:\SymStore*http://referencesource
.microsoft.com/symbols;SRV*E:\SymStore*http://msdl.microsoft.com/download/symbols
Executable search path is:
ModLoad: 00400000 00455000 hellos.exe
ModLoad: 7c900000 7c9af000 ntdll.dll
ModLoad: 7c800000 7c8f6000 C:\WINDOWS\system32\kernel32.dll
ModLoad: 77e70000 77f02000 C:\WINDOWS\system32\RPCRT4.dll
ModLoad: 77dd0000 77e6b000 C:\WINDOWS\system32\ADVAPI32.dll
ModLoad: 77fe0000 77ff1000 C:\WINDOWS\system32\Secur32.dll
(1320.1304): Break instruction exception - code 80000003 (first chance)
eax=00241eb4 ebx=7ffdf000 ecx=00000007 edx=00000080 esi=00241f48 edi=00241eb4
eip=7c90120e esp=0012fb20 ebp=0012fc94 iopl=0 nv up ei pl nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202
ntdll!DbgBreakPoint:
7c90120e cc int 3
0:000> bp hellos!HelloProc
*** WARNING: Unable to verify checksum for hellos.exe
0:000> g

Then let’s have the call block on the client side by simply running the client. Now let’s examine the client call list:

C:\Program Files\Debugging Tools for Windows>dbgrpc -a
Searching for call info ...
PID CELL ID PNO IFSTART TIDNUMBER CALLID LASTTIME PS CLTNUMBER ENDPOINT
------------------------------------------------------------------------------
0428 0000.003f 0009 4b112204 0000.0000 ffffffff 00019238 09 0000.003d LRPC000004
e4
0710 0000.0001 0000 7a98c250 0000.0000 00000001 00843c7a 0f 0000.0002 \pipe\hell
o
C:\Program Files\Debugging Tools for Windows>dbgrpc -l -P 710 -L 0.1
Getting cell info ...
Client call info
Procedure number: 0
Interface UUID start (first DWORD only): 7A98C250
Call ID: 0x1 (1)
Calling thread identifier: 0x0.0
Call target identifier: 0x0.2
Call target endpoint: \pipe\hello
C:\Program Files\Debugging Tools for Windows>dbgrpc -l -P 710 -L 0.2
Getting cell info ...
Call target info
Protocol Sequence: NMP
Last update time (in seconds since boot):8666.234 (0x21DA.EA)
Target server is: darkstar
C:\Program Files\Debugging Tools for Windows>

Notice how the CCALL information cell is associated with a target information cell containing the name of the remote host servicing the call. If we were unsure which remote calls were being made, we could extract the actual interface calls from the CCALL information entry (alternatively, a network protocol analyzer understanding MSRPC, such as Wireshark or Microsoft Network Monitor, could be used).

Now let’s see how a call from a remote client appears on the server end. We’ll wait for our breakpoint on the server-side stub to fire. At this point we’d have a SCALL entry to consider:

0:000> g
Breakpoint 0 hit
eax=004010f0 ebx=0055fd54 ecx=00000000 edx=00145700 esi=0055f950 edi=0055f92c
eip=004010f0 esp=0055f92c ebp=0055f940 iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
hellos!HelloProc:
004010f0 55 push ebp
0:001> |
. 0 id: 6dc create name: hellos.exe
C:\Program Files\Debugging Tools for Windows>dbgrpc -c -P 6dc
Searching for call info ...
PID CELL ID ST PNO IFSTART THRDCELL CALLFLAG CALLID LASTTIME CONN/CLN
----------------------------------------------------------------------------
06dc 0000.0004 02 000 7a98c250 0000.0002 00000001 00000001 009b0f9e 0000.0003

Notice how instead of the traditional process and thread identifiers, we have what appears to be a cell ID as the caller. Let’s see what information cells 3 and 4 contain:

C:\Program Files\Debugging Tools for Windows>dbgrpc -l -P 6dc -L 0.4
Getting cell info ...
Call
Status: Dispatched
Procedure Number: 0
Interface UUID start (first DWORD only): 7A98C250
Call ID: 0x1 (1)
Servicing thread identifier: 0x0.2
Call Flags: cached
Last update time (in seconds since boot):10162.78 (0x27B2.4E)
Owning connection identifier: 0x0.3
C:\Program Files\Debugging Tools for Windows>dbgrpc -l -P 6dc -L 0.3
Getting cell info ...
Connection
Connection flags: Exclusive
Authentication Level: Default
Authentication Service: None
Last Transmit Fragment Size: 49 (0x1002050)
Endpoint for the connection: 0x0.1
Last send time (in seconds since boot):10162.78 (0x27B2.4E)
Last receive time (in seconds since boot):10162.78 (0x27B2.4E)
Getting endpoint info ...
Process object for caller is 0xA14

Notice that the connection cell contains the remote PID of the caller, 0xA14.

0:001> |
. 0 id: a14 attach name: E:\Projects\hello\helloc.exe
0:001>

Unfortunately, the thread identifier is missing so you’ll have to use CCALL information on the client for that. Even more tragically, dbgrpc fails to name the name of the remote caller! You know it’s PID 0xA14, you just don’t know on what machine… You’ll have to make an educated guess, perhaps with the assistance of a network protocol analyzer.

Occasionally we won’t be in a situation that allows for breaking the server-side to facilitate blocking the client-side call for CCALL information examination. In such cases, we’ll want to break the client-side right after debug information for the call has been registered, but before the call has been sent to the server for completion. The various RPC transports utilize the CCALL::SetDebugClientCallInformation function for this purpose. Let’s see what happens when we break on it, let it do the registration and examine the CCALL table:

0:000> bp rpcrt4!CCALL::SetDebugClientCallInformation
0:000> g
Breakpoint 0 hit
eax=0012faa8 ebx=00000000 ecx=001450a8 edx=00000000 esi=001450a8 edi=0012fb3c
eip=77ec44de esp=0012fa68 ebp=0012fab8 iopl=0 nv up ei pl nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202
RPCRT4!CCALL::SetDebugClientCallInformation:
77ec44de 8bff mov edi,edi
0:000> k
ChildEBP RetAddr
0012fa64 77ea7b73 RPCRT4!CCALL::SetDebugClientCallInformation
0012fab8 77e808d0 RPCRT4!OSF_CCALL::FastSendReceive+0x72
0012fad4 77e80e1f RPCRT4!OSF_CCALL::SendReceiveHelper+0x58
0012fb00 77e7a326 RPCRT4!OSF_CCALL::SendReceive+0x41
0012fb0c 77e7a357 RPCRT4!I_RpcSendReceive+0x24
0012fb20 77ef3675 RPCRT4!NdrSendReceive+0x2b
*** WARNING: Unable to verify checksum for helloc.exe
0012fefc 004011b6 RPCRT4!NdrClientCall2+0x222
0012ff10 004010c5 helloc!HelloProc+0x16
0012ff6c 004020d7 helloc!main+0xc5
0012ffb8 00401faf helloc!__tmainCRTStartup+0x117
0012ffc0 7c816ff7 helloc!mainCRTStartup+0xf
0012fff0 00000000 kernel32!BaseProcessStart+0x23
0:000> gu
eax=00000000 ebx=00000000 ecx=00000002 edx=0000b10e esi=001450a8 edi=0012fb3c
eip=77ea7b73 esp=0012fa88 ebp=0012fab8 iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
RPCRT4!OSF_CCALL::FastSendReceive+0x72:
77ea7b73 3bc3 cmp eax,ebx
0:000>
C:\Program Files\Debugging Tools for Windows>dbgrpc -a
Searching for call info ...
PID CELL ID PNO IFSTART TIDNUMBER CALLID LASTTIME PS CLTNUMBER ENDPOINT
------------------------------------------------------------------------------
0428 0000.003f 0009 4b112204 0000.0000 ffffffff 00019238 09 0000.003d LRPC000004
e4
0504 0000.0001 0000 7a98c250 0000.0000 001440c8 00b10eb8 00 0000.0002

Oops… notice how the name of the endpoint is missing from the CCALL entry at this point! With some disassembly (left as an exercise for the reader) it is clear the caller copies the endpoint name into the debug information buffer right after setting up the entry:
0:000> bp rpcrt4!CCALL::SetDebugClientCallInformation
0:000> g
Breakpoint 0 hit
eax=0012faa8 ebx=00000000 ecx=001450a8 edx=00000000 esi=001450a8 edi=0012fb3c
eip=77ec44de esp=0012fa68 ebp=0012fab8 iopl=0 nv up ei pl nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202
RPCRT4!CCALL::SetDebugClientCallInformation:
77ec44de 8bff mov edi,edi
0:000> gu
eax=00000000 ebx=00000000 ecx=00000002 edx=0000bb8b esi=001450a8 edi=0012fb3c
eip=77ea7b73 esp=0012fa88 ebp=0012fab8 iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
RPCRT4!OSF_CCALL::FastSendReceive+0x72:
77ea7b73 3bc3 cmp eax,ebx
0:000> bp rpcrt4!strncpy
0:000> g
Breakpoint 1 hit
eax=00350034 ebx=00000001 ecx=0012fa79 edx=00000000 esi=001450a8 edi=0000000c
eip=77e952a0 esp=0012fa6c ebp=0012fab8 iopl=0 nv up ei pl nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202
RPCRT4!strncpy:
77e952a0 ff252813e777 jmp dword ptr [RPCRT4!_imp__strncpy (77e71328)] ds:
0023:77e71328={ntdll!strncpy (7c902c80)}
0:000> t
eax=00350034 ebx=00000001 ecx=0012fa79 edx=00000000 esi=001450a8 edi=0000000c
eip=7c902c80 esp=0012fa6c ebp=0012fab8 iopl=0 nv up ei pl nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202
ntdll!strncpy:
7c902c80 8b4c240c mov ecx,dword ptr [esp+0Ch] ss:0023:0012fa78=000000
0c
0:000> gu
eax=00350034 ebx=00000001 ecx=00000000 edx=006f6c6c esi=001450a8 edi=0000000c
eip=77ea7be8 esp=0012fa70 ebp=0012fab8 iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
RPCRT4!OSF_CCALL::FastSendReceive+0xe7:
77ea7be8 83c40c add esp,0Ch
0:000>
C:\Program Files\Debugging Tools for Windows>dbgrpc -a
Searching for call info ...
PID CELL ID PNO IFSTART TIDNUMBER CALLID LASTTIME PS CLTNUMBER ENDPOINT
------------------------------------------------------------------------------
0428 0000.003f 0009 4b112204 0000.0000 ffffffff 00019238 09 0000.003d LRPC000004
e4
07b4 0000.0001 0000 7a98c250 0000.0000 00000001 00bb8b2b 00 0000.0002 \pipe\hello

Ahh, that’s better. But if we examine the server name in the CCALL cell, we see it hasn’t yet been initialized. We need another round of strncpy for that. If we dig further into the transport code, we figure out that it would be better to break right before the function call dispatching the data to the server side. For instance, in the case of the named pipe transport, this would be the call to RPCRT4!OSF_CCALL::SendNextFragment from RPCRT4!OSF_CCALL::FastSendReceive. If we are using the LPC transport instead, other transport functions will be involved. To summarize – breaking the call after CCALL information has been completely registered but before it has been sent to the server is not so easy and is highly transport dependent. However, it is indeed quite possible if your scenario requires it.

And so the RPC debugging primer comes to conclusion. It is a messy ordeal, yet so much cooler than stepping through yet another SOAP web service in Visual Studio, isn’t it? :-)