Associating LPC clients and servers

Local Procedure Call (LPC) is a fast, lightweight interprocess communication mechanism used extensively by Windows system components. Microsoft’s documented approach for using LPC is through the higher-level RPC (Remote Procedure Call) API and its “ncalrpc” transport, which wraps the lower-level LPC interfaces. However, many Windows components use the LPC functions from the Native API directly and not through ncalrpc.

Like other remoting architectures, LPC presents a diagnostic challenge. We locate the remoting client thread and expect its flow to contain whatever information we are looking for, only to find it sending a message to the other end, magically receiving a response and continuing its merry way. In order to get a complete view of the complex interactions in the system, the task at hand is locating the server thread and examining its flow until we have the answer we are looking for.

As a case study, we shall consider an unlucky developer that must debug a custom security support provider or authentication package (SSP or AP) that runs in the context of the Windows Local Security Authority (LSA) subsystem. The developer wishes to single-step through the creation of a logon session. This sequence starts at the Winlogon process when it calls the LsaLogonUser API to authenticate a user’s credentials. The following treatise discusses Windows XP and traditional LPC. Vista features asynchronous LPC (ALPC) and is beyond the scope of this post.

We attach the trusty Windbg to the Winlogon process and set up a breakpoint on the API. We resume execution and use the Welcome screen GUI to initiate the actual logon attempt. For convenience, this would best be done with a remote debugging setup to a virtual machine. Once execution is suspended at the API of interest, we perform a trace to get an initial idea of the flow:

0:019> bp secur32!LsaLogonUser
0:019> g
ModLoad: 74980000 74a93000 C:\WINDOWS\system32\msxml3.dll
Breakpoint 0 hit
eax=0006ec78 ebx=00eedfb8 ecx=00000000 edx=00eedfdc esi=759799c4 edi=00eedfe4
eip=77fe33e8 esp=0006ec08 ebp=0006ecb8 iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
Secur32!LsaLogonUser:
77fe33e8 8bff mov edi,edi
0:000> wt
Tracing Secur32!LsaLogonUser to return address 77dec60d
43 0 [ 0] Secur32!LsaLogonUser
1 0 [ 1] ntdll!NtRequestWaitReplyPort
2 0 [ 1] ntdll!ZwRequestWaitReplyPort
2 0 [ 2] ntdll!KiFastSystemCall
1 0 [ 1] ntdll!ZwRequestWaitReplyPort
64 6 [ 0] Secur32!LsaLogonUser


70 instructions were executed in 69 events (0 from other threads)

Function Name Invocations MinInst MaxInst AvgInst
Secur32!LsaLogonUser 1 64 64 64
ntdll!KiFastSystemCall 1 2 2 2
ntdll!NtRequestWaitReplyPort 1 1 1 1
ntdll!ZwRequestWaitReplyPort 2 1 2 1

1 system call was executed

Calls System Call
1 ntdll!KiFastSystemCall

eax=c000006d ebx=00eedfb8 ecx=0006eda8 edx=00100004 esi=759799c4 edi=00eedfe4
eip=77dec60d esp=0006ec44 ebp=0006ecb8 iopl=0 nv up ei ng nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000282
ADVAPI32!L32pLogonUser+0x2cf:
77dec60d 8bf0 mov esi,eax

The debugging developer is disappointed the API simply invokes the NtRequestWaitReplyPort native API, which is used by the client of an LPC port to send a request message and wait synchronously for the server response. The other end is probably the LSA subsystem process, lsass.exe, intuition predicts. First, let’s verify this intuitive guess.

We’ll break again at an invocation of LsaLogonUser in the Winlogon process and this time proceed to the invocation of the LPC port and examine its arguments:


:000> g
ModLoad: 74980000 74a93000 C:\WINDOWS\system32\msxml3.dll
ModLoad: 74980000 74a93000 C:\WINDOWS\system32\msxml3.dll
Breakpoint 0 hit
eax=0006ec78 ebx=00eedfb8 ecx=00000000 edx=00eedfdc esi=759799c4 edi=00eedfe4
eip=77fe33e8 esp=0006ec08 ebp=0006ecb8 iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
Secur32!LsaLogonUser:
77fe33e8 8bff mov edi,edi
0:000> bp ntdll!NtRequestWaitReplyPort
0:000> g
Breakpoint 1 hit
eax=0006eb5c ebx=00eedfb8 ecx=000e000d edx=00eedfdc esi=00000000 edi=0006ebac
eip=7c90e3e1 esp=0006eb44 ebp=0006ec04 iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
ntdll!NtRequestWaitReplyPort:
7c90e3e1 b8c8000000 mov eax,0C8h
0:000> kb 2
ChildEBP RetAddr Args to Child
0006eb40 77fe347e 000006b4 0006eb5c 0006eb5c ntdll!NtRequestWaitReplyPort
0006ec04 77dec60d 000006b4 0006ec78 00000002 Secur32!LsaLogonUser+0xa0

We see that the prototype of NtRequestWaitReplyPort places the LPC port handle as the first argument of the API. We verify this assertion:
0:000> !handle 6b4
Handle 6b4
Type Port

Great. Now what? An LPC port is, in fact, one of the many kinds of NT kernel objects. If we wish to gain access to more information about it, we must access the kernel’s knowledge of the object. This is accomplished through the use of the kernel debugger, kd. Windbg can also act as a GUI wrapper for kernel debugging sessions. As we shall soon find out, the kernel debugger is quite useful even when you have nothing to do with drivers or other kernel-mode code.

Attaching a kernel debugger is covered extensively in the literature and is left as an exercise for the reader. Suffice to say you are much better off using a virtual machine and a named pipe over using a serial or FireWire connection to a physical system, unless some exotic circumstances force you to use an actual machine (i.e., you are debugging authentication of a fingerprint reader product you are developing, etc. and need the physical device to be present in the debugged system)

Handles, except system handles, are process-local. What we want to do is associate the Winlogon port handle with the kernel LPC port object. First, we’ll find the Winlogon process and set the kernel debugger’s context to that process:

0: kd> !process 0 0 winlogon.exe
PROCESS 82bcdda0 SessionId: 0 Cid: 026c Peb: 7ffdd000 ParentCid: 0220
DirBase: 0a3e0000 ObjectTable: e14f1a48 HandleCount: 534.
Image: winlogon.exe

0: kd> .process 82bcdda0
Implicit process is now 82bcdda0
WARNING: .cache forcedecodeuser is not enabled

Good, now lets have a look at the handle, this time from the kernel debugger’s point of view. Although the command has the same name, the kernel debugger command is distinct.

0: kd> !handle 6b4
processor number 0, process 82bcdda0
PROCESS 82bcdda0 SessionId: 0 Cid: 026c Peb: 7ffdd000 ParentCid: 0220
DirBase: 0a3e0000 ObjectTable: e14f1a48 HandleCount: 534.
Image: winlogon.exe

Handle table at e1ba8000 with 534 Entries in use
06b4: Object: e1582f68 GrantedAccess: 001f0001 Entry: e169cd68
Object: e1582f68 Type: (82bed5d0) Port
ObjectHeader: e1582f50 (old version)
HandleCount: 1 PointerCount: 1

We see the LPC port kernel object is at kernel-mode address 0xe1582f68. Each variety of kernel object might have a distinct debugger extension for further examination. In the LPC port case, we shall use the !lpc port command to examine the port:

0: kd> !lpc port e1582f68

Client communication port 0xe1582f68
Handles: 1 References: 1
The LpcDataInfoChainHead queue is empty
Connected port: 0xe15765a0 Server connection port: 0xe17426c8

Server communication port 0xe15765a0
Handles: 1 References: 1
The LpcDataInfoChainHead queue is empty

Server connection port e17426c8 Name: LsaAuthenticationPort
Handles: 1 References: 52
Server process : 8290e4b8 (lsass.exe)
Queue semaphore : 82a0a3e8
Semaphore state 0 (0x0)
The message queue is empty
The LpcDataInfoChainHead queue is empty

Our suspicion that lsass.exe is the server for this LPC port is confirmed. Furthermore, we now know that the LPC port is called LsaAuthenticationPort by this client/server pair. If LSASS has just one port server thread, we’re done. Let us examine it to see if we are that lucky. A word of caution to the uninitiated: debugging LSASS is a messy deal and you can easily deadlock your system by initiating operations that interact with LSA and expect it to be active. For example, do not use a symbol server based on an SMB share, since the network session setup will block while LSA is suspended. For our purposes, a non-invasive, non-suspending session, using the NTSD debugger’s “-pvr” command line switch, would be best for the initial investigation. Once started, we use the debugger to locate stack frames with LPC servers, by looking for the NtReplyWaitReceivePort native API on the stack:

0:000> !findstack ntdll!NtReplyWaitReceivePort
Thread 004, 1 frame(s) match
* 01 008dfe88 7575ba56 ntdll!NtReplyWaitReceivePort+0xc

Thread 006, 1 frame(s) match
* 01 00acfe18 77e765d3 ntdll!NtReplyWaitReceivePortEx+0xc

Thread 007, 1 frame(s) match
* 01 00b8fe40 75738f5c ntdll!NtReplyWaitReceivePort+0xc

Thread 010, 1 frame(s) match
* 01 00e5fe18 77e765d3 ntdll!NtReplyWaitReceivePortEx+0xc

Thread 016, 1 frame(s) match
* 01 0007fe18 77e765d3 ntdll!NtReplyWaitReceivePortEx+0xc

Thread 017, 1 frame(s) match
* 01 00b0fe18 77e765d3 ntdll!NtReplyWaitReceivePortEx+0xc

Thread 019, 1 frame(s) match
* 01 00c8fe18 77e765d3 ntdll!NtReplyWaitReceivePortEx+0xc

GetContextState failed, 0x8007001F
GetContextState failed, 0x8007001F
GetContextState failed, 0x8007001F
Unable to get program counter
GetContextState failed, 0x8007001F
Unable to get current machine context, Win32 error 0n31
GetContextState failed, 0x8007001F
GetContextState failed, 0x8007001F
GetContextState failed, 0x8007001F
GetContextState failed, 0x8007001F
Unable to get current machine context, Win32 error 0n31
ERROR: !findstack: extension exception 0x8007001f.
"Unable to get thread stack"
0:000>

We can ignore the spurious errors from !findstack on the bottom of the output, which are the result of our non-invasive attach. Unfortunately for us, the plot thickens as lsass.exe has a bunch of server threads and it is as of yet unclear which one will have the server-side flow of interest. How can figure out which of the seven potential servers is the right one?

We’ll get started by extracting the server port handle from each thread’s stack. As in the case of the client function, NtReplyWaitReceivePort takes the port handle as its first argument.

0:000> ~4 kb 2
ChildEBP RetAddr Args to Child
008dfe84 7c90e384 7575ba56 000000d0 00000000 ntdll!KiFastSystemCallRet
008dfe88 7575ba56 000000d0 00000000 00000000 ntdll!NtReplyWaitReceivePort+0xc
0:000> ~6 kb 2
ChildEBP RetAddr Args to Child
00acfe14 7c90e399 77e765d3 000002e8 00acff74 ntdll!KiFastSystemCallRet
00acfe18 77e765d3 000002e8 00acff74 00000000 ntdll!NtReplyWaitReceivePortEx+0xc
0:000> ~7 kb 2
ChildEBP RetAddr Args to Child
00b8fe3c 7c90e384 75738f5c 000002cc 00b8fe6c ntdll!KiFastSystemCallRet
00b8fe40 75738f5c 000002cc 00b8fe6c 00000000 ntdll!NtReplyWaitReceivePort+0xc
0:000> ~10 kb 2
ChildEBP RetAddr Args to Child
00e5fe14 7c90e399 77e765d3 000002b4 00e5ff74 ntdll!KiFastSystemCallRet
00e5fe18 77e765d3 000002b4 00e5ff74 00000000 ntdll!NtReplyWaitReceivePortEx+0xc
0:000> ~16 kb 2
GetContextState failed, 0x8007001F
Unable to get current machine context, Win32 error 0n31
^ Illegal thread error in '~16 kb 2'
0:000> ~17 kb 2
GetContextState failed, 0x8007001F
Unable to get current machine context, Win32 error 0n31
^ Illegal thread error in '~17 kb 2'
0:000> ~19 kb 2
ChildEBP RetAddr Args to Child
00c8fe14 7c90e399 77e765d3 000002a8 00c8ff74 ntdll!KiFastSystemCallRet
00c8fe18 77e765d3 000002a8 00c8ff74 00000000 ntdll!NtReplyWaitReceivePortEx+0xc

The non-invasive session has trouble accessing the context of threads 16 and 17. If we don’t find the server handle we are looking for elsewhere, we could retry with an invasive attach. For now, let’s note the server port handles appearing in the stack traces we do have.

Knowing which thread is associated with which port handle, we turn to the kernel debugger for associating the server port object with its handle in the lsass.exe process. Recall that we have the server port object’s address from the output of the !lpc port command on the client port object. We list lsass.exe’s port handles and look for the one associated with it:

0: kd> !process 0 0 lsass.exe
PROCESS 8290e4b8 SessionId: 0 Cid: 02b0 Peb: 7ffdf000 ParentCid: 026c
DirBase: 0a337000 ObjectTable: e1730338 HandleCount: 357.
Image: lsass.exe
0: kd> !handle 0 0 8290e4b8 Port
processor number 0, process 8290e4b8
Searching for handles of type Port
PROCESS 8290e4b8 SessionId: 0 Cid: 02b0 Peb: 7ffdf000 ParentCid: 026c
DirBase: 0a337000 ObjectTable: e1730338 HandleCount: 357.
Image: lsass.exe
Handle table at e173b000 with 357 Entries in use
0018: Object: e1736c58 GrantedAccess: 001f0001 (Protected)
00c4: Object: e172a578 GrantedAccess: 001f0001
00cc: Object: e173d030 GrantedAccess: 001f0001
00d0: Object: e173ff68 GrantedAccess: 001f0001
01c8: Object: e17ba3b8 GrantedAccess: 001f0001
02a8: Object: e1754bc8 GrantedAccess: 001f0001
02b4: Object: e17427a0 GrantedAccess: 001f0001
02b8: Object: e1b6c318 GrantedAccess: 001f0001
02c8: Object: e173d240 GrantedAccess: 001f0001
02cc: Object: e17426c8 GrantedAccess: 001f0001
02e8: Object: e173f410 GrantedAccess: 001f0001
030c: Object: e1742470 GrantedAccess: 001f0001
0314: Object: e1742548 GrantedAccess: 001f0001
033c: Object: e1744e58 GrantedAccess: 001f0001
03a4: Object: e17a4e00 GrantedAccess: 001f0001
03b0: Object: e17a4cc0 GrantedAccess: 001f0001
03c0: Object: e17a9640 GrantedAccess: 001f0001
03c8: Object: e17f29f8 GrantedAccess: 001f0001
03cc: Object: e17deb48 GrantedAccess: 001f0001
03d4: Object: e17f53d0 GrantedAccess: 001f0001
03dc: Object: e17f7dd8 GrantedAccess: 001f0001
03e4: Object: e17f7c10 GrantedAccess: 001f0001
03f4: Object: e190abf0 GrantedAccess: 001f0001
03f8: Object: e17fb690 GrantedAccess: 001f0001
0404: Object: e1b43310 GrantedAccess: 001f0001
0418: Object: e19149b8 GrantedAccess: 001f0001
0440: Object: e1942030 GrantedAccess: 001f0001
0454: Object: e19184a0 GrantedAccess: 001f0001
0458: Object: e1b3a770 GrantedAccess: 001f0001
0464: Object: e1758170 GrantedAccess: 001f0001
0480: Object: e1601ab8 GrantedAccess: 001f0001
049c: Object: e1cf59d8 GrantedAccess: 001f0001
04a4: Object: e13082a8 GrantedAccess: 001f0001
04f4: Object: e1366490 GrantedAccess: 001f0001
0508: Object: e1b466c0 GrantedAccess: 001f0001
0514: Object: e1c9f450 GrantedAccess: 001f0001
0564: Object: e1b5a640 GrantedAccess: 001f0001
0580: Object: e15765a0 GrantedAccess: 001f0001
05cc: Object: e1d814e0 GrantedAccess: 001f0001

Woah, that’s quite a list. We note the handle pointing to the server port object is 0x2cc. Cross-referencing with the handle stacks we extracted from the user-mode debugger, we hit the gold and figure out thread 7 is the thread we are looking for:


0:000> ~7 k
ChildEBP RetAddr
00b8fe3c 7c90e384 ntdll!KiFastSystemCallRet
00b8fe40 75738f5c ntdll!NtReplyWaitReceivePort+0xc
00b8ff74 75738d66 LSASRV!LpcServerThread+0xaf
00b8ffb4 7c80b6a3 LSASRV!LsapThreadBase+0x91
00b8ffec 00000000 kernel32!BaseThreadStart+0x37

Yup, that looks about right. We now know we should set our breakpoint at this thread, as it is expected to return from the blocking NtReplyWaitReceivePort system call upon an incoming LPC message on LsaAuthenticationPort. At that point, we can follow execution as the LSA server DLL assigns handling the call to a worker thread or whatever else it may be up to.

Once again, kd saves the day.

Advertisements

3 thoughts on “Associating LPC clients and servers

  1. Brilliant!
    My methods usually rely on guesswork, spraying a bunch of breakpoints in the context of the assumed server process at key locations where I’m guessing it might hit and then tracing back with the stack. Once I’ve found the relevant code I either resolve to static means or debug dynamically at the handler of the LPC call.
    Needless to say, your tactics are much more refined.
    Usually when it comes to Windows, the general under-the-hood know-how is enough to make a solid guess of where to look and pretty soon you can finger the culprit. However in the case of 3rd party software that uses system mechanisms such as LPC the intuitive approach can fail and so methodological analysis is a must.
    Now I wonder if I can make a monitor application for catching invocation through LPC/RPC so that I can set it to record, trigger an application’s response to an event and instantly get a narrowed-down list of where to look.

    Anyway, thank you for another great post.

  2. Pingback: Remote Procedure Call debugging « KK’s Blog

  3. You can also debug lsass.exe relatively safely with a dbgsrv.exe on the target machine and a windbg “smart client” on the dev box talking over IP. This keeps the symbols on your dev box.

    I’m using the “clicon” reverse connection right now, but I don’t recall if that’s necessary.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s