Replacing boot load drivers with the Windows Boot Debugger

Recently, I’ve been assigned to work on fixing several bugs in a Windows file system filter driver. Debugging native code has always been characterized by the tedious and cumbersome modify, compile and link, copy, run, repeat… cycle, but in the case of kernel-mode development, the overhead of that cycle is even more acute.

I’ve found that booting the target system or virtual machine every time you want to replace a driver file with an updated build and then rebooting to have the new driver loaded significantly prolongs the cycle. Therefore, I was happy to discover Windbg’s .kdfiles command.

The .kdfiles command configure’s the kernel debugger’s driver replacement map. Whenever the NT Memory Manager attempts to load a driver image, it consults the kernel debugger, if attached, asking it for an alternative driver image. If the debugger has one, it is transmitted over the kernel debugging connection from the host to the target, and used in lieu of the target’s local driver image.

Using the driver replacement map makes it easier to replace a driver with an updated version. However, in its usual form, the replacement map feature has a significant limitation – it cannot replace boot load drivers.

To understand the logic behind this restriction, one must consider the nature of boot driver loading. While demand-start drivers are started by the user-mode Service Control Manager (SCM) and system-start drivers are loaded by NTOSKRNL’s IoInitSystem function, boot drivers are, as their name suggests, required for the system to boot and are therefore loaded by osloader, a part of ntldr (this description is for pre-Vista systems).

By the time the NT kernel is up and its Memory Manager consults the kernel debugger and its driver replacement map, it is far too late to do anything about those drivers which have been pre-loaded by the OS loader. The initial breakpoint offered by the kernel debugger is simply too late.

Fortunately, Microsoft recognized the importance of providing a driver replacement map for boot load drivers and provides a somewhat esoteric solution in the form of the debug version of NTLDR.

The debug version of NTLDR expects the kernel debugger to attach to it during system startup. Unlike the kernel debugger, it is not configured with the boot.ini file and is always configured to a 115,200 baud connection on the COM1 serial port.

The documentation for .kdfiles points out that the Windows Driver Kit (WDK) bundles a debug version of NTLDR in the debug subdirectory. However, such a file is nowhere to be found there, probably because the WDK now contains the Vista checked kernel in its debug directory and the modern Vista boot loader is distinct from NTLDR. More on Windows Vista later, but for now let’s concentrate on Windows XP.

Failing to locate the debug NTLDR in the WDK, I turned back in time to the Windows Server 2003 SP1 IFS Kit, a variant of the Windows Server 2003 SP1 DDK for file system and file system filter developers. I was glad to find the ntldr_dbg file in its debug subdirectory.

However, my happiness quickly turned to disappointment when I replaced the original NTLDR with ntldr_dbg in a Windows XP virtual machine. The system refused to boot, claiming that NTLDR was corrupt. Since the debug directory in the IFS kit contains checked kernel binaries for Windows Server 2003 SP1, I figured that the provided version of ntldr_dbg is a match for that version, as well.

I turned to the archives, so to speak, and dusted off old MSDN Subscription CDs. I eventually turned up the rather antiquated Windows XP SP1 DDK. In there, I found another version of ntldr_dbg. I placed it as required and this time the system booted successfully.

It is unfortunate that one has to dig up the DDK of yore to locate the boot debugger. It really ought to be more accessible.

With the debug version of NTLDR is in place, when you boot the system, right before the OS loader menu appears, you see the following message:
Boot Debugger Using: COM1 (Baud Rate 115200)

Once the message is displayed, NTLDR blocks waiting for a kernel debugger to connect. I start the kernel debugger the way I’d usually start it:
windbg -b -k com:pipe,port=\\.\pipe\com_1

Soon enough, however, it is evident that this is no ordinary kernel debugging session:

Microsoft (R) Windows Debugger Version 6.9.0003.113 X86
Copyright (c) Microsoft Corporation. All rights reserved.

Opened \\.\pipe\com_1
Waiting to reconnect...
BD: Boot Debugger Initialized
Connected to Windows Boot Debugger 2600 x86 compatible target, ptr64 FALSE
Kernel Debugger connection established. (Initial Breakpoint requested)
Symbol search path is: C:\WINDOWS\Symbols;SRV*E:\SymStore*;SRV*E:\SymStore*
Executable search path is:
Module List address is NULL - debugger not initialized properly.
WARNING: .reload failed, module list may be incomplete
KdDebuggerData.KernBase < SystemRangeStart
Windows Boot Debugger Kernel Version 2600 UP Checked x86 compatible
Primary image base = 0x00000000 Loaded module list = 0x00000000
System Uptime: not available
Break instruction exception - code 80000003 (first chance)
0041cf70 cc int 3

Windbg has attached to the Windows Boot Debugger, a debugging environment provided by the debug version of NTLDR at a very early stage of system startup, well before the NT kernel has been loaded. Indeed, the initial breakpoint at the boot debugger occurs before an OS to start has been selected at the loader boot menu.

With the boot debugger at its initial breakpoint, we can set up the driver replacement map as desired. For instance, we can replace NTFS and NDIS with their counterparts from the checked build of Windows XP:

kd> .kdfiles -m \WINDOWS\system32\drivers\Ntfs.sys C:\Stuff\xpsp3checked\Ntfs.sys
Added mapping for '\WINDOWS\system32\drivers\Ntfs.sys'
kd> .kdfiles -m \WINDOWS\system32\drivers\Ndis.sys C:\Stuff\xpsp3checked\Ndis.sys
Added mapping for '\WINDOWS\system32\drivers\Ndis.sys'
kd> g
BD: osloader.exe base address 00400000
BD: \WINDOWS\system32\NTKRNLMP.CHK base address 80A02000
BD: \WINDOWS\system32\HALMACPI.CHK base address 80100000
BD: \WINDOWS\system32\KDCOM.DLL base address 80010000
BD: \WINDOWS\system32\BOOTVID.dll base address 80001000
BD: \WINDOWS\system32\DRIVERS\ACPI.sys base address 8014C000
BD: \WINDOWS\system32\DRIVERS\WMILIB.SYS base address 80007000
BD: \WINDOWS\system32\DRIVERS\pci.sys base address 80062000
BD: \WINDOWS\system32\DRIVERS\isapnp.sys base address 80012000
BD: \WINDOWS\system32\DRIVERS\compbatt.sys base address 80009000
BD: \WINDOWS\system32\DRIVERS\BATTC.SYS base address 8000C000
BD: \WINDOWS\system32\DRIVERS\intelide.sys base address 8001C000
BD: \WINDOWS\system32\DRIVERS\PCIIDEX.SYS base address 8017A000
BD: \WINDOWS\System32\Drivers\MountMgr.sys base address 80181000
BD: \WINDOWS\system32\DRIVERS\ftdisk.sys base address 8018C000
BD: \WINDOWS\System32\drivers\dmload.sys base address 8001E000
BD: \WINDOWS\System32\drivers\dmio.sys base address 801AB000
BD: \WINDOWS\System32\Drivers\PartMgr.sys base address 801D1000
BD: \WINDOWS\System32\Drivers\VolSnap.sys base address 801D6000
BD: \WINDOWS\system32\DRIVERS\atapi.sys base address 801E3000
BD: \WINDOWS\system32\DRIVERS\vmscsi.sys base address 80073000
BD: \WINDOWS\system32\DRIVERS\SCSIPORT.SYS base address 801FB000
BD: \WINDOWS\system32\DRIVERS\disk.sys base address 80213000
BD: \WINDOWS\system32\DRIVERS\CLASSPNP.SYS base address 8021C000
BD: \WINDOWS\system32\drivers\fltmgr.sys base address 80229000
BD: \WINDOWS\system32\DRIVERS\sr.sys base address 802A7000
BD: \WINDOWS\System32\Drivers\KSecDD.sys base address 802B9000
KD: Accessing 'C:\Stuff\xpsp3checked\Ntfs.sys' (\WINDOWS\System32\Drivers\Ntfs.sys)
File size 814K.... ....BD: Loaded remote file \WINDOWS\System32\Drivers\Ntfs.sys

BlLoadImageEx: Pulled \WINDOWS\System32\Drivers\Ntfs.sys from Kernel Debugger
BD: \WINDOWS\System32\Drivers\Ntfs.sys base address 802D0000
KD: Accessing 'C:\Stuff\xpsp3checked\Ndis.sys' (\WINDOWS\System32\Drivers\NDIS.sys)
File size 424K.... ....BD: Loaded remote file \WINDOWS\System32\Drivers\NDIS.sys

BlLoadImageEx: Pulled \WINDOWS\System32\Drivers\NDIS.sys from Kernel Debugger
BD: \WINDOWS\System32\Drivers\NDIS.sys base address 804DC000
Shutdown occurred...unloading all symbol tables.
Waiting to reconnect.

We can see that the boot debugger picked up our driver replacements and transferred them from the host to the target through the kernel debugger connection. Alas, this can be a lengthy process for an obese driver over the 115,200 baud link…

Beyond being useful for replacing your own drivers, which is what I had in mind when I looked into this feature, the boot debugger can be used to easily go back and forth between Windows free build and checked build operating system components, as illustrated above. However, such use is not without its problems.

For one, replacing the kernel and the HAL with their checked counterparts through the driver replacement map does not work. An error citing kernel corruption results from such an attempt. The traditional way of using a checked kernel, by placing an appropriate entry in boot.ini, is still required.

When testing a file system filter driver, apart from using the checked version of the I/O Manager through the use of the checked NT kernel, it is advantageous to use checked versions of underlying file system drivers such as NTFS. The checked versions can assert when you pass on requests to them in a way which violates the file system’s locking hierarchy and which may lead to deadlocks. Replacing the NTFS driver with the driver replacement map feature worked as expected, apart from causing NDIS to bugcheck during system boot with some sort of paging error. The issue was resolved by replacing NDIS with its checked counterpart through the driver replacement map, as well.

However, for a reason I do not understand, when placing the checked build of the Filter Manager, useful for debugging file system minifilters, there was no such luck. The boot loader complained after transferring the checked Filter Manager that the NTFS driver was corrupt. I disabled System File Protection and replaced the free drivers with the checked drivers on disk, the traditional way and the system booted with the checked NTFS and Filter Manager successfully. So it appears that the boot-time driver replacement map feature can be a bit flaky…

It is probably best to place checked operating system components the traditional way and only replace your own, frequently modified drivers with the boot debugger and the driver replacement map.

So much for Windows XP and the legacy NTLDR. But what about Windows Vista?

At first, the situation looked promising. In Windows Vista, the boot debugger is built-in. It can, for instance, be enabled for an existing boot entry with the Boot Configuration Database editor from an elevated command prompt:

C:\Windows\system32>bcdedit /enum

Windows Boot Manager
identifier {bootmgr}
device partition=C:
description Windows Boot Manager
locale en-US
inherit {globalsettings}
default {current}
displayorder {current}
toolsdisplayorder {memdiag}
timeout 30

Windows Boot Loader
identifier {current}
device partition=C:
path \Windows\system32\winload.exe
description Microsoft Windows Vista
locale en-US
inherit {bootloadersettings}
osdevice partition=C:
systemroot \Windows
resumeobject {694d30db-e737-11dc-814f-e01223f3682a}
nx OptIn

Windows Boot Loader
identifier {5761b19a-1e8a-11dd-bcd4-000c29797dc6}
device partition=C:
path \Windows\system32\winload.exe
description Debugging
locale en-US
inherit {bootloadersettings}
osdevice partition=C:
systemroot \Windows
resumeobject {694d30db-e737-11dc-814f-e01223f3682a}
nx OptIn
debug Yes

C:\Windows\system32>bcdedit /bootdebug {5761b19a-1e8a-11dd-bcd4-000c29797dc6} ON

The operation completed successfully.

Unlike the XP boot debugger, the Vista boot debugger is set for a specific boot loader menu entry. Once we reboot and pick the entry for which boot debugging is enabled, we can attach:

Microsoft (R) Windows Debugger Version 6.9.0003.113 X86
Copyright (c) Microsoft Corporation. All rights reserved.

Opened \\.\pipe\com_1
Waiting to reconnect...
BD: Boot Debugger Initialized
Connected to Windows Boot Debugger 6001 x86 compatible target, ptr64 FALSE
Kernel Debugger connection established. (Initial Breakpoint requested)
Symbol search path is: C:\WINDOWS\Symbols;SRV*E:\SymStore*;SRV*E:\SymStore*
Executable search path is:
Windows Boot Debugger Kernel Version 6001 UP Free x86 compatible
Primary image base = 0x00584000 Loaded module list = 0x00684e78
System Uptime: not available
Break instruction exception - code 80000003 (first chance)
005bce88 cc int 3
kd> k
ChildEBP RetAddr
00120c6c 005b0862 winload!RtlpBreakWithStatusInstruction
00120e84 005b0760 winload!vDbgPrintExWithPrefixInternal+0x100
00120e94 0058bdaf winload!DbgPrint+0x11
00120eb0 0058bf6d winload!BlBdStart+0x81
00120f48 005a2f88 winload!BlBdInitialize+0x172
00120f64 005a28c2 winload!InitializeLibrary+0x168
00120f7c 0058513a winload!BlInitializeLibrary+0x42
00120fe8 0044646a winload!OslMain+0x13a
WARNING: Frame IP not in any known module. Following frames may be wrong.
00000000 f000ff53 0x44646a
00000000 00000000 0xf000ff53

We can see that in Vista, the boot debugger’s initial break is in the new winload.exe, replacing the osloader.exe embedded in ntldr of yesteryear. At this point the boot load drivers have yet to be loaded, so it would be perfect to set the .kdfiles driver replacement map at this point.

Alas, no such luck. It turns out the boot load driver replacement map feature is MIA in Windows Vista. This is confirmed by Microsoft’s Doron Holan in a reply to a post (free registration required) in OSR’s WINDBG mailing list. It is unclear what is the point of bundling the boot debugger with the regular operating system, unlike in the case of the hard to find ntldr_dbg for XP, only for it to be completely useless… Anyone using the boot debugger for purposes other than boot load driver replacement is probably working for Microsoft, so why should the boot debugger be a part of the OS if it is now missing what seems to be its most important functionality?

Hopefully the boot load driver replacement map will make a comeback in the Windows 7 boot debugger…


5 thoughts on “Replacing boot load drivers with the Windows Boot Debugger

  1. Once I had to debug a system’s boot sector. That’s almost as early as it gets.
    I was using QEMU, a Pentium emulator that’s surprisingly speedy. QEMU can provide a gdb interface so you can debug the machine starting at the first instruction that is read from the virtual disk drive.
    I guess if you’re entirely out of luck, you can always debug using this technique. However if you prefer windbg over gdb (and who wouldn’t) then you’re in a mess. Your best bet would be to write a proxy that intercepts the traffic coming in and going out of the debug port/pipe and translate between windbg-ish and gdb-ish and vice versa. If performance is not an issue, you can actually do it in Python.
    I don’t know what windbg’s debug protocol looks like, but according to the documentation the gdb protocol is fairly straightforward. Heck, they might even look very much alike for all I know.

    • I think Bochs also has that kind of gdb integration.

      For debugging boot sectors, using an emulator is as good as it can get, but once you’re debugging something beyond that I think that using an emulator can become quite a horrendous experiencee

  2. Thank you for this wonderful service. It’s so appreciated in this day when every web site is geared toward the material reward they get when they sell something that probaly belongs to the American people.
    Drivers of course don’t and for that reason you’re going in my Favorites file. Thank you for your time and attention.
    Jack Manger

  3. Hi,
    Interesting article!!!
    I used ntldr from winddk 2003 debug folder but system failed to boot with an error ntldr is corrupt. I could not find the winxp ddk kit to get the respective ntldr. Could you please share debug version of ntldr from winddk xp?

  4. Pingback: Ring 0 debugging and Windbg – part 1 « L0werring's Blog

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s