Nynaeve

Here’s a little known tidbit of information that you might find useful while debugging things:

All the extra windows that you may have open in WinDbg (disassembly, registers, memory, and soforth) actually slow the debugger down. Really.

This is typically only noticible on targets with a non-local connection in between your local debugger and the target. Primarily, then, this applies to kernel debugging and remote debugging.

These windows slow down the debugger because they represent extra information that the debugger must fetch each time the target breaks into the debugger. The debugger engine tends to follow a rule of “lazy fetching”; that is, it will try to avoid asking the target for extra information until it absolutely must do so. If you don’t have your disassembly window open, then, you’ll save the debugger from having to read memory from the disassembly target address on every debugger event that causes a breakin (until you actually explicitly request that information via the “u” command, for instance). There are similar speed consequences for having the registers window open (or setting your default register mask to show all registers, as usermode debugging does by default – this is, I suspect, why kernel debugging defaults to a very minimal register mask that is displayed on each prompt), and for having the memory window open. Keeping these windows open will require the debugger to go out of its way to request extra register information (and to read memory on each debugger event).

The result of having these extra windows open is that the debugger will just seem significantly slower when you are doing things like stepping or hitting breakpoints and continuing execution. If you’ve ever wondered why sometimes it takes the debugger a couple seconds or more to come back after each step or trace operation, this is a big contributor to that (especially in kernel debugging).

So, close those extra windows unless you really need them, as they really do slow things down in kernel debugging and remote debugging scenarios. You’ll find that your kernel debugging and remote debugging experiences are much more responsive without the debugger having all those extra round trip times being inserted on each trace/step/break event.

Oh, and you can also temporarily suspend the behavior of keeping these windows up to date while still keeping them open with a command that was recently introduced to WinDbg: “.suspend_ui flag“, where flag is either 0 to disable UI window refreshing, or 1 to enable it. This is mostly useful for scenarios where you still find the disassembly or memory windows useful, but are about to execute a large number of traces or steps and want to reduce overhead for the duration of that activity only.

I’ve put out SDbgExt 1.09. This is an incremental release of my collection of WinDbg debugger extensions.

The 1.09 release primarily adds support for displaying exception handler data on x64. While there is “some” built-in debugger support for this (via the “.fnent”) command, this support is extremely minimal. You are essentially required to dump the unwind data structures yourself and manually parse them out, which isn’t exactly fun. So, I added support for doing all of that hard work to SDbgExt, via the !fnseh SDbgExt extension (display function SEH data). This support is complementary to the !exchain command supplied by ext.dll for x86 targets.

The “!fnseh” command supports displaying most of the interesting fields of the unwind metadata (besides information on how the prologue works). It also properly supports chained unwind information records (both the documented and undocumented formats). There is also basic support for detecting and processing CL’s C/C++ exception scope tables, if a function uses C language exception handling (__try/__except/__finally).

Here’s a couple quick examples of it:

1: kd> !fnseh nt!CcPinRead
nt!CcPinRead L6a 2B,10 [ U ] nt!_C_specific_handler (C)
> fffff800012bf937 L2 (fffff800012fe4c0 -> fffff80001000000)
> fffff800012bf939 L16 (fffff800012fe4c0 -> fffff80001000000)
> fffff800012bf94f L5b (fffff800012fe4c0 -> fffff80001000000)
> fffff800012bf9aa L48 (fffff800012fe4c0 -> fffff80001000000)
> fffff800012c5199 Ld (fffff800012fe4c0 -> fffff80001000000)
> fffff800012c51a6 L58 (fffff800012fe4c0 -> fffff80001000000)
> fffff800012c51fe L1b (fffff800012fe4c0 -> fffff80001000000)
1: kd> !fnseh nt!CcCopyRead
nt!CcCopyRead Lae 3A,10 [E  ] nt!_C_specific_handler (C)
> fffff80001272c01 Lbf (fffff800012fe2e0 -> fffff800012c4c39)
> fffff80001272cc0 Lc (fffff800012fe2e0 -> fffff800012c4c39)
> fffff800012871f4 L2b (fffff800012fe2e0 -> fffff800012c4c39)
> fffff8000128721f L5a (fffff800012fe2e0 -> fffff800012c4c39)
> fffff800012961b1 L8 (fffff800012fe2c0 -> fffff800012c4b93)
> fffff800012961b9 L56 (fffff800012fe2c0 -> fffff800012c4b93)
> fffff800012c4aae Lcc (fffff800012fe2c0 -> fffff800012c4b93)
> fffff800012c4b7a L19 (fffff800012fe2c0 -> fffff800012c4b93)
1: kd> !fnseh nt!NtAllocateVirtualMemory
nt!NtAllocateVirtualMemory L5e 30,10 [E  ]
nt!_C_specific_handler (C)
> fffff8000103f74f L22 ( -> fffff8000103f771)
> fffff8000103f7f9 L16 ( -> fffff8000109adbf)
> fffff8000105ed3e L46 (fffff800010f3dc0 -> fffff8000105f173)
> fffff8000105f14d L22 ( -> fffff8000105f16f)
1: kd> !fnseh nt!KiSystemCall64
nt!KiSystemCall64 L390 50,0C [EU ]
nt!KiSystemServiceHandler (assembler/unknown)
0:000> !fnseh ntoskrnl + 00001180
401180 L2e 29,06 [  C] <none> (none)
 ntoskrnl!CcUnpinFileDataEx L32 13,05 [   ] <none> (none)

The basic output format is for unwind information as presented by the extension is as so:

EH-start-address LEH-effective-length prologue-size,unwind-code-count [unwind-flags (Exception handler, Unwind handler, Chained unwind information)] exception-handler (exception-handler-language)

Additionally, if the extension thinks that the function in question is using C language support, it will display each of the scope table entries as well (scope table entries divide up the various regions in a function that may have a __try or __except; there is typically only one lowest-level exception handler per function, with the scope table being used to implement multiple __try/__except clauses per function):

> EH-start-address LEH-effective-length (__except-filter-function (optional) -> __except-handler-function)

This information can be useful for tracking down exception filters and handlers to breakpoint on and the like, as EH registrations are completely isolated from code on x64. Note that not all functions may have exception or unwind handlers, as an unwind information is required to be provided for all x64 functions that modify the stack or call other functions.

For more information on how x64 exception handling works under the hood, you might look at my article on the subject, or skape’s paper about x64 Windows binary analysis.

There is a little known feature of WinDbg, ntsd, cdb, kd, and anything else that uses DbgEng to open dump files.

It turns out that with anything powered by DbgEng, anywhere where you could open a dump file (user dump, kernel dump, etc), you can instead open a PE image (.exe/.dll/.sys/etc) and have the debugger treat it as a dump containing just the contents of the selected PE image.

This is actually a relatively useful feature. When you open a PE image as a dump file, the debugger maps it as an image as if it were loaded in-memory as executable code (though it doesn’t actually run any code, just maps it as if it were an executable and not a data file). This gets you an in-memory representation of your exe/dll/sys/other PE file as if you were debugging a live process (or a dump) that had the image in question loaded.

Like a dump debugging session, this is essentially a read-only session; you can’t really modify anything, as there is no target to control. Additionally, there is no real register context either (or stack or heap), although things like initialized and zero filled global variables and executable code belonging to the module will be in-memory. (The preferred image base for the module is used in this situation for basing the requested PE module in the virtual address space constructed for the debugging session.)

After you have loaded the target, you can do anything that you would normally do with a dump for the most part, as far as examining symbols and disassembling the target go. If you need a disassembler with symbol support and can’t start a process or whatnot to contain a PE image, this particular trick is a great quick-n-dirty replacement for a more full-featured disassembler program.

Note that a side effect of opening a PE image in dump mode is that the symbol server is used to retrieve the binary (which might seem a bit strange, until you consider that for dump files, the normal case is that you don’t have the entire binary saved in memory; just enough header information to retrieve the binary from the symbol server). Therefore, make sure that your symbol path is setup correctly before trying this particular trick.

Previously, I had touched some more on the when and why’s as to where hardware breakpoints can be useful.

If you have been following along so far, you should already know the ups and downs of each flavor of breakpoint, and have at least a fair idea as to when you should prefer one to another. There is one other aspect of breakpoint management that I have yet to cover, though, and it is perhaps the most useful feature of breakpoints in WinDbg: conditional breakpoints.

Conditional breakpoints allow you to, as you might imagine, set conditions for breakpoints. That is, the debugger will only actually stop for you to investigate something when both the breakpoint is triggered, and its associated condition is met. These kinds of breakpoints are very useful if you need to stop on a certain function, but only if a certain argument has a certain value (for instance).

However, in WinDbg (and the other DTW/DbgEng debuggers), the support for conditional breakpoints allows you to do much more than that. Specifically, you are permitted to define arbitrary commands that are automatically executed any time a breakpoint is hit. Aside from allowing you to create conditional breakpoints, this also allows you to perform a number of other highly useful tasks in a quick and automated fashion with the debugger. For example, you might want to alter arguments passed to a particular function, or you might want to log arguments (or a stack trace) to a particular function for future analysis.

For example, let’s say that you wanted to see anyone who called CreateFileW, which filename they provided, and what the call stack for each caller might be, what the return value is, and then continue execution. Now, you could do this manually with a “plain” breakpoint and repeating a certain set of commands every time the breakpoint hits, but it would be far superior to automate the whole process.

With DbgEng’s conditional breakpoint support, this is easy. All you need to do in order to have a set of commands that are executed after a breakpoint is to follow the breakpoint statement with a set of commands enclosed in double quotes. (If the commands themselves require the use of double quotes, then you’ll have to escape those, using \”).

From looking at CreateFile on MSDN, we can see that the first argument is a unicode string describing the name of the file that we are to create.

Armed with this information, we can construct a breakpoint that will perform the logging that we are looking for.

Here’s what you might come up with:

0:001> bp kernel32!CreateFileW "du poi(@esp+4);kv;gu;? @eax; g"

Let’s break down this breakpoint a little bit. As usual, the semicolon character (;) is used to separate multiple debugger commands appearing on the same line. The first command is fairly straightforward; it makes use of the du command to display the first argument. The du command simply displays a zero-terminated Unicode string at a given address.

Next, we take a full backtrace (k). After that, we allow execution to continue until we reach the return address of CreateFileW with the gu command (“Go Up one call level”). Finally, we display the eax register (which happens to be the return value register for x86), and continue execution with g.

In action, you might see something like so. In this instance, I am attached to cmd.exe and have executed the command “type c:\config.sys”…

0013fa4c  "c:\\CONFIG.SYS"
ChildEBP RetAddr  
0013e6e4 4ad02f2a kernel32!CreateFileW
0013e730 4ad02e91 cmd!Copen_Work+0x157
0013e744 4ad0dbff cmd!Copen+0x12
0013f7a8 4ad0db62 cmd!TyWork+0x48
0013fc58 4ad0daac cmd!LoopThroughArgs+0x1dd
0013fc6c 4ad05aa2 cmd!eType+0x17
0013fe9c 4ad013eb cmd!FindFixAndRun+0x1f5
0013fee0 4ad0bbba cmd!Dispatch+0x137
0013ff44 4ad05164 cmd!main+0x216
0013ffc0 7c816fd7 cmd!mainCRTStartup+0x125
0013fff0 00000000 kernel32!BaseProcessStart+0x23
Evaluate expression: 132 = 00000084

Essentially, we have turned the debugger into something to inspect function calls and provide us with detailed information about what is happening, in a completely automated fashion.

You can also use this technique for more active intervention in the target as well – for instance, skipping over a function call entirely, modifying what a function does when it is called, or any number of things. Previous articles have, for instance, used conditional breakpoints to alter function behavior (such as making all new virtual memory allocations come from the high end of the address space instead of the low end).

Conditional breakpoints are an invaluable tool in your debugging (and reverse engineering) arsenal; you should absolutely consider them any time that you need any sort of automation to record or alter the behavior of the target, in situations where it is not practical to manually perform the work on every single breakpoint hit.

Next up in this series: Other flow control mechanisms with the debugger, such as stepping and tracing.

In debugging parlance, there are two kinds of breakpoints that you may run across – “hardware” breakpoints, and “software breakpoints”. While the two overlap to a certain degree, it is important to know the differences between the two, and when it is better to use a “hardware” or “software” breakpoint.

For the purposes of this discussion, I’ll stick to using WinDbg on an x86 target. The same general concepts apply to other architectures (especially x64, which works near identically), and the commands to set breakpoints are the same, but details such as where and how many hardware or software breakpoints you may set slightly vary from platform to platform.

In most debugging scenarios, you have probably just used software breakpoints exclusively. Software breakpoints are issued by the bp or bu commands (breakpoint and deferred breakpoint, respectively). These breakpoints are fairly simple and straightforward; they cause the processor to halt in the debugger whenever a thread attempts to execute a piece of code that you set a breakpoint on. Typically, you may set any number of software breakpoints that you want at the same time. Software breakpoints may only be targetted at code; there is no support for setting a “memory breakpoint” via a software breakpoint. Many features such as stepping over a call or going to the return address of a function also implicitly use a temporary software breakpoint that is removed once execution hits it the first time.

Hardware breakpoints, on the other hand, are much more powerful and flexible than software breakpoints. Unlike software breakpoints, you may use hardware breakpoints to set “memory breakpoints”, or a breakpoint that is fired when any instruction attempts to read, write, or execute (depending on how you configure the breakpoint) a specific address. (There is also support for setting breakpoints on I/O port access, but I’ll not cover that feature here, as it is typically of very limited applicibility for every-day debugging tasks.) Hardware breakpoints have some limitations, however; the main limit being that the number of hardware breakpoints that you may have active is extremely limited (on x86, you may only have four hardware breakpoints active at the same time).

Now that we have a basic overview of what the two breakpoint types are, let’s dig a bit deeper and see how they work under the hood, and when you might use them.

The way software breakpoints work is fairly simple. Speaking about x86 specifically, to set a software breakpoint, the debugger simply writes an int 3 instruction (opcode 0xCC) over the first byte of the target instruction. This causes an interrupt 3 to be fired whenever execution is transferred to the address you set a breakpoint on. When this happens, the debugger “breaks in” and swaps the 0xCC opcode byte with the original first byte of the instruction when you set the breakpoint, so that you can continue execution without hitting the same breakpoint immediately. There is actually a bit more magic involved that allows you to continue execution from a breakpoint and not hit it immediately, but keep the breakpoint active for future use; I’ll discuss this in a future posting.

Now, you might be tempted to say that this isn’t really how software breakpoints work, if you have ever tried to disassemble or dump the raw opcode bytes of anything that you have set a breakpoint on, because if you do that, you’ll not see an int 3 anywhere where you set a breakpoint. This is actually because the debugger tells a lie to you about the contents of memory where software breakpoints are involved; any access to that memory (through the debugger) behaves as if the original opcode byte that the debugger saved away was still there.

Now that we know how software breakpoints work at a high level, it’s time to talk about the other side of the story, hardware breakpoints.

Hardware breakpoints are, as you might imagine given the name, set with special hardware support. In particular, for x86, this involves a special set of perhaps little-known registers know as the “Dr” registers (for debug register). These registers allow you to set up to four (for x86, this is highly platform specific) addresses that, when either read, read/written, or executed, will cause the processor to throw a special exception that causes execution to stop and control to be transferred to the debugger.

Given that on x86, you can only have four hardware breakpoints active at once, why would anyone possibly want to use them?

Well, the main strength of hardware breakpoints is that you can use them to halt on non-execution accesses to memory locations. This is actually an extremely useful capability; for example, if you were debugging a memory corruption problem where an initial instance of corruption eventually causes a crash, your initial reaction would probably be something on the lines of “gee, if I know who caused the corruption in the first place, this would be much, much easier to debug” – and this is exactly what hardware breakpoints let you do. In essence, you can use a hardware breakpoint to tell the processor to stop when a specific variable (address) is read or read/written to. You can also use hardware breakpoints to break in on code execution as well, although in the typical case, it is more common to use software breakpoints for that purpose due to the relaxed restrictions on how many breakpoints you may have active at once.

That’s the high level overview of the two main types of breakpoints you’ll encounter in a debugger. In some upcoming postings, I’ll go into some specifics as to how certain edge cases (such as stepping over a call) are implemented, and describe other situations where you’ll find it very useful to use one kind of breakpoint instead of another. I am also planning on discussing how some of the other debugger flow control features are really implemented (such as tracing / single step), and what the consequences of using each flow control method are on the debuggee.