Today, I’m heading out to Redmond for this coming Blue Hat. My first time going to Blue Hat, but it’ll hopefully be interesting, to say the least :)
I’m definitely looking forward to seeing some of the presentations, and meeting some interesting people. along the way.
When you’re debugging a program, the last thing you want to have to deal with is the debugger not working properly. It’s always frustrating to get sidetracked on secondary problems when you’re trying to focus on tracking down a bug, and especially so when problems with your debugger cause you to lose a repro or burn excessive amounts of time waiting around for the debugger to finish doing who knows what that is taking forever.
a
This is something that I get a fair amount of questions about from time to time, and so I’ve compiled a short list of some common issues that one can easily get tripped up by (and how to avoid or solve them).
To fix this problem, use the ntsd executable in the debugger installation directory.
If you allow a large number of unqualified breakpoints to become saved in a default workspace, this can make the debugger appear to be extremely slow no matter what program you are debugging.
To avoid getting bitten by this problem, don’t use unqualified breakpoints (breakpoints without a modulename! prefix on their address expression) unless absolutely necessary. Also, it’s typically a good idea to clear all your breakpoints before you save your workspace if you don’t need them to be saved for your next debugging session with that debugger workspace (by default, bu breakpoints are persisted in the debugger workspace, unlike bp breakpoints which go away after every debugging session). If you are in the habit of saving the workspace every time you attach to a running process, and you often use bu breakpoints, this will tend to clutter up the user default workspace and can quickly lead to very poor debugger performance if you’re not careful.
You can use the bc command to delete breakpoints (bc * to remove all breakpoints), although you will need to save the workspace to persist the changes. If the problem has gotten to the point where it’s not possible to even get past module loading in a reasonable amount of time so that you can use bc * to clear out saved breakpoints, you can remove the contents of the HKCU\Software\Microsoft\Windbg\Workspaces registry key and subkeys to return WinDbg to a pristine state. This will wipe out your saved debugger window positions and other saved debugger settings, so use it as a last resort.
Use !sym noisy, followed by .reload /f to determine what part of your symbol path is not working correctly. Then, fix or remove the offending part of the symbol path.
This problem can also occur when you are debugging a program that is in the packet path for packets destined to a location on the symbol path. In this case, the typical workaround I recommend is to set an empty symbol path, attach to the process in question, write a dump file, and then detach from the process. Then, restore the normal symbol path and open the dump file in the debugger, and issue a .reload /f command to force all symbols to be pre-cached ahead of time. After all symbols are pre-cached in the downstream store cache, change the symbol path to only reference the downstream store cache location and not any UNC or HTTP symbol server paths, and attach the debugger to the process in the packet path for symbol server access.
If you turn on noisy symbol loading (!sym noisy), a more descriptive error is typically given. If you see a complaint about E_PDB_CORRUPT, then you are probably falling victim to this issue. The debugger output that indicates this problem would look like something along the lines of this:
DBGHELP: c:\symbols\ntdll.pdb\2744327E50A64B24A87BDDCFC7D435A02\ntdll.pdb – E_PDB_CORRUPT
If you encounter this problem, simply delete the .pdb named in the error message and retry loading symbols via the .reload /f <modulename> command.
This problem is especially common if you are debugging an svchost instance, as there are a lot of important but unrelated pieces of code running in the various svchost instances, some of which are critical for network symbol server support to work. If you are debugging a process in the critical path for network symbol server support, and you have a symbol path with a network component set, then you may cause the debugger to deadlock (hang forever) the first time you try and load symbols.
One example of a situation that can cause this is if you are debugging code in the same svchost instance as the DNS cache service. In this case, when you try to load symbols and you have an HTTP symbol server link in your symbol path, the debugger will deadlock because it will try and make an RPC call to the DNS cache service when it tries to resolve the hostname of the server referenced in your symbol path. Because the DNS cache service will never respond until the debugger resumes the process, and the debugger will never resume the process until it gets a response from the RPC request to the DNS cache service, your debugging session will hang indefinitely.
Note that if you are simply debugging something in the packet path of a symbol server store, you will typically see the debugger become unresponsive for long periods of time but not hang completely. This is because the debugger can handle network timeouts (if somewhat slowly) and will eventually fail the request to the network symbol path. However, if the debugger tries to make an IPC request of some sort to the process being debugged, and the IPC request doesn’t have any built-in timeout (most local IPC mechanisms do not), then the debugger session will be lost for good.
This problem can be worked around similarly to how I typically recommend users deal with slow module loading or failed symbol server accesses with a program in the packet path for a symbol server referenced in the symbol path. Specifically, it is possible to pre-cache all symbols for the process by creating a dump of the process from a debugger instance with an empty symbol path, and then detaching and opening the dump with the full symbol path and forcing a download of all symbols. Then, start a debugging session on the live process with a symbol path that references only the local downstream store into which symbols were being downloaded to in order to prevent any dangerous network accesses from happening.
Another common way to get yourself into this sort of debugger deadlock problem is to use the clipboard to paste into WinDbg while you are debugging a program that has placed something into the clipboard. This results in a similar deadlock as WinDbg may get blocked on a DDE request to the clipboard owner, which will never respond by virtue of being debugged. In that case, the workaround is simply to be careful about copying or pasting text into or out of WinDbg.
Make sure that all peers in the remote debugging scenario are using the (same) latest debugger version. If you mix and match debugger versions with -remote, things will often break in strange and hard to diagnose ways in my experience (there doesn’t seem to be a whole lot of graceful support for backwards or forwards compatibility with respect to the debugger remoting protocol).
Also, several recent releases of the debugger package didn’t work at all in remote debugging mode on Windows 2000. This is, as far as I know, fixed in the latest release.
Most of these problems are simple to fix or avoid once you know what to look for (although they can certainly burn a lot of time if you’re caught unaware, having done that myself while learning about these “gotchas”).
If you’re experiencing a weird WinDbg problem, you should also not be shy about debugging the malfunctioning debugger instance itself. Often, taking a stack trace of all threads in the problematic debugger instance will be enough to give you an idea of what sort of problem is holding things up (remember that the Microsoft public symbol server has symbols for the debugger binaries as well as the OS binaries).
A few days ago, VMware put out VMware Server 1.0.4. Although there appear to be some plain old bug fixes, the real news here is that a couple of very nasty security bugs that have been fixed, including “VM-breakout” class bugs that could allow a malicious VM to compromise the host.
I’d imagine that it’s only a matter of time before VM monitor / hypervisor bugs become as big a deal as standard operating system bugs. For those that are already deploying virtualized infrastructure, that is probably already true even now to an extent.
What’s nasty about VM breakout bugs is that they can very easily lead to large numbers of machines being compromised in a fairly stealthy way. There’s been a whole lot of good discussion about hypervisor-based rootkits and malware recently. Although detecting an unexpected layering of hypervisor is one thing, telling the difference between a hypervisor that’s supposed to be running and a compromised hypervisor that’s supposed to be running, from the context of a guest, is an entirely different factor altogether. To a clever attacker, a hypervisor compromise is a pretty scary thing.
Now, I’m not trying to call fire and brimstone down on VMware or anything like that, and the bug in this case is reportedly not remotely exploitable without remote admin access to a guest. But all the same, hypervisor / VM monitor bugs are certainly nothing to be shaking a stick at.
Anyways, if you use VMware and you’ve got VMs that are either untrusted or allow non-admistrative access, it’s time to (borrowing a friend’s term) ready the patch brigade if you have not already.
From time to time, it can be useful to save a chunk of memory for whatever reason when you’re debugging a program. For instance, you might need to capture a long buffer argument to a function for later analysis (perhaps with a custom analysis tool outside the scope of the debugger).
There are a couple of options built in to the debugger to do this. For example, if you just want to save the contents of memory for later perusal, you could always write a complete minidump of the target. However, this has a few downsides; for one, unless you build in dump file processing capability into your analysis program, dump files are typically going to be less than easily accessible to simple analysis tools. (Although one could write a program utilizing MiniDumpReadDumpStream, this is more work than necessary.)
Furthermore, complete dumps tend to be large, and in the case of a kernel debugger connection over serial port, it can take many hours to save a kernel memory dump just to gain access to a comparatively small region of memory.
Instead of writing a dump file, another option is to use one of the display memory commands to save the contents of memory to a debugger log file. For instance, one might use “db address len“, write it to a log file, and parse the output. This is much less time-consuming than a kernel memory dump over kd, and in some cases it might be desirable to have the hex dump for you (that db provides) in plain text, but if one just wants the raw memory contents, that too is less than ideal.
Fortunately, there’s a third option: the .writemem command, which as the name implies, writes an arbitrary memory range to a file in raw binary form. There are two arguments, a filename and a range. For instance, one usage might be:
.writemem C:\\Users\\User\\Stack.bin @rsp L1000
This command would write 0x1000 bytes of stack to the file. (Remember that address ranges may include a space-delimited component to specify the length.)
The command works on all targets, including when one is using the kernel debugger, making it the command of choice for writing out arbitrary chunks of memory.
There also exists a command to perform the inverse operation, .readmem, which takes the same arguments, but instead reads memory from the file given and writes it to the specified address range. This can be useful for anything between substituting large arguments to a function out at run-time to applying large patches to replace non-trivial sections of code as a whole.
Furthermore, because the memory image format used by both commands is just the raw bits from the target, it becomes easy to work with the written out data with a standard hex editor, or even a disassembler. (For instance, another common use case of .writemem is to when dealing with self-modifying code, write the code out to a file after it has been finalized, and then load the resulting raw memory image up as raw opcodes in a more full-featured disassembler than the debugger.)
I am not a happy camper.
Today, I got in to work and unpacked my laptop from my laptop bag and discovered that it had gone into hibernation due to a critically low battery event. That was fairly strange, because last night I had suspended my laptop (fully charged) and placed it into my laptop bag. Somehow, it managed to consume a full battery charge between my putting it into my bag last night, and my getting in to work.
This is obviously not good, because it meant that the laptop had to have been powered on while in my laptop bag for it to have possibly used that much battery power (a night in suspend is a drop in the bucket as far as battery life is concerned). Let me tell you a thing or two about laptop bags: they’re typically padded (in other words, also typically insulated to a degree) and generally don’t have a whole lot of ventilation potential designed into them, at least as far as laptop compartments go. Running a laptop in a laptop bag for a protracted period of time is a bad thing, that’s for certain.
Given this, I was not all that happy to discover that my laptop had indeed been resumed and had been running overnight in my laptop bag until the battery got low enough for it to go into emergency hibernate mode. (Fortunately, it appears to have sustained no permanent damage from this event. This time….)
So, I set out to find out the culprit. The event log showed the system clearly waking just seconds after the clock hit 3AM local time. Hmm. Well, that’s a rather interesting thing, because Windows Update is by default set to install updates at 3AM local time, and this Tuesday was Patch Tuesday. A quick examination of the Windows Update log (%SystemRoot%\WindowsUpdate.log) confirmed this fact:
2007-09-13 03:00:11:521 408 f4c AU The machine was woken up by Windows Update
Great. Well, according the event logs, it ran for another 1 hour and 45 minutes or so before the battery got sufficiently low for it to go into hibernate. So, apparently, Windows Update woke my laptop, on battery, in my laptop bag to install updates. It didn’t even bother to suspend it after the fact (gee, thanks), leaving the system to keep running until either it ran out of battery or something broke due to running in a confined place with zero ventilation. Fortunately, I lucked out and the former happened this time.
But that’s not all. This is actually not the first time this has happened to me. In fact, on August 29 (last month), I got woken up at 3AM because Windows Update decided to install some updates in the middle of the night. That time, it apparently needed to reboot after installing updates, and I got woken up by the boot sound (thanks for that, Windows Update!). At the time, I wrote it off as “intended” behavior, as the system happened to be plugged in to wall power overnight and a friend pointed out to me that Windows Update states that while plugged in, it will resume a computer to install updates and put it back into suspend afterwards.
Well, that’s fine and all (aside from the waking me up at 3AM part, which sucked, but I suppose it was documented to be that way). Actually, I’ll take that back, my computer waking me up in the middle of the night to automatically install updates is far from fine, but that pales in comparison to what happened the second time around. The powering the system on while it was on battery power to install updates, however, is a completely different story indeed.
This is, in my opinion, a spectacular failure of the “left hand not knowing what the right is doing” sort at Microsoft. One of the really great things about Windows Vista was that it was taking back power management from all the uncooperative programs out there. Except for, I suppose, Windows Update.
Consider that for a portable (laptop/notebook) computer, it is often the case that it’s downright dangerous to just wake the computer at unexpected times. For example, what if I was on an airplane during takeoff and Windows Update decided that, in its vast and amazing knowledge of what’s best for the world, it would just power on my laptop and enable Bluetooth, 802.11, etc. Or say my laptop was sitting in its laptop bag (somewhere I often leave it overnight to save myself the trouble of putting it there in the morning before I go to work), and it powers on to do intensive tasks like install updates and reboot with no ventilation to the system, and overheats (suffering irrreparable physical damage as a result). Oh, wait, that is what happpened… except that the laptop in question survived, this time around. I wonder if everyone else with a laptop running Windows Vista will be so lucky (or if I’ll be so lucky next time).
What if I was for some reason carrying my laptop in my laptop bag and, say, walking, going up a flight of stairs, running, whatnot, and Windows Update decided that it was so amazingly cool, that it would power on my computer without asking me and the hard drive crashed from being spun up while being moved under unsafe (for a hard drive) conditions?
In case whoever is responsible for this (amazingly negligent) piece of code in Windows Update every reads this, let me spell it out for you. Read my lips (or words): It is unacceptable to wake my computer up without asking me. U-N-A-C-C-E-P-T-A-B-L-E, in case that was not clear. I don’t care what the circumstances are. It’s kind of like Fight Club: Rule no. 1 is that you do not do that, ever, no matter what the circumstances are. And I really dare anyone at Microsoft to say that by accepting the default settings for Windows Update, I was consenting to it running my laptop for prolonged periods of time in a laptop bag. Yeah, I thought so…
You can absolutely bet that I’ll be filing a support incident about this while I do my best to have this code, how shall I say, permanently evicted from Windows Update, on behalf of all laptop owners. I can see how someone might have thought that it would be a cool idea to install updates even if you suspend your computer at night (which being the default in Vista would happen often). However, it just completely, jaw-droppingly drops the ball with portable computers (laptops), in the absolute worst way possible. That is, one that can result in physical damage to the end user’s computer and/or permanent data loss. This is just so obvious to me that I literally could not believe what had happened when I woke up, that something that ships by default and is turned on by default with Vista would do something so completely stupid, so irresponsible, so negligent.
I was pretty happy with the power management improvements in Windows Vista up until now. Windows Update just completely spoiled the party.
The worst thing is that if the consequences of resuming laptops without asking weren’t already so blindingly obvious, this topic (forcing a system resume programmatically) comes up on the Microsoft development newsgroups from time to time, and it’s always shot down immediately because of the danger (yes, danger) of unexpected programmatic resumes with laptop computers.
(Sorry if this posting comes off as a bit of a rant. However, I don’t think anyone could disagree that the possibility of Windows Update automatically powering on the laptop in the scenarios I listed above could possibly be redeemable.)