Nynaeve

This question came up on the microsoft.public.windbg newsgroup and turned out to be more complicated to solve than it might appear, so I’m posting about it here.

Someone asked if there is a way to use the DTW tools to detect if a binary is present on a symbol repository.  Now, you might naively assume that you can just use symchk.exe on the binary and it will work, and indeed if you try this, you might be fooled into thinking that it does work.  However, it really doesn’t – if you do this, all that symchk will do is verify that the symbols for the binary are on the symbol repository.

If you need to make sure that the binary is there (i.e. so you can do dump file debugging using the symbol repository), then you need to use the “/id filename” command line parameter with symchk.  This tells symchk that you want to verify that symbols for a dump file exist on the symbol server.  Because DbgEng lets you load a single PE image (.dll/.exe/.sys/etc) as a dump file, and because dump files require loading the actual binary itself from the symbol path, this forces symchk to verify that the binary (and not just the symbols) exist on the symbol repository.

If you are working with a process that has many of your DLLs loaded in it, and your DLLs tend to be loaded and unloaded dynamically, then you can sometimes save yourself a lot of trouble when debugging a problem by making sure that each of your DLLs has a unique base address.

That way, if you have a bug where one of your DLLs is called after unloaded, you can easily figure out which DLL the call was supposed to go to and which function it should have gone to by loading the DLL using the dump file loading support (start WinDbg as if you were going to debug a dump file, but select the .DLL in question instead of a .dmp file) and unassembling the address that was referenced.

On Windows Server 2003 and later, NTDLL maintains a list of the last few unloaded modules and their base addresses in user mode (accessible using “lm” in WinDbg) which can make debugging this kind of problem a bit more manageable even if you don’t rebase your DLLs, but rebasing is easy, improves loading performance (and especially scalability under Terminal Server), so I would highly recommend going the rebasing route anyway.

If you weren’t on Windows Server 2003 and didn’t rebase your DLL, then chances are the loader relocated it at load time to some not-predictable location, which makes finding the actual DLL being called and which function / global / etc in the DLL was being referenced when the crash occured much more difficult than if the DLL always loads at its preferred base address and you can simply look up the address in the DLL directly.

If you have a symbol proxy setup and you need to have it talk to a symbol server path that references a UNC share on a server that isn’t on the same domain as the IIS webserver hosting symproxy, then you may need to do some hackery to get the symbol proxy to work properly without prompting for credentials to use on the network.

 Specifically, the problem here is that there is no way to tell IIS to try and map a UNC share with a particular username and/or password before processing a request unless the request itself points to a network share.

One way to work around this is to use a simple ISAPI filter that I wrote (UncAccessFilter) to make sure any required UNC paths are mapped before the symproxy ISAPI filter is called.  After installing the ISAPI filter in the usual way, make sure that it is prioritized above the symproxy filter.  To configure it, you will need to manually set up some values in the registry.

 Create the key “HKEY_LOCAL_MACHINE\Software\Valhalla’s Legends\Skywing\UncAccessFilter” and ensure that the user web requests will be running in has read access to it.  You will probably want to ensure that only the web access user, administrators, and the system account have read access to this key because it will have passwords stored in it (be aware of this as a potential security risk if someone gets access to the registry key, as the passwords are not obfuscated in any way).  Then, for each share, create a REG_SZ value whose name is the share path you want to map (e.g. \\fileserver\fileshare) and whose contents are of the format “username;password”, for instance, “fileserver\symbolproxyuser;mypassword”.

To debug the filter, you can create a REG_DWORD value in that key named “DebugEnabled” and set it to 1, in which case the IIS worker process under which the ISAPI filter is running in will do some diagnostic OutputDebugString calls about what operations it is performing if you have a debugger attached to the process.  Assuming you configured the filter properly, on startup, you should see a series of messages listing the configured UNC shares (you may need to attach to the svchost process that creates the w3wp worker processes and use `.childdbg 1′ to catch this message for the new worker processes on startup).

If you are using the prebuilt binaries then make sure to install the VC++ 8 runtimes on the IIS server first.  Note that the prebuilt binaries are 32-bit only at this time, you’ll need to rebuild the ISAPI filter from source if you want to use the filter in 64-bit mode.

Be aware that the ISAPI filter is fairly simple and is not extraordinarily robust (and may be a bit slow if you have high traffic volumes, since it enumerates mapped network shares on every incoming request).  Additionally, be aware that if one of the servers referenced in the registry is down, it can make web requests that you have configured to be filtered by UncAccessFilter take a long time as the filter tries unsuccessfully to reconnect to the configured share on that server.  However, when properly configured, it should get the job done well enough in most circumstances.

Note that if you can get away with using the same account for all of your shares, a better solution is to simply change the account the web application associated with the symbol proxy is running under.  If you need to use multiple accounts, however, this doesn’t really do what you need.

Update: It would help if I had posted the download url.

Over the past week or two, I’ve written about some of the various remote debugging options available to you through the Debugging Tools for Windows (DTW) package.  I’ve covered most of the major debugging mechanisms available at this point and given brief descriptions of their strengths and weaknesses.

Here’s an indexed listing of the posts on this topic so far:

1. Overview of WinDbg remote debugging
2. Remote debugging with remote.exe
3. Remote debugging with KD and NTSD
4. Remote debugging with -server and -remote
5. Reverse debugging -server and -remote
6. Securing -server and -remote remote debugging sessions
7. Remote debugging with process servers (dbgsrv)
8. Activating process servers and connecting to them
9. Remote debugging with kdsrv.exe
10. Remote debugging review

At this point, you should be able to use all of the above remoting mechanisms in their basic usage cases.  There are a couple of obscure features that I did not cover, such as doing -server/-remote over serial ports, but between my posts and the documentation you should be able to figure out what to do if you ever find a use for such estorica (let me know if you do!).  What remains to be told is some general advice on which remoting mechanism is the best for a particular problem.

In general, the most important factors in choosing a remoting mechanism are:

• Available bandwidth and latency between your computer and the remote system.  Some remoting mechanisms, like dbgsrv, perform very poorly without a high bandwidth, low latency link.
• Whether symbol access needs to be done on the client or the debugging target.  This consideration is important if you are debugging a problem on a customer site.
• What types of targets you need to support.  Some mechanisms, such as process servers, do not support all target types (for instance, lack of dump file debugging support).
• Whether you need special support for working through a firewall (i.e. reverse connection support).
• Ease of use with respect to setting up the remoting session.

These are the general factors I use to decide which remoting mechanism to use.  For example, in ideal cases, such as debugging a problem on a LAN or on a virtual machine hosted on the same computer, I will almost always use a process server for remote debugging, simply because it lets me keep my own WinDbg workspace settings and symbol access without having to set up anything on the target computer.  Over the Internet, process servers are usually too slow, so I am often forced to fall back to -server/-remote style remoting.

Taking into account the guidelines I mentioned above, here are the major scenarios that I find useful for each particular remoting mechanism:

• Process servers and smart clients (dbgsrv).  This is the remote debugging mechanism of choice for remotely debugging things on virtual machines, on a LAN or other fast connection, or even on the same computer (which can come in handy for certain Wow64 debugging scenarios, or cross-session debugging under Terminal Server prior to Windows XP).  Process server debugging is also useful for debugging early-start system services remotely, where the intrastructure to do symbol access (which touches many system components, for things like authentication support) is not yet available – for this scenario, you can use the “-cs command-line” parameter with dbgsrv to start a target process suspended when you launch dbgsrv, which is handy for using Image File Execution Options to have dbgsrv act as a debugger for early start services.  This can be more reliable than -server and -remote if you are trying to do symbol access, as if you are debugging certain services, you might deadlock the debugger and lose the debugging session if the debugger has to talk to the service you are debugging in order to complete a network symbol access request.
• -server and -remote.  If I am doing debugging over the Internet, I’ll usually use this mechanism as it’s relatively quick even over lower quality connections.  This mechanism is also useful for collaborating with a debugging session (for instance, if you want to show someone how to perform a particular debugging task), as you can have multiple users connect to the same debugging session.  Additionally, -server/-remote are handy if you have a large dump file on a customer site and you want to debug it remotely instead of copying it to your computer, but would like to do so from the context of your local computer so that you have easier access to source code and/or documentation.  Finally, -server/-remote support remote kernel debugging where process servers do not.
• KdSrv.exe.  If you need to do remote kernel debugging over a LAN, this is the mechanism of choice.  Be aware that kernel debugging is even more latency and bandwidth sensitive than process servers, making this mechanism useless unless you have a very fast, LAN-like connection to the target.  If these conditions hold true, KdSrv.exe provides the main benefits that a process server does for user mode debugging; local symbol access, local debugger extensions, and allowing the debugger to use your workspace settings on the local computer as opposed to setting up your UI on a remote system.
• NTSD through KD.  This is useful in a couple of specialized scenarios, such as debugging very early start processes or performing user mode debugging in conjunction with the kernel debugger.  While controlling NTSD through KD is much less convenient than through a conventional remote debugging session, you won’t have to worry about your session going away or getting disconnected while the remote system is frozen in the kernel debugger.  In particular, this is useful for doing things like debugging things that make calls to the LSA from kernel mode, or other situations where kernel mode code you are debugging is extensively interacting with user mode code.
• Remote.exe.  I have never really found a situation that justifies the use of this as a preferred remoting mechanism, as its capabilities are far eclipsed by the other remoting services available and the benefits (low network utilization) are relatively minimal compared to -server/-remote in today’s world of cable modem and xDSL connections.

If you are debugging a problem on a customer site, you will likely find reverse connection debugging highly useful.  All of the modern remote debugging mechanisms support reverse connections except NTSD over KD, for obvious reasons.

Another consideration to take into account when selecting which mechanism to use is that you can mix and match multiple remoting mechanisms within a debugging session if it makes sense to do so.  For instance, you can start a process server, connect to it with ntsd, and launch a -server/-remote style server with “.server” that you then connect to with WinDbg.  This capability is usually not terribly useful, but there are a couple of instances where it can come in handy.

That’s all for this series on remote debugging.  I may come back and revisit this topic again in the future, but for the moment, I’ll be focusing on some different subjects for upcoming posts.

Most of the debugging mechanisms I have gone through so far will also support kernel debugging, though I have not focused on this fact.  You can use remote.exe for controlling KD remotely, and -server/-remote for controlling one KD through another KD or WinDbg.  Both of these mechanisms can be used to control a kernel debugger remotely (keep in mind that you still need a separate computer to run kd.exe on from the target computer of course), however, they do not allow the same flexibility as dbgsrv.exe does.  This means no client-side symbol access, and no client-side debugger extensions.

However, there is a way to get this functionality with the kernel debugger as you would with the user mode debuggers when using dbgsrv.exe.  Enter kdsrv.exe, the kernel debugger server.  Kdsrv.exe is an analogue of dbgsrv.exe and fullfills the same basic functional requirements; it allows multiple debugger clients to connect to it and begin kernel debugging sessions on resources that are connected to the computer running kdsrv.exe.  Like dbgsrv.exe, kdsrv.exe is used with one debugger client per debugging session, and also like dbgsrv.exe, kdsrv.exe does not start any debugging sessions on its own and leaves that up to clients that connect remotely.  It also allows for secured connections and reverse connections, just like dbgsrv.exe (using the same connection string values).

Kdsrv.exe allows the same rich experience as dbgsrv.exe when it comes to doing remote kernel debugging.  It allows you do perform symbol access and debugger extension calls on the local debugger client and not from a kd.exe instance running on the remote system.  It also has many of the same limitations of dbgsrv.exe, such as no support for remote dump file debugging.

To activate a kdsrv.exe server, use the same syntax that I described with with “Activating process servers and connecting to them“.  The command options are identical to dbgsrv.exe with respect to specifying a connection string and starting the server (some of the little-used other command line options to dbgsrv.exe that relate to starting a process along with the debugger server are not supported by kdsrv.exe).  For example, you could use:

kdsrv.exe -t tcp:port=port,password=secret

You’ll get an error message box if you give kdsrv.exe an unacceptable command line, otherwise, it will simply run in the background.

Connecting to a kdsrv.exe instance uses a slightly more complex connection string syntax, which is an adaptation of the one used by smart clients with -premote.

The client connection string is given in the format:

kdsrv:server=@{tcp:port=port,server=server-ip,password=password},trans=@{kd-string}

(Password is optional.)

The “kd-string” value is what you would normally pass to kd.exe or windbg.exe to start a kd session.  It specifies a remote resource to connect to that resides on the machine running kdsrv.exe.  For instance, you might use “com:port=comport,baudrate=baudrate” to direct kdsrv.exe to connect the kernel debugger over a com port connection using the specified baud rate.

To activate a kdsrv.exe client, use a command line formatted as follows;

debugger -k connection-string

, where “connection-string” is the “kdsrv:…” string discussed above.  Here are some examples of starting a server and connecting to it:

Starting the kdsrv instance:

kdsrv.exe -t tcp:port=1234,password=secret

Connecting kd:

kd.exe -k kdsrv:server=@{tcp:port=1234,server=127.0.0.1,
password=secret},trans=@{com:port=com1,baudrate=115200}

After that, you should be set to go.

You can use a variety of different underlying debugger targets with kdsrv.exe, including serial (com), 1394, and serial-over-named pipe (virtual machine) targets.