Nynaeve

The mechanisms used to activate a process server are fairly similar to those used to work with the -server and -remote remoting mechanism.

To start a process server, you must use a special utility called dbgsrv.exe that is distributed with DTW.  This program is the server end of debugger connection.

Like -server and -remote, process server / smart client debugging uses connection strings for both the client and the server.  The syntax for client and server connection strings are for the most part compatible, so you should look back on my existing posts about basic remote connectivity, reverse connections, and securing debugger connections.  For the most part, things work exactly the same as -server and -remote; all of the connectivity options supported by -server and -remote work for process servers and smart clients, and all of the features (such as reverse connections and secured connections) are available using the same mechanisms as well.

The main differences are how you pass the connection string to dbgsrv.  Where you would previously use “-server connection-string“, you will use “-t connection-string” with dbgsrv.  For instance, an example command line would be:

dbgsrv.exe -t tcp:port=port,password=password

To connect to this process server, you can use the “-premote” parameter with a debugger client (NTSD, CDB, and WinDbg).  This parameter functions in the same way as “-remote”, except that it is for connecting to process servers and not -server-style remote debugging servers.

debugger -premote tcp:port=port,server=server,password=password

Note that if you are using NTSD or CDB, you will need to specify a pid to connect to (e.g. -p pid) on the command line as well.  This is because a process server is not just fixed on one target process, but allows you to debug any process it has permissions to debug on the remote system.  With WinDbg, you can just open the process list (e.g. F6) like you would with a local system, and you will be presented with a list of running process to debug on the remote system.

After the connection is established, you can continue to debug as if you were running the debugger directly on the remote system.  Remember that with process servers, things like symbol access and extension dll calls are actually performed by the client debugger and not the remote system.  This means that if you set the symbol path, you are setting the symbol path for your debugger and not the process server (which has no concept of a symbol path).  As a result, your system will be the one to access symbol repositories and not the remote system.

The important thing to remember about process servers is that unlike the other remote debugging mechanisms that I have discussed thus far, process servers provide you with an entire “view” of the remote system and not just a remote view of a single debugging session.  This is in effect widening the scope of the remote debugging session from just one target to any programs running on the target system.

For the last few entries, I have been discussing the -server and -remote debugging mechanism.  While this remoting mechanism is good for a number of scenarios, in some cases, you want the debugger client to do the “heavy lifting” (such as managing symbols).  This does require significantly more bandwidth and is latency sensitive, but can provide advantages in several cases, such as if you are debugging a problem at a customer site live and need to access symbols, but for security reasons you can’t grant the debugger running on the customer site direct access to your internal symbol store.

The solution is to use what is called a “process server” (dbgsrv.exe) and a “smart client” (“-premote” command line parameter for the DTW debuggers).  This remoting mechanism starts a small stub server process (dbgsrv.exe) on the target computer which can accept one or more connections from debugger clients.  Once debugger clients connect, they can see a list of processes (e.g. F6 in WinDbg) on the target computer and select a process to debug.

The process server mechanism is very useful in the scenario where you need to do symbol access on your client.  It is also useful to do cross-session debugging under Terminal Server in Windows 2000, as native cross-session debugging is not supported under Windows 2000.  For that to work, you must start dbgsrv.exe under the session you want to debug processes in, and then connect to it using a DTW debugger running in your session.

Process servers are, however, much more bandwidth and latency sensitive than -server/-remote.  As a result, I would not recommend using them on a bandwidth- or latency- constrained network link.  Keep this in mind when choosing whether to use them to debug a problem on a customer site.

Also, process servers are restricted to just live user mode processes, and cannot be used to debug dump files or perform kernel debugging (like the other remoting mechanisms allow).  Additionally, you cannot have multiple remote debugger clients working on the same process like you could with -server/-remote.

Although there are some downsides (bandwidth usage and latency-sensitivity), process servers give the richest remote debugging experience for user mode processes of all of the mechanisms I have debugged thus far (provided your network connection can support this method).  Besides the ability to do local symbol access, process servers let you run custom extension DLLs locally without having to ship them to a customer site (important if you have custom extensions that you don’t necessarily want to become public, but are nonetheless useful for troubleshooting customer problems).  When you are using a process server, the full capabilities of the debugger should be available to you as if you were sitting at the remote machine and not just remote controlling a debugger, including the ability to stop debugging one remote process and switch to another just like you would with a local process.

Like -server and -remote, process servers operate through the use of a server connection string and a client connection string.  For the most part, these connection strings follow a syntax very similar to -server and -remote.  In the next installment of this series, I’ll go into detail about what you need to do in order to setup a successfull process server / smart client remote debugging session.

Previously, I’ve discussed the basics of -server and -remote, and how to do reverse debugging connections.  This covers most of the interesting functionality for this remote debugging mechanism, with one big exception: securing your remote debugger connection when you are operating over an untrusted network (such as the Internet).

Used as-is, none of the remote debugging techniques I have discussed so far provide any real security, other than the inherent difficulty of hijacking a TCP connection.  Data is sent in plaintext and there is no authentication of potentially sensitive commands that could direct a debugger to do dangerous operations and even potentially take control of the target’s computer if a privileged process is being remotely debugged.

If you are using the NTSD over KD remoting technique, this is usually less of a concern, since you are usually operating over a serial or 1394 cable physically connecting the two computers.  For plain -server and -remote debugging, or remote.exe debugging, however, this problem is more serious as these methods are typically used over networks.

For remote.exe, there is not all that much that you can do to secure connections.  The -server and -remote mechanism has a happier tale to tell, however.  This remoting mechanism does in fact have built-in support for secure connections using both SSL over the TCP transport or SSL over the named pipe transport.  I’ll primarily cover SSL over TCP, but the general concepts apply to the SSL over named pipe (“SPIPE”) transport as well.  There is additional support for simple password authentication, for all transports, which I’ll discuss as well.  However, this only adds a very basic form of protection and usually buys you only minimally more security than an unpassworded connection.

To use password authenticated connections, simply append the “password=password” parameter to the connection string for both the client and the server.  For instance, “tcp:port=1234,server=127.0.0.1” becomes “tcp:port=1234,server=127.0.0.1,password=secret”.  That’s all there is to this mechanism; after using it, both ends must specify the same password or the connection will fail.  I should emphasize that this again does not really provide strong protection, and the SSL alternatives should usually be used instead.

The SSL options are slightly more complicated.  These require a certificate that supports server authentication and is known to both ends of the remote connection.  The certificate that you use needs to have the “server authentication” role enabled for it.  Additionally, note that the same certificate is needed by both parties; that is, both parties must have the private key for the certificate.  This makes using the SSL option unfortunately more cumbersome than it could be.  To get this to work, you will typicallly have to request a server authentication certificate from your domain CA, and then install it (with the private key) on both computers.  Then, you can use it with SSL or SPIPE remote debugging.

The general format of an SSL transport connection string is very similar to the TCP transport, with some additional options added.  For the client, it is in the format of “ssl:port=port,proto=ssl-protocol,server=server-ip,[certuser|machuser]=cert-name-or-thumbprint“.  For a server, use “ssl:port=port,proto=ssl-protocol,[certuser|machuser]=cert-name-or-thumbprint“.  Remember that the certificate must match for both parties.

The “proto” parameter specifies which dialect of SSL to use, and can be one of tls1, pct1, ssl2, or ssl3.  The protocol must be the same for both the client and the server.

The “certuser” parameter specifies either the name (e.g. “Subjectname”) or thumbprint (e.g. “12345689abcdef…”) of a certifcate in the user certificate store.  Alternative, you can use “machuser” instead of “certuser” to specify that the certificate is contained within the machine store (using “machuser” typically requires that the debugger be run with administrator or localsystem privileges).  To determine the subject name or thumbprint of a certificate, run mmc.exe, add the “Certificates” snapin for either your user or the computer user (if you are an administrator and want to use “machuser”), locate the certificate you want to use (typically under “Personal\Certificates”), open the certificate property sheet, and view the “Details” tab.  The common name (“CN = “) under “Subject” is the subject name you will use, and the hex string under “Thumbprint” is the thumbprint you will use (you only need one).  If you are using the thumbprint, then remove all spaces between the hex digits when providing the parameter to “certuser” or “machuser”.

Putting all of these together, once you get the certificate in place on both computers, activating an SSL remote debugging session is basically the same as with a TCP remote debugging session.  To start an SSL debugging server, you might do this:

debugger -server ssl:port=1234,proto=tls1,
machcert=1111111111111111111111111111111111111111

Likewise, to start the client, you would use something like this:

debugger -remote ssl:port=1234,server=127.0.0.1,proto=tls1,
machcert=1111111111111111111111111111111111111111

Afterwards, the debugging connection should operate as any other -server/-remote session would.  All of the usual other considerations for -server/-remote apply to SSL as well; for instance, you can use reverse debugging and you can use “.server” instead of “-server”.

Because of the difficulty in setting up the certificate for SSL debugging, I usually recommend using something else to secure the remote debugging session other than the built-in SSL support, such as a VPN.

Next time: Debugger process servers and smart clients.

Last time, I provided a basic overview of some of the options available for remote debugging using -server and -remote.  There are still a couple of interesting things to consider about this particular debugging facility which I did not mention last time, though.

One of the more important extra features of -server and -remote is the ability to do reverse connections when using the TCP transport.  You are probably already familiar with this concept if you have used VNC (or similar tools) to assist with resolving a problem at a customer site.  Reverse connections allow you to remotely debug a computer that is firewalled off, provided you have an open port for your computer.  To perform reverse connection debugging, there are a couple of changes that need to be made to how I previously talked about using -server and -remote.

First, you should start the debugger client before the debugger server.  This is because the debugger client will be the program that actually performs a listen call on a socket, not the debugger server.

To start a reverse connection debugger client, use the connection string “tcp:port=port,clicon=0.0.0.0″.  (The IP address you supply to “clicon” is apparently ignored, and the debugger always listens on the wildcard address).  For example, you could use:

debugger -remote tcp:port=1234,clicon=0.0.0.0

Additionally, if you are using WinDbg, you can use the Ctrl-R / Connect to Remote Debugger Session UI to accomplish this task if you are not already in a debugging session.

After starting the reverse connection client, it will appear to be frozen.  In the case of WinDbg, the UI will appear to be unresponsive; this is the normal and expected behavior and not indicative of a problem!

After the reverse connection debugger is started, then the next step is to start the debugger server on the target computer.  Use a connection string in the form of “tcp:port=port,clicon=client-ip-address“.  For example:

debugger -server tcp:port=1234,clicon=127.0.0.1

After you start the debugger server, it will attempt to connect out to client debugger.  If all goes well, then you should be able to interact with the remote target the same as if you were using conventional -server and -remote.

As with conventional -server and -remote, you can use the “.server” command instead of the “-server” command line parameter to start a debugger session on an already active debugger target.

Next time: Additional connection string options that you can use for securing your -server/-remote remote debugger connections on an untrusted network.

Moving onwards to the more modern remote debugging services available, the next option available for remote debugging with the DTW package is -server/-remote.   This remote debugging mechanism is one that you may find yourself using fairly frequently.  Its advantages are rich functionality and integration with the WinDbg GUI (as a remote client), and reasonably low bandwidth usage, though not quite as lightweight as the previous options.

This mechanism allows you to connect one (or more) debugger clients to a debugger server.  This can allow for a limited set of collaboration between multiple people debugging the same problem.  Unlike the other mechanism discussed thus far, -server/-remote utilizes a more advanced protocol that while leaving most of the “hard work” (including symbol management) to the debugger server, allows things like the various WinDbg debuggee status windows to work and receive useful information (i.e. not just command window text).  So, when using this protocol with WinDbg as the client, you can utilize the memory / disassembly / register windows (and soforth).  Note that you can use any of WinDbg/ntsd/cdb/kd as either a client or server with this protocol.  It can be used for any debugging type that is supported by these debuggers, including dump file debugging.

The protocol also provides for varying forms of security.  You can use simple plain text password authentication, or if you want true security, SSL over named pipes or TCP.  The protocol can operate over several underlying transports; tcp, com (serial port), 1394, or named pipes (note that this is the transport used to communicate between the debugger server and debugger client, not related to the medium that is used to connect to the target in kd).

When using -server/-remote, it is important to ensure that both the debug client and debug server are from the same DTW package; otherwise, unexpected results may occur (typically resulting in the connection failing silently). 

There are many different ways to activate this remote debugging mechanism; you can start a debugger with -server to create a debugger server, or with -remote to act as a debugger client.  Alternatively, you can use the .server command to create a debugger server out of an existing debugging session.  If you are using WinDbg, you can use File->Connect to Remote Session (Shortcut: Ctrl-R) to connect to a remote debugging server instead of using the -remote command line parameter.

To use this debugging mechanism, you will need to create a connection string to be used by the debugger server and then the debugger client.  The general format of a connection string is “transport:transport-parameters”.  For our examples, I will use tcp as the transport; this is likely to be the most common you’ll use in the real world.  You can look up the usage of the other transports in the documentation if you are curious.

A simple connection string to use if you want to create a tcp server that listens on port 12345 would be “tcp:port=12345”.  The client connection string corresponding to this would be “tcp:port=12345,server=ip_or_hostname_of_server”.  (Note that if you are using the DTW 6.6.3.5 package, then it is required that you specify port= before server= in the client connection string or the connection will fail, due to a parser bug in the debugger).  If you use the “.server tcp:port=12345” command, you’ll see something like this:

0:001> .server tcp:port=12345
Server started.  Client can connect with any of these command lines
0: -remote tcp:Port=12345,Server=COMPUTERNAME

Then, you can connect to it using WinDbg and specifying either “tcp:port=12345,server=localhost” (assuming you are running on the same computer) to either the Ctrl-R/Connect to Remote Session dialog or the -remote parameter:

Microsoft (R) Windows Debugger  Version 6.6.0007.5
Copyright (c) Microsoft Corporation. All rights reserved.

Server started.  Client can connect with any of these command lines
0: -remote tcp:Port=12345,Server=COMPUTERNAME
COMPUTERNAME\User (tcp 127.0.0.1:2025) connected at Mon Jul 24 01:33:20 2006

On the server, you’ll see a notice that someone has connected:

COMPUTERNAME\User (tcp 127.0.0.1:2025) connected at Mon Jul 24 01:33:20 2006

Henceforth, all output will be sent to all clients and the local server text output.  Additionally, clients will be able to use the debuggee status UI windows (e.g. disassembly) as well as viewing command output text.  After all this is setup, then you can continue to debug as if you were on the server computer (keep in mind, however, that symbol paths are relative to the server and not the client computer).  If you connect multiple clients to the same session, then each additional client will be updated as any debugger client changes the state of the debugging session.

That’s a quick overview of -server and -remote.  However, there’s a bit more to this particular remote debugger mechanism than what I have talked about so far.  I’ll elaborate on some other features and usage cases of -server and -remote in the next post in this series.

One of the cool things included with WinDbg is an ISAPI DLL that allows you to host what is called a “symbol proxy” on any Windows Server 2003 computer running IIS.

What is a symbol proxy, you might ask, and why would I want one?

A symbol proxy allows you to serve up multiple symbol paths using a single HTTP symbol server path.  Additionally, the symbol proxy will cache everything that it downloads locally, so that the next time you request the same symbol, it won’t need to hit the Internet (or a slow WAN link) to download the symbols again.  You can configure one symbol proxy to cover your internal symbols as well as the public Microsoft symbols available at the Microsoft symbol server.  This can be really handy if you are debugging in many different VMs and don’t want to wait for every VM to download a bunch of MS symbols over the Internet each time you use one – instead, they’ll be locally cached at your site and will come up much faster.  Additionally, you won’t have to set a multitude of symbol paths to talk to the MS symbol server and your internal one, as you can have the symbol proxy talk to both “behind the scenes”.

There is some documentation relating to setting up the symbol proxy included with the DTW distribution.  I would highly recommend taking a look and considering setting up a symbol proxy yourself if you do Windows debugging, as it can significantly reduce the hassle of finding the right symbols for both MS binaries and your binaries.

N.B. There is a known problem with the previously current version (6.6.3.5) of the symbol proxy DLL included with the DTW distribution that causes it to be unable to talk to HTTP-based symbol repositories if you do not configure a WinHttp HTTP proxy with proxycfg.  You might send a mail to the WinDbg feedback alias (windbgfb at microsoft.com) and ask for a version of symproxy.dll that has this bug fixed if you cannot use a WinHttp-compatible HTTP proxy).  [Note: I have not tested to see if this bug is fixed in the release that came out two days ago.  It should be fixed, however, from what I know.]

Besides remote.exe, the next remote debugging option available is controlling NTSD through the kernel debugger.  This technique has also fallen by the wayside in recent days like remote.exe, but there are still some circumstances under which it is useful.

This remote debugging technique is based upon controlling NTSD through the kernel debugger connection.  As a crude form of remote debugging, this option allows you to control NTSD on the target computer from a different computer over a serial/1394/USB debugger cable.  This is useful in situations where you are debugging a user mode process in conjunction with the kernel debugger, and want to hide many of the kernel-debugger-specific overhead that is typically associated with doing user mode debugging from the kernel debugger – for instance, having to deal with whether a particular piece of code or data is paged out or not.

Under the hood, this mechanism works by having NTSD use DbgPrint and DbgPrompt instead of console output and console input, respectively.  Additionally, all other dependencies upon CSRSS by NTSD for debugging are disabled, so that NTSD can be used for debugging CSRSS (the Win32 subsystem).  The kernel debugger displays DbgPrint outputs and allows input through DbgPrompt, which allows you to control the user mode debugger through the kernel debugger.  One consequence of this is that the entire system is going to be frozen while you are inputting commands to NTSD, which can have implications for debugging RPC or network related programs, as connections may time out or go away while you are providing input to NTSD, since the networking stack will be frozen and be unable to acknowledge packets.

To activate this mechanism, you can start ntsd with the “-d” parameter (in addition to the usual parameters to specify what you are debugging) which “sends all debugger output to kernel debugger via DbgPrint” according to the documentation.  You must have the kernel debugger active in order for this option to be effective.  After you initially continue execution from NTSD (if applicable, depending on if you use “-g” or not), then you will need to cause a breakpoint exception (or other exception) in the process being debugged by NTSD through the kernel debugger (or another program) in order to return control back to NTSD.  The “.sleep” command is also mildly useful here, as effectively a “delayed breakin” type command that allows you to instruct the NTSD instance to continue execution and break back in to the kernel debugger with a prompt after a certain period of time.  This is necessary because there is no way to directly transfer control back to NTSD after you are done doing something in the kernel debugger.

Like remote.exe, this mechanism simply redirects input/output, although this time through the kernel debugger connection and not a pipe.  The main reasons to use this mechanism over remote.exe (or the other options) are for debugging certain situations that make it difficult or impossible to use a conventional user mode debugger, for instance, debugging CSRSS.exe.  Since the user interface I/O is redirected only, things like symbol processing are performed on the NTSD instance and not the local kernel debugger user interface.

Although the kernel debugger connection currently only supports serial cables, 1394, and USB for debugging, you can extent the NTSD-over-KD remoting mechanism to be useful for computers in remote locations by remoting the KD instance controlling NTSD through a network aware mechanism such as remote.exe, -server, or kdsrv.exe.

For most purposes, though, I would recommend not using this mechanism.  The other remoting mechanisms provide greater flexibility and are easier to user with remote computers.

This mechanism is, however, a valuable technique for certain special situations where you are either debugging the lowest level parts of the user mode NT infrastructure, or where you need to coordinate between a kernel debugger and user mode debugger on the same machine.  In the latter case, the NTSD-over-KD technique is superior to a network aware remote debugger connection to a NTSD/CDB/WinDbg instance running on the computer being kernel debugged because the NTSD-over-KD connection will not time out while you are broken into the kernel debugger like a network connection would.

That’s all for this installment of the remote debugging series.  I’ll talk about some of the more modern remoting mechanisms that you are likely to use in every-day debugging next time.

Debugging Tools for Windows 6.6.7.5 was released yesterday.

(Note that the changelog includes things from 6.6.3.5, which makes it kind of a pain to see what really changed…).

Download links:

• 32-bit – x86
• 64-bit – x64, IA64

Note that for many remote debugging scenarios, the debugger packages on both computers must be the same version or the remote debugging process will silently fail, so take care in planning when to upgrade (and to upgrade everything at once) if you use remote debugging.

One of the oldest mechanisms for remote Windows debugging is the venerable remote.exe, which ships with the DTW package.  Remote.exe essentially just pipes console input/output over a the network, which means that you can only use it for the console debuggers (cdb, kd).  This mechanism has the server end of the remote debugging session do all of the “hard work”, with the client acting as just a dumb user interface.

To start a remote server, use the following:

remote.exe /s "command-line" instance-name

where “command-line” is the debugger command line (for instance, “cdb notepad.exe”), and instance-name is a unique identifier (you can pick anything) that allows the remote client to select what program to connect to.

To connect to a remote server, you can use this command:

remote.exe /c computer-name instance-name

where computer-name is the name of the computer running the remote server, and instance-name matches the value you passed to remote.exe /S.  To quit the remote client, you can send the special string “@K” (and hit enter) on its own line, which will terminate the remote session (but will leave the remote server running).  Likewise, you can enter the “@K” command on the server end of the console to quit the server and terminate the redirected process.

That’s all there is to it.  You can actually use remote.exe to control console programs other than cdb/kd remotely, as it doesn’t really have any special intelligence about interacting with the program you are running under remote.exe.

The remote.exe method is the most lightweight of all of the remote debugging options available to you, but it’s also the most limited (and it doesn’t have any concept of security, either).  You might find it useful in scenarios where you are very limited on bandwidth, but for most cases you are much better off using one of the other remote debugging mechanisms.

Update: Pavel Lebedinsky points out that you can set a security descriptor on the remote.exe pipe via “/u” and “/ud” (though this will require that you have either local accounts with the same password as a remote user, or a trust relationship [e.g. domain] with the remote computer).  This does allow for some form of access control, though it is generally convenient only if both computers are on the same domain from what I can tell.

One of the most powerful features of the DTW debuggers (including WinDbg) is the ability to do debugging remotely.  Besides the obvious capability to debug someone else’s computer from your computer, the remote debugging support built in to the DTW debuggers turns out to be useful for much more.

For instance, you can use some of the remote debugging options to show someone else what you are working on with the debugger if you get stuck, or you can use it to debug programs that you would otherwise be unavailable to in Terminal Server under Windows 2000.

The range of remote debugging options available to spans from “dumb terminal” type solutions where text input/output is simply redirected over the network to having the local debugger do the real work using an RPC-like protocol over the network.  Remote debugging is available for both user mode and kernel mode debugging.

Over the next couple days I’m going to do a quick run through of each of the major remote debugging facilities included with the DTW debuggers, what their major benefits are, and how to pick which remote debugging option for a debugging session (as the various methods are useful under different conditions).  Although there is some (rather sparse) documentation in the WinDbg help file covering some of the remote debugging topics, there are some limitations (and usage considerations) that the help file does not articulate which are very important for deciding what you are going to do.  I’ll try to address these in this post series.