Nynaeve

Current VMKD release version: 1.1.1.7

VMKD (source available under the projects page) is a program that provides high speed kernel debugging support for VMware virtual machines. VMKD allows you to debug a VMware VM in a high speed fashion, instead of using the much slower and lower bandwidth virtual serial port mechanism.

This page documents the general usage and installation of VMKD. There is, however, a post series with in-depth coverage of how VMKD works “under the hood” if you are interested in the implementation details of VMKD.

When you use VMKD to debug a VM, VMKD creates a named pipe on the machine hosting the VM that you can connect to using the usual kernel debugging over named pipe support in WinDbg. However, unlike conventional VM kernel debugging, which is done by creating a virtual serial port in the VM and exposing it to the host system as a named pipe, VMKD does not internally use a virtual serial port to communicate with the kernel running in the VM. Instead, VMKD uses a high speed interface that takes advantage of the fact that the kernel is running in a VM to enhance the performance and responsiveness of the kernel debugging experience.

VMKD has presently only been tested against VMware Server 1.0.3 and 1.0.4. It is designed in a fashion that is intended to be portable to future VMware versions, however this forwards compatibility is fairly fragile and may break on future releases. VMKD does not support Microsoft Virtual Server or other virtualization products. Do not attempt to use VMKD with other virtualization programs or with a physical machine.

The main benefits of using VMKD instead of conventional serial port debugging are:

1. Responsiveness. VMKD provides a very low latency link between the kernel debugger and the VM if you are running the kernel debugging on the same computer hosting the VM. This means that most kernel debugger commands will respond much quicker than with normal kernel debugging (many commands are typically fairly close to local kernel debugging (lkd) speed, such as !process 0 0, which typically returns in 1-2 seconds or less even with 40-50 running processes when operating with VMKD). This improved response time even makes complex conditional breakpoints on “warm” kernel code paths feasible!
2. Data transfer speed. VMKD can move data to and from a VM much faster than the virtual serial port debugging mechanism. For example, I typically received around ~200KBps throughput while doing bulk memory reads on a VM, far beyond that possible with a virtual serial port. Most of the overhead now remaining in terms of bulk data transfer is reflective of design limitations of the protocol that VMKD uses to talk to the kernel debugger client (DbgEng.dll). Note that 1394 can still write physical memory dumps faster than VMKD, because 1394 KD can essentially DMA the target’s physical memory across the wire due to special support in the 1394 DbgEng KD protocol client. However, in most other aspects, VMKD provides equivalent or superior performance to even 1394 KD.
3. Processor usage. Normally, when you are kernel debugging a computer, the target is spinning on the kernel debugger I/O hardware (such as the serial port or 1394 controller). With a VM, this is a particular problematic condition, as it causes the VM to monopolize one CPU with useless polling. VMKD allows the VM to sleep while waiting for input from the kernel debugger, eliminating the tendancy of conventional virtual serial port debugging to severely degrade overall system performance on the host computer.

However, VMKD is not perfect. Because it was written without the assistance of either VMware or Microsoft, integration with the Windows kernel and VMware is a bit rough around the edges. Due to this, there are some steps that need to be followed to use VMKD. For some kernel debugging tasks, it may simply be easier to just use virtual serial port debugging and live with the limitations of the virtual serial port than to set up a VMKD debugging session.

Additionally, VMKD is experimental software. Although I have attempted to test it on common configurations, it is possible that you may encounter bugs that may crash your KD target VM, or even the vmware-vmx.exe process that you are using to host the target VM.

In order to use VMKD, you will need to do the following:

1. Download VMKD. The package includes three distinct binaries, as well as documentation. The first of which is vmxpatch.dll, which is a DLL that is loaded into the vmware-vmx.exe process that is hosting the VM that you wish to debug with VMKD. The second binary is vmxinject.exe. vmxinject.exe is a simple tool that allows you to load vmxpatch.dll into a target process (intended to be used on a vmware-vmx.exe process). It takes a single command line argument, the process ID of the vmware-vmx.exe process to load vmxpatch.dll into.

   The final binary included is kdvmware.sys (x86 and x64 flavors), which is a driver that needs to be installed in the guest operating system.

2. Create a virtual serial port on the VM that you want to debug, if you have not already. This is required so that you can enable kernel debugging on the VM. You will then need to enable KD on the target VM (any parameters will work, so long as they are valid; I recommend setting the VM up like you would for a normal virtual serial port debugging session).
3. Copy kdvmware.sys into the target VM and install it. I would recommend using the following command line to create the driver service for it:
   

   sc create kdvmware type= kernel start= demand binPath= c:\windows\system32\drivers\kdvmware.sys DisplayName= kdvmware

   (c:\windows\system32\drivers\kdvmware.sys corresponds to the location where you have placed kdvmware.sys on the target VM). Do not start the driver service yet. I recommend leaving the driver service as manual start, as it will attempt to establish communications with vmxpatch.dll host-side and assume control of the kernel debugging interface as soon as the driver is started.

4. Copy vmxpatch.dll and vmxinject.exe into a location on the computer hosting the VMware VM. Both files should be placed in the same directory. When you start the VM that you want to debug, find the PID corresponding to the vmware-vmx.exe instance hosting the VM. Then, open a command prompt window, change to the directory in which you placed vmxpatch.dll and vmxinject.exe, and issue the command: vmxinject <vmware-vmx-process-id>. This will load vmxpatch.dll into the vmware-vmx.exe instance.

   If you are not certain what vmware-vmx.exe instance corresponds to the VM that you want to kernel debug, then I recommend using a tool such as Process Explorer or WinDbg to examine either the command line or current directory of each vmware-vmx.exe process until you find one that references the directory containing the VM you want to debug.

5. Load vmxpatch.dll into the appropriate vmware-vmx.exe instance using vmxinject.exe as described above, if you have not already.
6. Start the kdvmware driver service on the target VM. It will attempt to connect to vmxpatch.dll, which should have been loaded into the vmware-vmx.exe instance that is hosting the VM. If you did not load vmxpatch.dll into the correct vmware-vmx.exe instance, the VM will have several “The operating system was not found.” dialog boxes in the VMware console, and the kdvmware.sys driver will refuse to load. These dialog boxes are harmless and can be dismissed (internally, VMKD repurposes the “operating system not found” VM monitor command to communicate with vmxpatch.dll). If you do encounter any such dialog boxes, then you will need to load vmxpatch.dll into vmware-vmx.exe and then start the kdvmware driver service in the target VM once more.

   If the driver successfully loads (and you do not get an error from trying to start the driver, such as by “net start kdvmware”), then it has successfully established communications with vmxpatch.dll on the host.

7. Connect the kernel debugger to VMKD. VMKD creates a named pipe in the name “\\.\pipe\kdvmware_<virtual machine directory name>“, where virtual machine directory name corresponds to the last directory component in the path to the VM’s configuration file. For example, if your VM is located in “C:\VMs\myvm”, the pipe created will be of the name “\\.\pipe\kdvmware_myvm”. (Note that the pipe is named based off of the directory in which the VM is located and not the name of the VM. This approach was chosen to minimize the number of internal vmware-vmx.exe functions that would need to be called, which in turn improves compatibility with future VMware versions.)

   You should be able to use the built in debugger named pipe support to communicate with the VM using the VMKD named pipe, just the same as you would with a virtual serial port named pipe. Make sure that you connect to the VMKD named pipe and not the virtual serial port named pipe, as the kernel in the VM will no longer be receiving data from the virtual serial port.

For subsequent uses of VMKD, you will just need to use vmxinject.exe to load vmxpatch.dll into the correct vmware-vmx.exe instance and then from there, start the kdvmware driver service once you are ready to cut kernel debugging for the VM over to VMKD.

Note that on x64 systems, it may be necessary to disable driver signature enforcement to load the kdvmware.sys driver. Additionally, on x64 systems, it will be necessary to disable PatchGuard to use kdvmware.sys (the kdvmware driver rewrites the active kernel debugger protocol module, which is unfortunately protected by PatchGuard). This can be accomplished by attaching a debugger to the virtual serial port at boot time so that PatchGuard is not enabled, and then after boot cutting the system over to VMKD. Due to these limitations, it is typically more convenient to use VMKD on 32-bit VMs.

The vmxpatch.dll module that is loaded into the vmware-vmx.exe process has various debug prints enabled in this release, which will be activated if a debugger is active in the vmware-vmx.exe process. If you experience difficulties while trying to connect to the VMKD named pipe and synchronize with the target kernel, try attaching a debugger to the vmware-vmx.exe process and examining the debug output. Additionally, you can enable DbgEng KD protocol debugging with the Ctrl-D key combination (Ctrl-Alt-D for WinDbg). Keep in mind that watching vmxpatch.dll debug output with a debugger attached to vmware-vmx.exe will significantly lower both the throughput and responsiveness of the VMKD connection.