Nynaeve

If you are porting an program to x64, one of the first things that you might have to debug are 64-bit portability problems. The most common types of these problems are pointer truncation problems, where assumptions are made by your (previously 32-bit) program that a LONG/DWORD/other 32-bit integral type can completely contain a pointer. On x64, this is no longer the case, and if you happen to be given a pointer with more than 32 significant bits, you’ll probably crash.

Although the compiler has very good support for helping detect some of these problems, (the /Wp64 command line option, or “Detect 64-bit portability issues” in the VC++ GUI) sometimes it won’t catch all of them. Fortunately, using your debugging knowledge, you can help catch many of these problems very quickly.

There is some built-in support to do this already, in the way of a feature that forces the operating system to load DLLs top-down on 64-bit Windows. This means that instead of starting at the low end of the user mode address space and going upwards when looking for free address space, the memory manager will start at the high end and move downwards (when loading DLLs). In practical terms, this means that instead of usually getting base addresses that are entirely contained within 32 significant bits of address space, you will often get load addresses that are above the 4GB boundary, thus quickly exposing pointer truncation problems with global variable pointers or function pointers. You can enable this support with the gflags utility in the Debugging Tools for Windows package.

Unfortunately, as far as I could tell, there isn’t any corresponding functionality to randomize other memory allocations. This means that things like heap allocations or VirtualAlloc-style allocations will still often get back pointers that are below 4GB, which can result in pointer truncation bugs being masked when you are testing your program and only showing up in high load conditions, maybe even on a customer site. Not good!

However, we can work around this with a conditional breakpoint in the DTW debuggers. Conditional breakpoints are extremely useful, and what we’ll use one for here is to set a particular flag that causes allocations to be done in a top-down fashion to the lowest level memory allocation routine (that ultimately the Win32 heap manager and things built on top of it, such as new or malloc will call) that is accessible to user mode: NtAllocateVirtualMemory. This function is the system call interface to ask the memory manager to allocate a block of address space (and possibly commit it). It is what VirtualAlloc is implemented against, and what the heap manager is implemented against, so by passing the appropriate flag here, we can guarantee that almost all user mode allocations will be top down.

How do we do this? Well, it’s actually pretty simple. Create a process under the debugger and then enter the following command:

bp ntdll!NtAllocateVirtualMemory "eq @rsp+28 qwo(@rsp+28)|100000;g"

This command sets a breakpoint on NtAllocateVirtualMemory that sets the 0x100000 flag in the fifth parameter (recall my previous discussion on x64 calling conventions). After altering that parameter, execution is resumed and the program continues to run normally.

If we look at the prototype for NtAllocateVirtualMemory:

// NtAllocateVirtualMemory allocates
// virtual memory in the user mode
// address range.
NTSYSAPI
NTSTATUS
NTAPI
NtAllocateVirtualMemory(
IN HANDLE ProcessHandle,
IN OUT PVOID *BaseAddress,
IN ULONG ZeroBits,
IN OUT PULONG AllocationSize,
IN ULONG AllocationType,
IN ULONG Protect
);
 

 

… we can see that we are modifying the “AllocationType” parameter. Compare this to the documentation of the VirtualAlloc function, and you’ll see what is going on here (the flAllocationType parameter is passed as AllocationType). The flag we passed is MEM_TOP_DOWN, which, according to MSDN, “allocates memory at the highest possible address”.

After performing this modification, most allocations will have more than 32 significant bits, which will help catch pointer truncation bugs that deal with dynamic memory allocations very quickly.

Earlier, I said that this will only affect most allocations. There are a couple of caveats for this tecnique:

• It does not modify data section view mappings (file mappings). I leave it as an exercise for the reader to make a similar conditional breakpoint for ntdll!NtMapViewOfSection.
• It does not catch the first heap segment in the first heap (the process heap) normally, unless you go out of your way to apply the breakpoint before the process heap is created. One workaround is to just add some dummy allocations at the start of the program to consume the first heap segment, such that subsequent allocations are forced to go through a new heap segment which will be allocated in the high end of the address space.

Despite these limitations, however, I think you’ll find this to be an effective tool to help catch pointer truncation bugs quickly.

For my next few posts, I’m going to take a break from x64 debugging topics and focus on a different topic for a bit. Stay tuned!

Update: Pavel Lebedinsky commented that you can set HKLM\System\CurrentControlSet\Control\Session Manager\Memory Management\AllocationPreference (REG_DWORD) to 0x100000 to achieve a similar effect as the steps I posted, without some of the caveats of the conditional breakpoint I described above (in particular, the initial heap segments will reside in the high end of the address space).  This is a more elegant solution than the one I proposed, so I would recommend using it instead.  Note that ths alters the allocation granularity on a system-wide basis instead of a process-wide basis.

This entry was posted on Monday, July 17th, 2006 at 7:00 am and is filed under Debugging, Windows. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.