If you have done debugging work on Win32 (x86) for any length of time, you probably know that there are many different calling conventions. While I have already covered the Win64 (X64) calling convention, I haven’t yet gone into details about how to work with the various calling conventions present on x86 Windows. This miniseries is going to go into some detail relating to how the calling conventions work from the perspective of debugging and reverse engineering.
There are three major calling conventions in use in modern Win32 (on x86, anyway): __stdcall, __cdecl, and __fastcall. All three have different characteristics that are important if you are debugging or reverse engineering something, and it is important to be able to recognize and work with functions written against all three calling conventions – even if you don’t have access to symbols. To start off, it’s probably best to get a feel for what all of the different calling conventions do, how they work, and soforth.
Most of the calling conventions have some things in common. All of the calling conventions have the same set of volatile registers, which are not required to remain the same across a call site: eax, ecx, and edx. Additionally, all three calling conventions use eax for 32-bit return values, and eax:edx for 64-bit return values (high 32 bits in edx). For large return values (e.g. functions that return a structure, and not a pointer to a structure), a hidden argument is passed to a hidden local variable in the caller’s stack frame that represents the large return value, and the return value is filled in using the hidden argument (that is, large return values are actually implemented as a hidden pointer parameter).
- __cdecl is the default calling convention for the Microsoft C/C++ compiler. This is an entirely stack based calling convention for parameter passing; the caller cleans the arguments off of the stack when the call returns.
- __stdcall has the same semantics as __cdecl, except that the callee (target function) cleans the arguments off of the stack instead of the caller.
- __fastcall passes the first two register-sized arguments in ecx and edx, with the remaining arguments passed on the stack like __stdcall. If any stack based arguments were present, the callee cleans them off of the stack.
Additionally, there is a fourth calling convention implemented by CL, known as __thiscall, which is like __stdcall except that there is a hidden pointer argument representing a class object (“this” in C++) that is passed in ecx. As you might imagine, __thiscall is used for non-static class member function calls (by default). If you explicitly override the calling convention of member functions, then “this” becomes a hidden (first) argument that is passed according to the conventions of the overriding calling convention. In general, __thiscall is not really used in the Windows API.
That’s a general overview of the basic concepts behind all of the major Win32 x86 calling convention. In some upcoming posts, I’ll explore the implications behind the different attributes of these calling conventions, their usage cases, and how they apply to you when you are debugging or reverse engineering something.