Catching Memory Scanners With Trap Pages

I have recently learned about a clever technique that can be used to detect memory scanners operating on a Windows process. The technique relies upon the way that Windows allocates virtual memory to provide a method of notifying the process that its memory has been accessed by external software.

Why would we want to do detect memory scanning?

When reverse engineering a complex piece of software, it can be difficult to quickly find useful information. Static analysis can be painful in cases where only a particular piece of logic is of interest, such as locating the small piece of code which calculates the player's health in a game. Dynamic analysis through debugging is also generally painful here for similar reasons.

Fortunately for reverse engineers, there exist memory scanning tools such as Cheat Engine. One function of these tools is to scan process memory for particular values of interest. For example, if we are interested in the code which modifies the player's health, and we know that the on-screen health indicator is currently showing the value 100, we can tell the scanner to look for the integer value 100. The scanner will locate all memory addresses which have an integer value 100 stored there. This process can be repeated (with changing health values) until the exact memory address is found, which usually only takes a few minutes to do.

Once an address is found, the reverse engineer will now know where to look. If the address is static, the reverse engineer can simply open up their favourite static analysis tool such as Ghidra and jump directly to that address. From there, they can see which pieces of code reference this address and locate the relevant logic very efficiently. If the address is dynamic, a debugger could be attached to watch this memory address and see what code modifies it. This information can then be used to create cheating software which can be distributed to other players.

In summary, memory scanning is a technique that speeds up the reverse engineering process significantly by pinpointing memory addresses of interest. These addresses can be used for focused analysis on the relevant code, so it would be nice to stop this.

How does it work?

Let's look at this call to VirtualAlloc:

VirtualAlloc(NULL, 1, MEM_RESERVE | MEM_COMMIT, PAGE_EXECUTE_READWRITE);

This call will attempt to allocate a page of (virtual) memory. Assuming this succeeds, we will receive the address of our newly-allocated memory. We can then trust that this memory exists and begin to use it, trusting that it is zeroed out ready for use - Pretty simple.

However, the documentation for MEM_COMMIT states that “Actual physical pages are not allocated unless/until the virtual addresses are actually accessed.". This means that until we access this allocated memory in some way, there is no physical backing for the memory; we just have a promise that the memory will by physically-backed by the time it is accessed.

If we allocate this memory and simply don't access it, the memory should never receive a physical backing… But what if it does? Well, this would indicate that the memory has been accessed by something other than our own code.

We can check this allocated page by using QueryWorkingSetEx like so:

PSAPI_WORKING_SET_EX_INFORMATION info;
info.VirtualAddress = trapPageAddress;
if (QueryWorkingSetEx(hProcess, &info, sizeof(PSAPI_WORKING_SET_EX_INFORMATION)))
{
	if(info.VirtualAttributes.Valid)
    {
        // possible memory scanning detected
    }
}

Example Implementation

You can find a full example program which implements this here. Try compiling it, then running the standalone program and attempting some memory scans with a tool like Cheat Engine.