How to debug self-modifying code in Windows?

[xuancong]

I have debuggers like Visual C++ 2008, Visual C++ 6, IDA Pro 5.2, etc.

However, when I debug self-modifying exe files instruction-by-instruction, the results I obtained is different from when I press run&continue. Apparantly, the debugger is not aware of any code change caused by the program itself. So does anyone knows any tool which can debug self-modifying exe files properly?

How about in Linux?

[Tosi]

As far as I know, there is no way to debug self-modifying code.
If you want to be able to debug, don't write self-modifying code.
If you want self-modifying code, then don't expect to be able to debug.

[Solar]

If you want to be able to debug, don't write self-modifying code.

I.e., don't write self-modifying code.

[davidv1992]

Then what would you call what V8 does, cause that sure looks like self modifying code to me and that is one of the fastest javascript implementations available today.

[Tosi]

It's dynamic code generation. It is basically a just-in-time Javascript compiler. The generated machine code, I assume, does not modify itself during executed, which makes it not self-modifying code. This is typically the fastest way to run an interpreted language, in the same way that dynamic recompilation speeds up an emulator more than simple interpretation of opcodes does. Since the generated code does not modify itself, it avoids many of the problems of self-modifying code, such as cache problems.

[qw]

Is self-modifying code even possible on Windows? Didn't know that.

[Tosi]

I know it's possible with 32-bit Windows, but I don't know about 64-bit.

[a5498828]

windows use flat address space. Linear addresses are accessible using equal offsets using data or code selectors. Why you brought segment issue here? Im missing something about self mod~ code? Its just code that write where it execute, so instructions change dynamicly.

I would use trap flag to debug it. x86 allows it, windows - i dont think so. trap flag is very nice for debugging because your handler is executed after each instruction. things like popf you can emulate to hide trap flag state from program. And debugging ring3-1 code there is no way to escape this (is things like rdtsc are locked).

[Tosi]

Segmentation doesn't magically go away with protected mode. With 32-bit segmentation you can mark blocks of memory as "non-writeable" or "executable." PE files have separate sections such as .text/.code (executable, read-only), and data (read/write). I don't know which versions, if any, of Windows make use of the GDT or LDT in such a manner.

[a5498828]

windows use flat address space.

With 32-bit segmentation you can mark blocks of memory as "non-writeable" or "executable."

Rather readable (code) or writeable (data). Only 32 bit segmentation? I bet 16 bit one also works the same (with 24bit base and no granuality).


The only nonflat thing about windows is TIB. Each thread = diffrent base, FS stays same on cswitch, only its base changes.