Weird behaviour with x86_64 fast syscalls and int3

[sleirsgoevy]

Here's an excerpt from GDB prompt:
```
(gdb) x/2i $pc
=> 0xffffac4bc2bf5fe1: int3
0xffffac4bc2bf5fe2: int3
(gdb) stepi
0xffffac4bc2bf5fe2 in ?? ()
(gdb)
0xffffac4bc2bed901 in ?? ()
```
You can see that execution passes through the first int3, but the second int3 actually jumps to the handler.
Further experiments showed that whenever the instruction at 0x...fe1 is run, it is treated as 1-byte nop, no matter what byte was there.
For context: I'm playing with the Linux kernel, 0x...fe1 is the value of IA32_LSTAR that I've set (turns out that using a trampoline has the same effect, it's not tied to the LSTAR). The issue is present both on raw QEMU and on KVM, thus I suppose that its on my side and not an emulation bug. I wasn't able to find any info about special treatment of LSTAR/special behaviour of int3/special alignment requirements in the Intel manual.
P.S. When registering, I was asked which fault causes a reboot, with the answer being "triple". The irony is, the very PC I used to register actually hangs when a triple fault is issued (and yes, I can print to screen and it won't disappear, gg)
P.P.S. Bochs wasn't able to boot my test VM to a shell, so haven't tested there.

[Octocontrabass]

For context: I'm playing with the Linux kernel, 0x...fe1 is the value of IA32_LSTAR that I've set (turns out that using a trampoline has the same effect, it's not tied to the LSTAR). The issue is present both on raw QEMU and on KVM, thus I suppose that its on my side and not an emulation bug.

It could still be an emulation bug, if the breakpoint itself is being emulated. But why are you trying to insert the INT3 instruction in the first place?

P.S. When registering, I was asked which fault causes a reboot, with the answer being "triple". The irony is, the very PC I used to register actually hangs when a triple fault is issued (and yes, I can print to screen and it won't disappear, gg)

Rebooting after a triple fault is actually a compatibility feature. Some 286 OSes relied on it. Hardware designers aren't too concerned about 286 compatibility nowadays.

P.P.S. Bochs wasn't able to boot my test VM to a shell, so haven't tested there.

How far did it get? It might just need more RAM - Bochs defaults to a very small amount.

[feryno]

int3 is a TRAP type of exception, the exception is generated AFTER the instruction completes
FAULT type of exception is generated at the address of faulting instruction, like divide by zero etc.

[Demindiro]

int3 is a TRAP type of exception, the exception is generated AFTER the instruction completes

For me it jumps immediately to the handler so I doubt that's it.

The issue is present both on raw QEMU and on KVM

KVM does not play well with gdb.

I do not know why it doesn't work properly for you without KVM though (which is what I assume you mean with "raw QEMU", i.e. you use something like `-cpu max`). Have you tried running with `-d int`?