OSDev.org

Hello everyone, I'm an italian student, and I'm rewriting for the fourth time (I still have a lot to learn about OS development), my simple OS. I'm trying to improve this version so much, so I decided to implement long mode, and maybe graphics for the first time, after a studying pause.
I'm still working on the bootloader, so my environment consists in NASM, bochs, other universal tools like an hex-editor, running in LinuxMint 18.2 'Sonya' on a HP P6 Pavilion (2011) machine. I successfully set up Long Mode, identity paging the first 4Mb of memory, and a basic environment, like keyboard's and RTC's IRQ handler(still developing to achieve HPET), but I found something strange when debugging my system call handler (INT 80h).
My syscall handler needs some quadword data to be pushed on the stack, just before the (r_flags if I remember right) data pushed by the INT instruction. Unlike common practice, I make the callee clean the stack, by moving the stack 8 byte upwards unless it moves the byte pointed by current ESP, then adding ESP,08h . This is done for every parameter needed by the called function. The only problem of this method is that it needs to know the number of byte pushed by the INT instruction, that's a known value and shouldn't be hard to find.
However, I found only partial data on INT instruction behavior in x86-64, so I tried to check it out by the following code(relying on Bochs magic breakpoint tool):
And this is the code of the INT 81h handler: That's very simple, when setting ESP on every 16-byte aligned address, at the breakpoints I got always the same RSP values:

1. 1000h
2. 0FD8h
3. 1000h

So seemed to be pushed 0x28 bytes. But then I repeated the experiment by pushing a QWORD value, before calling INT 81h, so setting RSP to 0x0FF8:
In this case, and in all cases with all 8-byte aligned RSP values before the INT call, the RSP values at the breakpoints are the following:

1. 0FF8h
2. 0FC8h (and not FD0 as expected)
3. 0FF0h (8-bytes under the original value!!!)

I haven't noticed that yet, because every handler worked well, as Bochs seems to push the important values AFTER pushing the content of RBP or RCX (it seems to choose randomly between these).
This introduces a very dangerous misalignment in the stack, as a "useless" information is pushed and never popped.
Can anybody please explain why is this happening? Is a Bochs bug, or does long mode need to work with 16-byte aligned stack addresses, or anything else?

The interrupt invocation in 64-bit mode aligning stack to 16-bye boundary:

NTER-PRIVILEGE-LEVEL-INTERRUPT:
.....
IF (IA32_EFER.LMA = 0) (* Not IA-32e mode *)
THEN
IF instruction pointer from IDT gate is not within new code-segment limits
THEN #GP(EXT); FI; (* Error code contains NULL selector *)
ESP ← NewESP;
SS ← NewSS; (* Segment descriptor information also loaded *)
ELSE (* IA-32e mode *) <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< read here
IF instruction pointer from IDT gate contains a non-canonical address
THEN #GP(EXT); FI; (* Error code contains NULL selector *)
RSP ← NewRSP & FFFFFFFFFFFFFFF0H; <===== align RSP
SS ← NewSS;
FI

Thank you very much! So, I think that the only solution is to make stack work only in 16bytes boundaries... Maybe by storing the not 16byte aligned pushed data in a qword location in Memory, and "emulating" push and pop to make them always work in 16byte boundaries. Or simpler by storing data only in the lower part of those 16bytes... But that seems really a bad idea

It's more conventional to pass parameters to a system call in registers rather than on the stack.