CPU still in Ring3 on Interrupt

[zhiayang]

Help! I've some problems getting my TSS to work. I just get a #GPF when I do the 'ltr' instruction...
I've looked through the thoroughly confusing AMD manual and managed to come up with this:

Code: Select all

// 64-bit GDT
GDT64:
	GDTNull:
		.word 0				// Limit (low)
		.word 0				// Base (low)
		.byte 0				// Base (middle)
		.byte 0				// Access
		.byte 0				// Granularity / Limit (high)
		.byte 0				// Base (high)
	GDTCode:
		.word 0xFFFF		// Limit (low)
		.word 0				// Base (low)
		.byte 0				// Base (middle)
		.byte 0x9A			// Access
		.byte 0xAF			// Granularity / Limit (high)
		.byte 0				// Base (high)
	GDTData:
		.word 0xFFFF		// Limit (low)
		.word 0				// Base (low)
		.byte 0				// Base (middle)
		.byte 0x92			// Access
		.byte 0xAF			// Granularity / Limit (high)
		.byte 0				// Base (high)
	GDTCodeR3:
		.word 0xFFFF		// Limit (low)
		.word 0				// Base (low)
		.byte 0				// Base (middle)
		.byte 0xFA			// Access
		.byte 0xAF			// Granulariry / Limit (high)
		.byte 0				// Base (high)
	GDTDataR3:
		.word 0xFFFF		// Limit (low)
		.word 0				// Base (low)
		.byte 0				// Base (middle)
		.byte 0xF2			// Access
		.byte 0xAF			// Granulariry / Limit (high)
		.byte 0				// Base (high)
	GDTTSS:
		.word 0x0068		// Limit (low)
		.word 0xF000		// Base (Addr of TSS)
		.byte 0x00			// middle
		.byte 0xE9
		.byte 0x80
		.byte 0x00
		.long 0x00
		.long 0x00



		// Pointer
	GDT64Pointer:
		.word GDT64Pointer - GDT64 - 1	// Limit
		.long GDT64						// Base
		.long 0

That's the GDT in its entirety.

Here's the code I use to load the TSS:

Code: Select all

Memory::Set32((uint8_t*)0xF004, 0x00008000, 1);
	asm volatile("push %ax");
	asm volatile("mov $0x33, %ax; ltr %ax");
	asm volatile("pop %ax");

The #GPF error code is 0x30, as expected. As you can see, I intend for the kernel's stack to be at 0x8000 and the TSS itself to be at 0xF000.

I'm in long mode btw. Any pointers?

Thanks

[bluemoon]

Is there special reason you want to load TR with 0x33 instead of 0x30?

Second, the descriptor for TSS should be:
[G=0][AVL=0][P][DPL=0][TYPE=1001]
ie. 0x89 instead of E9 (which mean directly accessible by ring3, you don't want it)

Third, your inline assembly don't tell the compiler you clobbered eax, note that the 3 blocks to the compiler are, 3 individual blocks.
It may works with volatile and magic keywords but you better learn the proper way to communicate with compiler, ie. input/output/clobber list.

Furthermore, if you use IST in interrupts, you may need to initialize them in TSS.

[zhiayang]

Is there special reason you want to load TR with 0x33 instead of 0x30?

Second, the descriptor for TSS should be:
[G=0][AVL=0][P][DPL=0][TYPE=1001]
ie. 0x89 instead of E9 (which mean directly accessible by ring3, you don't want it)

Third, your inline assembly don't tell the compiler you clobbered eax, note that the 3 blocks to the compiler are, 3 individual blocks.
It may works with volatile and magic keywords but you better learn the proper way to communicate with compiler, ie. input/output/clobber list.

Furthermore, if you use IST in interrupts, you may need to initialize them in TSS.

Hmm yes, thanks for your response

1. Apparently the bottom 2 bits of the selector index tell the CPU what ring it is: here, taken straight from Getting to Ring 3

Code: Select all

mov ax, 0x2B      ; Load the index of our TSS structure - The index is
                     ; 0x28, as it is the 5th selector and each is 8 bytes
                     ; long, but we set the bottom two bits (making 0x2B)
                     ; so that it has an RPL of 3, not zero.
   ltr ax            ; Load 0x2B into the task state register.
   ret

And that article also sets DPL=3 for the actual TSS...

Note that I don't use an IST for interrupts. I'd assume they are disabled until a TSS with entries for ISTs is loaded?
Also:
Problem appears to be solved, my math was a huge failure:

0x8 + 0x8 + 0x8 + 0x8 + 0x8
= 0x28, not 0x30
And setting the bottom 2 bits makes it 0x2B, not 0x33.

EDIT:
Finally: about the clobbering of registers:
I know how I should specify the clobber list. A friend of mine (sitting beside me) suggested i push/pop %ax (for no reason, but he insisted!).

As for the volatile keyword, I use that with every ASM statement and I don't think it's a bad thing?
And if you don't mind, could you help clarify the issue with DPL? Thanks bluemoon

Sorry, bad day.

[bluemoon]

If the task descriptor has DPL=3, a ring3 app may CALL the task gate and create all kind of troubles.

I don't know why the wiki specify RPL=3, since LTR is and must be executed at ring0 (CPL=0) anyway, any RPL has just no effect. I may have missed something.

[bluemoon]

Note that I don't use an IST for interrupts. I'd assume they are disabled until a TSS with entries for ISTs is loaded?

The opposite, IST entries in TSS are ignored until an interrupt with IST is triggered, which the CPU then load such values from TSS (or TSS cache).

[bluemoon]

A closer look at your code I see a potential issue, the 64-bit GDTR uses 8 byte linear address

Code: Select all

   GDT64Pointer:
      .word GDT64Pointer - GDT64 - 1   // Limit
      .long GDT64                  // Base
      .long 0

Your code may get problem (a smart assembler should refuse to assemble!), since GDT64 may not fit in a long (I suppose it is 4 byte), especially for higher half kernel.


For your reference, here is mine (you may also want to include the CODE32DPL3 just to make syscall/sysret happy with the arrangement)

Code: Select all

align 16
gdtr    dw  8 *8 -1
        dq  gdt
        dw  0

align 16
gdt     dd  0, 0
        dd  0x0000FFFF, 0x00AF9A00              ; 0x08 CODE64 DPL0
        dd  0x0000FFFF, 0x008F9200              ; 0x10 DATA64 DPL0
        dd  0x0000FFFF, 0x00CFFA00              ; 0x18 CODE32 DPL3
        dd  0x0000FFFF, 0x008FF200              ; 0x20 DATA64 DPL3
        dd  0x0000FFFF, 0x00AFFA00              ; 0x28 CODE64 DPL3
        dd  0, 0, 0, 0                          ; 0x30 TSS

[zhiayang]

Right... So i've managed to get the TSS working.. somewhat.
I'll probably post it once the problem is solved, for future forum-goers.

But anyway: #GPF on changing to user mode.
Here's the code, gotten from the wiki

Code: Select all

.global DoUsermode
DoUsermode:
	mov $0x23, %ax
	mov %ax, %ds
	mov %ax, %es
	mov %ax, %fs
	mov %ax, %gs

	mov %rsp, %rax
	push $0x23
	push %rax
	pushf
	push $0x1B
	mov $0xF0000040, %rax
	push %rax
	iretq

Bochs says 3rd (12) exception with no resolution, on

Code: Select all

push rbp

at 0xF0000040, which exists the entry point for an ELF program (currently just while(true);)

So it's a stack fault exception.
Problem is:
1. where do I put the stack pointer for the user-program?
2. I can't do it before the iretq because obviously that pops values from the stack
3. Do I need like a bare asm-only trampoline to set a proper %rsp before any pushes (usual function-call preamble) are made?

Also:
I'm getting a headache from reading the manuals, so I'll ask here: How do I make an interrupt handler have an IST?
If I understood correctly an IST is essentially a stack pointer to a known-good location to make sure the handler doesn't crash right?


Thanks!

(Also thanks for telling me about the GDT64 pointer issue bluemoon!)

[bluemoon]

1. where do I put the stack pointer for the user-program?

I use sysret, and therefore I just change rsp directly as needed.

Code: Select all

; void enter_ring3  ( unsigned long ring3_ip, unsigned long ring3_sp );
enter_ring3:
    mov     rsp, rsi
    mov     rcx, rdi
    mov     r11, 0x0202
    db 0x48
    sysret

For IRET approach it should be similar to 32-bit code, which you put the user stack pointer on ring0 stack before iret.

2. I can't do it before the iretq because obviously that pops values from the stack

You can. Check how it's done on 32-bit examples.

3. Do I need like a bare asm-only trampoline to set a proper %rsp before any pushes (usual function-call preamble) are made?

No but I strongly recommend this, unless you want to confuse the compiler, or otherwise mess with no-return and other magic compiler-specific attributes.

so I'll ask here: How do I make an interrupt handler have an IST?

There is a field in interrupt descriptor, if set to non-zero the IST mechanism is used.
The stack for such interrupt is then setup from IST field of TSS instead of ring0 stack; otherwise everything work similarly.

However, IST has tricky re-entrance issues, search the forum for more information.

[zhiayang]

Right... taking your advice, I'm looking into SYSRET...
Firstly, I just want to get into user mode without having things crash...

According to AMD's Volume 3,

If the return is to 64-bit mode, CS is updated with the value of STAR.SYSRET_CS + 16.
The SS selector is updated to point to the next descriptor-table entry after the CS descriptor (STAR.SYSRET_CS +

Based on the GDT above, where selector 0x18 is UserCode and 0x20 is UserData (and 0x28 is the TSS)
Should I be putting 0x8 into STAR 48:63?

Here's the current code:

Code: Select all

DoUsermode:
	mov $0xC0000081, %rcx
	rdmsr
	mov $0x000B000000000000, %rax
	wrmsr
	mov $0xF0000060, %rcx
	mov $0x9000, %rsp
	mov $0x0202, %r11

	sysretq

The TSS is unchanged; anyway I think it should only be used when going from user->kernel right? (or is that not even required when using SYSCALL?)

So the code above basically sets the CS-16 selector to 0x0B...
0xF0000060 contains a valid instruction.
0x9000 is just some free space: do I need to push anything on this stack for SYSRET to work?


For some odd reason, BOCHS crashes with a page-fault.

[bluemoon]

You need to enable syscall/sysret with MSR 0xC0000080, then put both SEG_CODE32_3 and SEG_CODE64_0 into the STAR (even you do not use CODE32 at all, it's that arrangement to make the cpu happy), and you probably want to supply LSTAR for syscall entry too. check my previous post for my GDT layout.

anyway, I do this:

Code: Select all

    mov     ecx, 0xC0000081     ; IA32_STAR
    xor     eax, eax
    mov     edx, ((SEG_CODE32_3+3)<<16) | SEG_CODE64_0
    wrmsr

[zhiayang]

You need to enable syscall/sysret with MSR 0xC0000080, then put both SEG_CODE32_3 and SEG_CODE64_0 into the STAR (even you do not use CODE32 at all, it's that arrangement to make the cpu happy), and you probably want to supply LSTAR for syscall entry too. check my previous post for my GDT layout.

anyway, I do this:

Code: Select all

    mov     ecx, 0xC0000081     ; IA32_STAR
    xor     eax, eax
    mov     edx, ((SEG_CODE32_3+3)<<16) | SEG_CODE64_0
    wrmsr

Ugh. I probably sound extremely whiny now, but I'm confused by this stuff to no end. You'd think a multi-million dollar company that designs the most popular desktop CPU would write better manuals.

1. DATA64R3 goes *before* CODE64R3?
2. What in the world is that CODE32R3 doing? If a CPU requires that sort of **** then I've lost faith.

I dug around and found something that could work:

Code: Select all

mov $0x23, %ax			// Load the new data segment descriptor with an RPL of 3.
	mov %ax, %dx			// Propagate the change to all segment registers.
	mov %ax, %es
	mov %ax, %fs
	mov %rsp, %rax			// Save the stack pointer before pushing anything
	push $0x23				// Push the new stack segment with an RPL of 3.
	push %rax				// Push what the was ESP before pushing anything.
	pushfq					// Push EFLAGS
	// pop %rax
	// or $0x200, %rax			// Enable interrupts
	// push %rax
	push $0x1B				// Push the new code segment with an RPL of 3.
	mov $0xF0000060, %rax
	push %rax				// Push the EIP to IRET to.
	iretq

Problem is, it only works on QEMU (as these things usually do) and not on Bochs or VBox.
Clearly something *very* wrong, but I cannot grasp my head around this, given the poor organisation of the AMD manual in general.


EDIT:

I've changed the GDT a bit: The GDTCodeR3 got its 0xFA changed to a 0xF8.

Code: Select all

GDT64:
	GDTNull:
		.word 0				// Limit (low)
		.word 0				// Base (low)
		.byte 0				// Base (middle)
		.byte 0				// Access
		.byte 0				// Granularity / Limit (high)
		.byte 0				// Base (high)
	GDTCode:
		.word 0xFFFF		// Limit (low)
		.word 0				// Base (low)
		.byte 0				// Base (middle)
		.byte 0x9A			// Access
		.byte 0xAF			// Granularity / Limit (high)
		.byte 0				// Base (high)
	GDTData:
		.word 0xFFFF		// Limit (low)
		.word 0				// Base (low)
		.byte 0				// Base (middle)
		.byte 0x92			// Access
		.byte 0xAF			// Granularity / Limit (high)
		.byte 0				// Base (high)
	GDTCodeR3:
		.word 0xFFFF		// Limit (low)
		.word 0				// Base (low)
		.byte 0				// Base (middle)
		.byte 0xF8			// Access
		.byte 0xAF			// Granulariry / Limit (high)
		.byte 0				// Base (high)
	GDTDataR3:
		.word 0xFFFF		// Limit (low)
		.word 0				// Base (low)
		.byte 0				// Base (middle)
		.byte 0xF2			// Access
		.byte 0xAF			// Granulariry / Limit (high)
		.byte 0				// Base (high)
	GDTTSS:
		.word 0x0068		// Limit (low)
		.word 0xF000		// Base (Addr of TSS)
		.byte 0x00			// middle
		.byte 0xE9
		.byte 0x80
		.byte 0x00
		.long 0x00
		.long 0x00



		// Pointer
	GDT64Pointer:
		.word GDT64Pointer - GDT64 - 1	// Limit
		.quad GDT64						// Base

[bluemoon]

Code: Select all

mov $0x23, %ax			// Load the new data segment descriptor with an RPL of 3.
	mov %ax, %dx			// Propagate the change to all segment registers.
	mov %ax, %es
	mov %ax, %fs
	mov %rsp, %rax			// Save the stack pointer before pushing anything
	push $0x23				// Push the new stack segment with an RPL of 3.
	push %rax				// Push what the was ESP before pushing anything.
	pushfq					// Push EFLAGS
	// pop %rax
	// or $0x200, %rax			// Enable interrupts
	// push %rax
	push $0x1B				// Push the new code segment with an RPL of 3.
	mov $0xF0000060, %rax
	push %rax				// Push the EIP to IRET to.
	iretq

A few problems, or improvement can be made:

1. do you meant to mov ds instead of dx?
2. I would set DS, ES after entered ring3, to void the mess if interrupt occurs before iretq (although no observable difference for flat address space)
3. I strongly against reusing the ring0 stack for application, the protection will get ugly. Instead, allocate a new page for it.

(3) is probably causing you trouble now.

[zhiayang]

A few problems, or improvement can be made:

1. do you meant to mov ds instead of dx?
2. I would set DS, ES after entered ring3, to void the mess if interrupt occurs before iretq (although no observable difference for flat address space)
3. I strongly against reusing the ring0 stack for application, the protection will get ugly. Instead, allocate a new page for it.

(3) is probably causing you trouble now.

1. Right, haha thanks for spotting that
2. okay...
3. Right. Didn't think of that.

Code: Select all

cli
	push $0x23				// Push the new stack segment with an RPL of 3.
	push $0xA000			// Push what the was ESP before pushing anything.
	pushfq					// Push EFLAGS
	// pop %rax
	// or $0x200, %rax			// Enable interrupts
	// push %rax
	push $0x1B				// Push the new code segment with an RPL of 3.
	mov $0xF0000070, %rax
	push %rax				// Push the EIP to IRET to.
	iretq

That's the go-to-ring3-code, this is the actual ring3 code:

Code: Select all

void HelloTest()
{
	asm volatile("mov $0x23, %ax \
					mov %ax, %ds \
					mov %ax, %es \
					mov %ax, %fs \
					mov %ax, %gs");
	mset((void*)0xFD000000, 0x33, 1024 * 640 * 4);
	// asm volatile("int $0");
	while(true);
}

extern "C" void HelloStart()
{
	HelloTest();
}

Results are the same:
Stack fault in BOCHS, VBox hangs (assume exception unhanded), QEMU works... although I don't know if it's actually in ring3.

[bluemoon]

Results are the same:
Stack fault in BOCHS, VBox hangs (assume exception unhanded), QEMU works... although I don't know if it's actually in ring3.

Now you have some debug works to do.

1. Does the fault occurs in ring0 or ring3? (check the code)
2. do not enable interrupt (by push 0x0002 instead of 0x0202, and break at 0xF0000060 to see if you can land there. (don't do pushf which could be unknown state)
3. double check the write permission of your stack, you need to grant access even in page directories.
4. check if you have rsp0(or IST) properly set on TSS, so an exception can be handled from ring3.

Code: Select all

push $0xA000         // Push what the was ESP before pushing anything.

LoL, write to VGA screen? note that you should push the address of the end of stack - stack goes downward.

[zhiayang]

Results are the same:
Stack fault in BOCHS, VBox hangs (assume exception unhanded), QEMU works... although I don't know if it's actually in ring3.

Now you have some debug works to do.

1. Does the fault occurs in ring0 or ring3? (check the code)
2. do not enable interrupt (by push 0x0002 instead of 0x0202, and break at 0xF0000060 to see if you can land there. (don't do pushf which could be unknown state)
3. double check the write permission of your stack, you need to grant access even in page directories.
4. check if you have rsp0(or IST) properly set on TSS, so an exception can be handled from ring3.

Code: Select all

push $0xA000         // Push what the was ESP before pushing anything.

LoL, write to VGA screen? note that you should push the address of the end of stack - stack goes downward.

1. I've checked, it's in ring3... Is there a way to find out for certain that I'm in ring 3? (In bochs)
2. Right.
3. Every page mapped is 0x7
4. There's not much info I can find; you mentioned setting something in the IDT: how do I do that?

Also:
Isn't 0xA0000 the VGA graphics memory, not 0xA000?




Next:
Since I've said it crashes in ring3, here's the ring3 code:

Code: Select all

00000000f0000020 <_Z9HelloTestv>:
    f0000020:	66 87 db             	xchg   %bx,%bx
    f0000023:	b8 00 00 00 fd       	mov    $0xfd000000,%eax
    f0000028:	c6 00 33             	movb   $0x33,(%rax)
    f000002b:	0f 1f 44 00 00       	nopl   0x0(%rax,%rax,1)
    f0000030:	eb fe                	jmp    f0000030 <_Z9HelloTestv+0x10>
    f0000032:	66 66 66 66 66 2e 0f 	data32 data32 data32 data32 nopw %cs:0x0(%rax,%rax,1)
    f0000039:	1f 84 00 00 00 00 00 

00000000f0000040 <HelloStart>:
    f0000040:	55                   	push   %rbp
    f0000041:	48 89 e5             	mov    %rsp,%rbp
    f0000044:	66 87 db             	xchg   %bx,%bx
    f0000047:	e8 d4 ff ff ff       	callq  f0000020 <_Z9HelloTestv>

HelloStart() is the entry point... You'll notice the addresses don't match up, I change them each time for now.



Bochs says the exception is here, a "3rd (12) exception with no resolution"
mov byte ptr ds:[rax], 0x33 ; c60033


Note that this should be in ring3 already.