Page 1 of 4
CPU still in Ring3 on Interrupt
Posted: Wed Aug 21, 2013 3:24 am
by zhiayang
Help! I've some problems getting my TSS to work. I just get a #GPF when I do the 'ltr' instruction...
I've looked through the thoroughly confusing AMD manual and managed to come up with this:
Code: Select all
// 64-bit GDT
GDT64:
GDTNull:
.word 0 // Limit (low)
.word 0 // Base (low)
.byte 0 // Base (middle)
.byte 0 // Access
.byte 0 // Granularity / Limit (high)
.byte 0 // Base (high)
GDTCode:
.word 0xFFFF // Limit (low)
.word 0 // Base (low)
.byte 0 // Base (middle)
.byte 0x9A // Access
.byte 0xAF // Granularity / Limit (high)
.byte 0 // Base (high)
GDTData:
.word 0xFFFF // Limit (low)
.word 0 // Base (low)
.byte 0 // Base (middle)
.byte 0x92 // Access
.byte 0xAF // Granularity / Limit (high)
.byte 0 // Base (high)
GDTCodeR3:
.word 0xFFFF // Limit (low)
.word 0 // Base (low)
.byte 0 // Base (middle)
.byte 0xFA // Access
.byte 0xAF // Granulariry / Limit (high)
.byte 0 // Base (high)
GDTDataR3:
.word 0xFFFF // Limit (low)
.word 0 // Base (low)
.byte 0 // Base (middle)
.byte 0xF2 // Access
.byte 0xAF // Granulariry / Limit (high)
.byte 0 // Base (high)
GDTTSS:
.word 0x0068 // Limit (low)
.word 0xF000 // Base (Addr of TSS)
.byte 0x00 // middle
.byte 0xE9
.byte 0x80
.byte 0x00
.long 0x00
.long 0x00
// Pointer
GDT64Pointer:
.word GDT64Pointer - GDT64 - 1 // Limit
.long GDT64 // Base
.long 0
That's the GDT in its entirety.
Here's the code I use to load the TSS:
Code: Select all
Memory::Set32((uint8_t*)0xF004, 0x00008000, 1);
asm volatile("push %ax");
asm volatile("mov $0x33, %ax; ltr %ax");
asm volatile("pop %ax");
The #GPF error code is 0x30, as expected. As you can see, I intend for the kernel's stack to be at 0x8000 and the TSS itself to be at 0xF000.
I'm in long mode btw. Any pointers?
Thanks
Re: Problems with GDT and TSS structure
Posted: Wed Aug 21, 2013 3:57 am
by bluemoon
Is there special reason you want to load TR with 0x33 instead of 0x30?
Second, the descriptor for TSS should be:
[G=0][AVL=0][P][DPL=0][TYPE=1001]
ie. 0x89 instead of E9 (which mean directly accessible by ring3, you don't want it)
Third, your inline assembly don't tell the compiler you clobbered eax, note that the 3 blocks to the compiler are, 3 individual blocks.
It may works with volatile and magic keywords but you better learn the proper way to communicate with compiler, ie. input/output/clobber list.
Furthermore, if you use IST in interrupts, you may need to initialize them in TSS.
Re: Problems with GDT and TSS structure
Posted: Wed Aug 21, 2013 4:06 am
by zhiayang
bluemoon wrote:Is there special reason you want to load TR with 0x33 instead of 0x30?
Second, the descriptor for TSS should be:
[G=0][AVL=0][P][DPL=0][TYPE=1001]
ie. 0x89 instead of E9 (which mean directly accessible by ring3, you don't want it)
Third, your inline assembly don't tell the compiler you clobbered eax, note that the 3 blocks to the compiler are, 3 individual blocks.
It may works with volatile and magic keywords but you better learn the proper way to communicate with compiler, ie. input/output/clobber list.
Furthermore, if you use IST in interrupts, you may need to initialize them in TSS.
Hmm yes, thanks for your response
1. Apparently the bottom 2 bits of the selector index tell the CPU what ring it is: here, taken straight from
Getting to Ring 3
Code: Select all
mov ax, 0x2B ; Load the index of our TSS structure - The index is
; 0x28, as it is the 5th selector and each is 8 bytes
; long, but we set the bottom two bits (making 0x2B)
; so that it has an RPL of 3, not zero.
ltr ax ; Load 0x2B into the task state register.
ret
And that article also sets DPL=3 for the actual TSS...
Note that I don't use an IST for interrupts. I'd assume they are disabled until a TSS with entries for ISTs is loaded?
Also:
Problem appears to be solved, my math was a huge failure:
0x8 + 0x8 + 0x8 + 0x8 + 0x8
= 0x28, not 0x30
And setting the bottom 2 bits makes it 0x2B, not 0x33.
EDIT:
Finally: about the clobbering of registers:
I know how I should specify the clobber list. A friend of mine (sitting beside me) suggested i push/pop %ax (for no reason, but he
insisted!).
As for the volatile keyword, I use that with every ASM statement and I don't think it's a bad thing?
And if you don't mind, could you help clarify the issue with DPL? Thanks bluemoon
Sorry, bad day.
Re: Problems with GDT and TSS structure
Posted: Wed Aug 21, 2013 4:20 am
by bluemoon
If the task descriptor has DPL=3, a ring3 app may CALL the task gate and create all kind of troubles.
I don't know why the wiki specify RPL=3, since LTR is and must be executed at ring0 (CPL=0) anyway, any RPL has just no effect. I may have missed something.
Re: Problems with GDT and TSS structure
Posted: Wed Aug 21, 2013 4:29 am
by bluemoon
requimrar wrote:Note that I don't use an IST for interrupts. I'd assume they are disabled until a TSS with entries for ISTs is loaded?
The opposite, IST entries in TSS are ignored until an interrupt with IST is triggered, which the CPU then load such values from TSS (or TSS cache).
Re: Problems with GDT and TSS structure
Posted: Wed Aug 21, 2013 5:23 am
by bluemoon
A closer look at your code I see a potential issue, the 64-bit GDTR uses 8 byte linear address
Code: Select all
GDT64Pointer:
.word GDT64Pointer - GDT64 - 1 // Limit
.long GDT64 // Base
.long 0
Your code may get problem (a smart assembler should refuse to assemble!), since GDT64 may not fit in a long (I suppose it is 4 byte), especially for higher half kernel.
For your reference, here is mine (you may also want to include the CODE32DPL3 just to make syscall/sysret happy with the arrangement)
Code: Select all
align 16
gdtr dw 8 *8 -1
dq gdt
dw 0
align 16
gdt dd 0, 0
dd 0x0000FFFF, 0x00AF9A00 ; 0x08 CODE64 DPL0
dd 0x0000FFFF, 0x008F9200 ; 0x10 DATA64 DPL0
dd 0x0000FFFF, 0x00CFFA00 ; 0x18 CODE32 DPL3
dd 0x0000FFFF, 0x008FF200 ; 0x20 DATA64 DPL3
dd 0x0000FFFF, 0x00AFFA00 ; 0x28 CODE64 DPL3
dd 0, 0, 0, 0 ; 0x30 TSS
Re: Problems with GDT and TSS structure
Posted: Wed Aug 21, 2013 5:45 am
by zhiayang
Right... So i've managed to get the TSS working.. somewhat.
I'll probably post it once the problem is solved, for future forum-goers.
But anyway: #GPF on changing to user mode.
Here's the code, gotten from the wiki
Code: Select all
.global DoUsermode
DoUsermode:
mov $0x23, %ax
mov %ax, %ds
mov %ax, %es
mov %ax, %fs
mov %ax, %gs
mov %rsp, %rax
push $0x23
push %rax
pushf
push $0x1B
mov $0xF0000040, %rax
push %rax
iretq
Bochs says 3rd (12) exception with no resolution, on
at 0xF0000040, which exists the entry point for an ELF program (currently just while(true);)
So it's a stack fault exception.
Problem is:
1. where do I put the stack pointer for the user-program?
2. I can't do it before the iretq because obviously that pops values from the stack
3. Do I need like a bare asm-only trampoline to set a proper %rsp before any pushes (usual function-call preamble) are made?
Also:
I'm getting a headache from reading the manuals, so I'll ask here: How do I make an interrupt handler have an IST?
If I understood correctly an IST is essentially a stack pointer to a known-good location to make sure the handler doesn't crash right?
Thanks!
(Also thanks for telling me about the GDT64 pointer issue bluemoon!)
Re: Problems with GDT and TSS structure
Posted: Wed Aug 21, 2013 6:00 am
by bluemoon
requimrar wrote:1. where do I put the stack pointer for the user-program?
I use sysret, and therefore I just change rsp directly as needed.
Code: Select all
; void enter_ring3 ( unsigned long ring3_ip, unsigned long ring3_sp );
enter_ring3:
mov rsp, rsi
mov rcx, rdi
mov r11, 0x0202
db 0x48
sysret
For IRET approach it should be similar to 32-bit code, which you put the user stack pointer on ring0 stack before iret.
requimrar wrote:2. I can't do it before the iretq because obviously that pops values from the stack
You can. Check how it's done on 32-bit examples.
requimrar wrote:3. Do I need like a bare asm-only trampoline to set a proper %rsp before any pushes (usual function-call preamble) are made?
No but I strongly recommend this, unless you want to confuse the compiler, or otherwise mess with no-return and other magic compiler-specific attributes.
requimrar wrote:so I'll ask here: How do I make an interrupt handler have an IST?
There is a field in interrupt descriptor, if set to non-zero the IST mechanism is used.
The stack for such interrupt is then setup from IST field of TSS instead of ring0 stack; otherwise everything work similarly.
However, IST has tricky re-entrance issues, search the forum for more information.
Re: Problems with GDT and TSS structure
Posted: Thu Aug 22, 2013 1:40 am
by zhiayang
Right... taking your advice, I'm looking into SYSRET...
Firstly, I just want to get into user mode without having things crash...
According to AMD's Volume 3,
If the return is to 64-bit mode, CS is updated with the value of STAR.SYSRET_CS + 16.
The SS selector is updated to point to the next descriptor-table entry after the CS descriptor (STAR.SYSRET_CS +
Based on the GDT above, where selector 0x18 is UserCode and 0x20 is UserData (and 0x28 is the TSS)
Should I be putting 0x8 into STAR 48:63?
Here's the current code:
Code: Select all
DoUsermode:
mov $0xC0000081, %rcx
rdmsr
mov $0x000B000000000000, %rax
wrmsr
mov $0xF0000060, %rcx
mov $0x9000, %rsp
mov $0x0202, %r11
sysretq
The TSS is unchanged; anyway I think it should only be used when going from user->kernel right? (or is that not even required when using SYSCALL?)
So the code above basically sets the CS-16 selector to 0x0B...
0xF0000060 contains a valid instruction.
0x9000 is just some free space: do I need to push anything on this stack for SYSRET to work?
For some odd reason, BOCHS crashes with a page-fault.
Re: Problems with GDT and TSS structure
Posted: Thu Aug 22, 2013 2:46 am
by bluemoon
You need to enable syscall/sysret with MSR 0xC0000080, then put both SEG_CODE32_3 and SEG_CODE64_0 into the STAR (even you do not use CODE32 at all, it's that arrangement to make the cpu happy), and you probably want to supply LSTAR for syscall entry too. check my previous post for my GDT layout.
anyway, I do this:
Code: Select all
mov ecx, 0xC0000081 ; IA32_STAR
xor eax, eax
mov edx, ((SEG_CODE32_3+3)<<16) | SEG_CODE64_0
wrmsr
Re: Problems with GDT and TSS structure
Posted: Thu Aug 22, 2013 4:59 am
by zhiayang
bluemoon wrote:You need to enable syscall/sysret with MSR 0xC0000080, then put both SEG_CODE32_3 and SEG_CODE64_0 into the STAR (even you do not use CODE32 at all, it's that arrangement to make the cpu happy), and you probably want to supply LSTAR for syscall entry too. check my previous post for my GDT layout.
anyway, I do this:
Code: Select all
mov ecx, 0xC0000081 ; IA32_STAR
xor eax, eax
mov edx, ((SEG_CODE32_3+3)<<16) | SEG_CODE64_0
wrmsr
Ugh. I probably sound extremely whiny now, but I'm confused by this stuff to no end. You'd think a multi-million dollar company that designs the most popular desktop CPU would write better manuals.
1. DATA64R3 goes *before* CODE64R3?
2. What in the world is that CODE32R3 doing? If a CPU requires that sort of **** then I've lost faith.
I dug around and found something that could work:
Code: Select all
mov $0x23, %ax // Load the new data segment descriptor with an RPL of 3.
mov %ax, %dx // Propagate the change to all segment registers.
mov %ax, %es
mov %ax, %fs
mov %rsp, %rax // Save the stack pointer before pushing anything
push $0x23 // Push the new stack segment with an RPL of 3.
push %rax // Push what the was ESP before pushing anything.
pushfq // Push EFLAGS
// pop %rax
// or $0x200, %rax // Enable interrupts
// push %rax
push $0x1B // Push the new code segment with an RPL of 3.
mov $0xF0000060, %rax
push %rax // Push the EIP to IRET to.
iretq
Problem is, it only works on QEMU (as these things usually do) and not on Bochs or VBox.
Clearly something *very* wrong, but I cannot grasp my head around this, given the poor organisation of the AMD manual in general.
EDIT:
I've changed the GDT a bit: The GDTCodeR3 got its 0xFA changed to a 0xF8.
Code: Select all
GDT64:
GDTNull:
.word 0 // Limit (low)
.word 0 // Base (low)
.byte 0 // Base (middle)
.byte 0 // Access
.byte 0 // Granularity / Limit (high)
.byte 0 // Base (high)
GDTCode:
.word 0xFFFF // Limit (low)
.word 0 // Base (low)
.byte 0 // Base (middle)
.byte 0x9A // Access
.byte 0xAF // Granularity / Limit (high)
.byte 0 // Base (high)
GDTData:
.word 0xFFFF // Limit (low)
.word 0 // Base (low)
.byte 0 // Base (middle)
.byte 0x92 // Access
.byte 0xAF // Granularity / Limit (high)
.byte 0 // Base (high)
GDTCodeR3:
.word 0xFFFF // Limit (low)
.word 0 // Base (low)
.byte 0 // Base (middle)
.byte 0xF8 // Access
.byte 0xAF // Granulariry / Limit (high)
.byte 0 // Base (high)
GDTDataR3:
.word 0xFFFF // Limit (low)
.word 0 // Base (low)
.byte 0 // Base (middle)
.byte 0xF2 // Access
.byte 0xAF // Granulariry / Limit (high)
.byte 0 // Base (high)
GDTTSS:
.word 0x0068 // Limit (low)
.word 0xF000 // Base (Addr of TSS)
.byte 0x00 // middle
.byte 0xE9
.byte 0x80
.byte 0x00
.long 0x00
.long 0x00
// Pointer
GDT64Pointer:
.word GDT64Pointer - GDT64 - 1 // Limit
.quad GDT64 // Base
Re: Problems with GDT and TSS structure
Posted: Thu Aug 22, 2013 5:29 am
by bluemoon
Code: Select all
mov $0x23, %ax // Load the new data segment descriptor with an RPL of 3.
mov %ax, %dx // Propagate the change to all segment registers.
mov %ax, %es
mov %ax, %fs
mov %rsp, %rax // Save the stack pointer before pushing anything
push $0x23 // Push the new stack segment with an RPL of 3.
push %rax // Push what the was ESP before pushing anything.
pushfq // Push EFLAGS
// pop %rax
// or $0x200, %rax // Enable interrupts
// push %rax
push $0x1B // Push the new code segment with an RPL of 3.
mov $0xF0000060, %rax
push %rax // Push the EIP to IRET to.
iretq
A few problems, or improvement can be made:
1. do you meant to mov ds instead of dx?
2. I would set DS, ES after entered ring3, to void the mess if interrupt occurs before iretq (although no observable difference for flat address space)
3. I strongly against reusing the ring0 stack for application, the protection will get ugly. Instead, allocate a new page for it.
(3) is probably causing you trouble now.
Re: Problems with GDT and TSS structure
Posted: Thu Aug 22, 2013 5:51 am
by zhiayang
bluemoon wrote:
A few problems, or improvement can be made:
1. do you meant to mov ds instead of dx?
2. I would set DS, ES after entered ring3, to void the mess if interrupt occurs before iretq (although no observable difference for flat address space)
3. I strongly against reusing the ring0 stack for application, the protection will get ugly. Instead, allocate a new page for it.
(3) is probably causing you trouble now.
1. Right, haha thanks for spotting that
2. okay...
3. Right. Didn't think of that.
Code: Select all
cli
push $0x23 // Push the new stack segment with an RPL of 3.
push $0xA000 // Push what the was ESP before pushing anything.
pushfq // Push EFLAGS
// pop %rax
// or $0x200, %rax // Enable interrupts
// push %rax
push $0x1B // Push the new code segment with an RPL of 3.
mov $0xF0000070, %rax
push %rax // Push the EIP to IRET to.
iretq
That's the go-to-ring3-code, this is the actual ring3 code:
Code: Select all
void HelloTest()
{
asm volatile("mov $0x23, %ax \
mov %ax, %ds \
mov %ax, %es \
mov %ax, %fs \
mov %ax, %gs");
mset((void*)0xFD000000, 0x33, 1024 * 640 * 4);
// asm volatile("int $0");
while(true);
}
extern "C" void HelloStart()
{
HelloTest();
}
Results are the same:
Stack fault in BOCHS, VBox hangs (assume exception unhanded), QEMU works... although I don't know if it's actually in ring3.
Re: Problems with GDT and TSS structure
Posted: Thu Aug 22, 2013 6:00 am
by bluemoon
requimrar wrote:Results are the same:
Stack fault in BOCHS, VBox hangs (assume exception unhanded), QEMU works... although I don't know if it's actually in ring3.
Now you have some debug works to do.
1. Does the fault occurs in ring0 or ring3? (check the code)
2. do not enable interrupt (by push 0x0002 instead of 0x0202, and break at 0xF0000060 to see if you can land there. (don't do pushf which could be unknown state)
3. double check the write permission of your stack, you need to grant access even in page directories.
4. check if you have rsp0(or IST) properly set on TSS, so an exception can be handled from ring3.
Code: Select all
push $0xA000 // Push what the was ESP before pushing anything.
LoL, write to VGA screen? note that you should push the address of the end of stack - stack goes downward.
Re: Problems with GDT and TSS structure
Posted: Thu Aug 22, 2013 7:36 am
by zhiayang
bluemoon wrote:requimrar wrote:Results are the same:
Stack fault in BOCHS, VBox hangs (assume exception unhanded), QEMU works... although I don't know if it's actually in ring3.
Now you have some debug works to do.
1. Does the fault occurs in ring0 or ring3? (check the code)
2. do not enable interrupt (by push 0x0002 instead of 0x0202, and break at 0xF0000060 to see if you can land there. (don't do pushf which could be unknown state)
3. double check the write permission of your stack, you need to grant access even in page directories.
4. check if you have rsp0(or IST) properly set on TSS, so an exception can be handled from ring3.
Code: Select all
push $0xA000 // Push what the was ESP before pushing anything.
LoL, write to VGA screen? note that you should push the address of the end of stack - stack goes downward.
1. I've checked, it's in ring3... Is there a way to find out for certain that I'm in ring 3? (In bochs)
2. Right.
3. Every page mapped is 0x7
4. There's not much info I can find; you mentioned setting something in the IDT: how do I do that?
Also:
Isn't 0xA0000 the VGA graphics memory, not 0xA000?
Next:
Since I've said it crashes in ring3, here's the ring3 code:
Code: Select all
00000000f0000020 <_Z9HelloTestv>:
f0000020: 66 87 db xchg %bx,%bx
f0000023: b8 00 00 00 fd mov $0xfd000000,%eax
f0000028: c6 00 33 movb $0x33,(%rax)
f000002b: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
f0000030: eb fe jmp f0000030 <_Z9HelloTestv+0x10>
f0000032: 66 66 66 66 66 2e 0f data32 data32 data32 data32 nopw %cs:0x0(%rax,%rax,1)
f0000039: 1f 84 00 00 00 00 00
00000000f0000040 <HelloStart>:
f0000040: 55 push %rbp
f0000041: 48 89 e5 mov %rsp,%rbp
f0000044: 66 87 db xchg %bx,%bx
f0000047: e8 d4 ff ff ff callq f0000020 <_Z9HelloTestv>
HelloStart() is the entry point... You'll notice the addresses don't match up, I change them each time for now.
Bochs says the exception is here, a "3rd (12) exception with no resolution"
mov byte ptr ds:[rax], 0x33 ; c60033
Note that this should be in ring3 already.