Reading the disk with AHCI.

[foliagecanine]

Reading all of the discussion above I assume that I'm not going anywhere without correctly enabling paging...
This is what I love/hate about os-dev, you never run out of things to implement.
So, ok, I'll study paging and I'll implement it. Then I'll come back and implement all of the steps listed above. I hope it doesn't take too long XD

Have you tried using the code in my previous post? It should identity map everything up to 4GiB.

[Bonfra]

Have you tried using the code in my previous post? It should identity map everything up to 4GiB.

I spent all day studying in deep the concepts of paging and now I just need to implement it.
I've copied your code to start from but now I'm trying to convert to intel syntax that little line of inline assembly.
This is the original line you posted:

Code: Select all

asm volatile("mov %0, %%cr3":: "r"(new_cr3));

and this is how I translated:

Code: Select all

asm volatile("mov cr3, %0":: "r"(new_cr3));

but it does not work, actually, if I run the code with the debugger line by line after executing that line it halts forever. Maybe I'm doing it in the wrong moment. I'm starting the kernel with the "week paging" I implemented the bootloader and then I call this new function in the kernel at some point. Is it wrong?

[xeyes]

Have you tried using the code in my previous post? It should identity map everything up to 4GiB.

I spent all day studying in deep the concepts of paging and now I just need to implement it.
I've copied your code to start from but now I'm trying to convert to intel syntax that little line of inline assembly.
This is the original line you posted:

Code: Select all

asm volatile("mov %0, %%cr3":: "r"(new_cr3));

and this is how I translated:

Code: Select all

asm volatile("mov cr3, %0":: "r"(new_cr3));

but it does not work, actually, if I run the code with the debugger line by line after executing that line it halts forever. Maybe I'm doing it in the wrong moment. I'm starting the kernel with the "week paging" I implemented the bootloader and then I call this new function in the kernel at some point. Is it wrong?

If you use GCC inline assembly the syntax is ATT not Intel unless you are somehow forcing things in Makefiles?

Maybe print CR3's value after you step through this move to be sure that it has the value you have in mind.

This is what I love/hate about os-dev, you never run out of things to implement.

Good news is that since there are so many things, you can choose to focus on what interests you the most.

Although some does have prerequisites and IMHO since AHCI involves juggling around multiple ring buffers and data buffers as well as the need to access MMIO registers, a working understanding of paging is needed.

[Bonfra]

If you use GCC inline assembly the syntax is ATT not Intel unless you are somehow forcing things in Makefiles?

Yes, this flags switch GCC to Intels syntax -masm=intel

Maybe print CR3's value after you step through this move to be sure that it has the value you have in mind.

I couldn't print it because it crashes right after the instruction but with GDB I managed to ensure it contained the right value. Being not this line the problem I also randomly started talking about this problem with paging on this discord server and they pointed out this thing:"your page tables are of the wrong size. all page table levels are 512 entries in long mode" but not being my code I don't know exactly what to change to follow their suggestion. Is it just the size of the arrays or something else?

[foliagecanine]

Being not this line the problem I also randomly started talking about this problem with paging on this discord server and they pointed out this thing:"your page tables are of the wrong size. all page table levels are 512 entries in long mode" but not being my code I don't know exactly what to change to follow their suggestion.

Hmm... This was my error in the code. I assumed that long mode simply extended the protected mode paging method, but it appears that ALL page tables are converted to 64-bit. Therefore, there can only be 512 entries of any table entry per page. I guess I didn't read far enough in the AMD64 Manual. If you want to look at it, it's the AMD64 Manual, Section 5.3.3

Try this code:

Code: Select all

typedef struct {
	uint8_t present:1;
	uint8_t readwrite:1;
	uint8_t user:1;
	uint8_t writethru:1;
	uint8_t cachedisable:1;
	uint8_t access:1;
	uint8_t reserved:3;
	uint8_t avl:3;
	uint64_t address:36;
	uint16_t ignore:15;
	uint8_t execdisable:1;
} __attribute__((packed)) pml4t_entry;

typedef struct {
	uint8_t present:1;
	uint8_t readwrite:1;
	uint8_t user:1;
	uint8_t writethru:1;
	uint8_t cachedisable:1;
	uint8_t access:1;
	uint8_t reserved:3;
	uint8_t avl:3;
	uint64_t address:36;
	uint16_t ignore:15;
	uint8_t execdisable:1;
} __attribute__((packed)) pdpt_entry;

typedef struct {
	uint8_t present:1;
	uint8_t readwrite:1;
	uint8_t user:1;
	uint8_t writethru:1;
	uint8_t cachedisable:1;
	uint8_t access:1;
	uint8_t dirty:1;
	uint8_t size:1;
	uint8_t global:1;
	uint8_t avl:3;
	uint64_t address:36;
	uint16_t ignore:15;
	uint8_t execdisable:1;
} __attribute__((packed)) page_dir_entry;

typedef struct {
	uint8_t present:1;
	uint8_t readwrite:1;
	uint8_t user:1;
	uint8_t writethru:1;
	uint8_t cachedisable:1;
	uint8_t access:1;
	uint8_t dirty:1;
	uint8_t pat:1;
	uint8_t global:1;
	uint8_t avl:3;
	uint64_t address:36;
	uint16_t ignore:11;
	uint8_t: pke:4;
	uint8_t execdisable:1;
} __attribute__((packed)) page_table_entry;

pml4t_entry pml4_table[512] __attribute__((aligned(4096)));
pdpt_entry pdp_table[512] __attribute__((aligned(4096)));
page_dir_entry page_dir[512*4] __attribute__((aligned(4096)));
page_table_entry page_table[512*512*4] __attribute__((aligned(4096)));

void identity_map_everything() {
	pml4_table[0].present = 1;
	pml4_table[0].readwrite = 1;
	pml4_table[0].user = 0; // You'll need to set this to 1 if you are using usermode
	pml4_table[0].execdisable = 0;
	pml4_table[0].address = ((intptr_t)&pdp_table[0])>>12;
	
	for (int i = 0; i < 4; i++) {
		pdp_table[0].present = 1;
		pdp_table[0].readwrite = 1;
		pdp_table[0].user = 0; // 1 if using usermode
		pdp_table[0].execdisable = 0;
		pdp_table[0].address = ((intptr_t)&page_dir[i*512])>>12;
	}

	for (int i = 0; i < 512; i++) {
		page_dir[i].present = 1;
		page_dir[i].readwrite = 1;
		page_dir[i].user = 0; // 1 if using usermode
		page_dir[i].address = ((intptr_t)&page_table[i*512])>>12;
	}

	for (int i = 0; i < 512*512; i++) {
		page_table[i].present = 1;
		page_table[i].readwrite = 1;
		page_table[i].user = 0; // 1 if using usermode
		page_table[i].address = i;
	}
	
	uint64_t new_cr3 = (uint64_t)(intptr_t)&pml4_table[0];
	asm volatile("mov cr3, %0":: "r"(new_cr3)); // (Since you said you were using Intel Syntax)
}

Also, where is the base address that your kernel loaded at?

[nullplan]

I assumed that long mode simply extended the protected mode paging method,

It does, but they extended the PAE method of paging, not the non-PAE method. Therefore all entries are 64-bits and there are 512 entries per page.

[Bonfra]

Also, where is the base address that your kernel loaded at?

the compiled elf file is loaded at 0x00100000 so the code starts at 0x00101000.

The code above has the same problem as before.. after the cr3 update, it does not work anymore...
I pushed all the code to a git repo so you can check if I messed up something somewhere else

[foliagecanine]

I pushed all the code to a git repo so you can check if I messed up something somewhere else

Thank you. I'll see if I can get it to work.

[foliagecanine]

Alright! I got the paging thing to work. I haven't tested any AHCI code, but at least we're getting somewhere.
It was mostly a problem of forgetting to zero out the data and forgetting to index pdp_table.

Rather than posting the code here, here's a link to a GitHub Gist:
https://gist.github.com/foliagecanine/6 ... 3ba2c9755b

(May I suggest adding some sort of license to BonsOS? I don't know exactly your goals for this project, so it's up to you.)

[Bonfra]

Alright! I got the paging thing to work.

Ok paging works! I just needed to move the stack because theese big data structures overlapped with it.
I tried to access that field in the AHCI code but I still get a page fault. I've checked the ABAR of the device I'm detecting and it is at 0xFEBD5000. Is this mapped by the paging code we discussed before?

(May I suggest adding some sort of license to BonsOS? I don't know exactly your goals for this project, so it's up to you.)

First or then I need to add it you are right

[foliagecanine]

Uhm.. yea paging setup works but for some reason it never returns from the function...

Yep. I figured that out too. It's because the stack is located at 0x200000.
Unfortunately, due to the amount of space needed for the page tables, your kernel now requires more than 1MiB of memory, so it overwrites the stack.
Good news is this is fixable.

boot/include/memory_layout.inc:

Code: Select all

... Replace Line 31-32:
Mem.Kernel.Stack.Bottom             equ 0x0E000000
Mem.Kernel.Stack.Top                equ 0x01000000
...

boot/include/paging.inc:

Code: Select all

... Insert after Line 60:
		mov dword [es:di + 0x28], 0x00A00000 | .StdBits | .LargePage
		mov dword [es:di + 0x30], 0x00C00000 | .StdBits | .LargePage
		mov dword [es:di + 0x38], 0x00E00000 | .StdBits | .LargePage
...

That should fix your problem.

EDIT:
I've updated the Gist to use 2MiB pages instead of 4KiB pages to make the kernel smaller. This code works with the existing stack but I would still recommend increasing the stack to prevent errors once your kernel grows beyond 1MiB.

[Bonfra]

It's because the stack is located at 0x200000.
Unfortunately, due to the amount of space needed for the page tables, your kernel now requires more than 1MiB of memory, so it overwrites the stack.

Uh, I'm starting to understand bugs XD

Good news is this is fixable.
...
That should fix your problem.

This solves the problem with paging and it works perfectly, but I still get a page fault accessing the AHCI device memory.
I tried to access that field in the AHCI code but I still get a page fault. I've checked the ABAR of the device I'm detecting and it is at 0xFEBD5000. Is this mapped by the paging code we discussed before?

[foliagecanine]

Okay. There was one mistake (I forgot to multiply the number of page directories to fill by 4).
This caused only the first 1GiB to be mapped.
It's fixed and the Gist is updated to now use 2MiB pages (makes kernel smaller).

Also, I would recommend changing
Mem.Kernel.Stack.Bottom to 0x00400000 and
Mem.Kernel.Stack.Top to 0x00500000

and

Code: Select all

pfa_init((void *)0x00500001);
...
pfa_deinit_region(0, 0x00500000);

instead of what you have currently (I assume that the memory layout should be Paging stuff, then Kernel, then stack, then heap)

[Octocontrabass]

2MiB pages

Be careful: behavior is undefined when a 2MiB page crosses a memory type boundary.

[foliagecanine]

Be careful: behavior is undefined when a 2MiB page crosses a memory type boundary.

Is this referring to AMD64 Architecture Programmer's Manual section 7.7.4 (MTRRs and Page Cache Controls)?

sigh

Edit:
Are there specific paging flags (Cachedisable, writethrough) that need to be set on memory used by AHCI (and others)?
How do you know which flags are necessary?