Page 1 of 1

Paging Questions

Posted: Sat Apr 13, 2013 10:45 am
by PearOs
Hey guys, so I am in Long Mode and I have encountered a problem that I have never seen before. I have paging currently enabled from enabling it in Protected mode to get into long mode, so PAE is enabled. But when I try to go outside of my 2 MB page mappings from Protected Mode I get a page fault, well obviously this would happen cause I'm outside of a mapped page. Now my question is, if I disable paging and try to read/write lets say in the 16mb mark will I still get a page fault? Thanks, Matt

Re: Paging Questions

Posted: Sat Apr 13, 2013 10:48 am
by Opcode
Paging is required in Longmode. You cannot disable it.

Re: Paging Questions

Posted: Sat Apr 13, 2013 11:02 am
by PearOs
Opcode wrote:Paging is required in Longmode. You cannot disable it.
Oh ok. So I have read tons and tons of tutorials, and I still don't really understand Paging. And I feel stupid because of it. So, I have a label in my 64bit Assembly Operating System called 'EndCode' and its just a marker to where the kernel and operating system end. Now I want to use the memory past that for objects and such. My whole system is loaded at the 1 mb mark. So I guess my understanding of Virtual Memory and Physical Memory are off. So according to paging, I can page up to 256TB of Data even though lets say the system only has 4gb of Ram. I don't need that, I just want to use all the memory past the 'EndCode' mark for allocation of data. But the system uses some of that memory for ACPI and other stuff I believe. So my main question is, with paging how do I setup pages so that after the 2mb mark its page up until the last section of usable memory. So lets say I have 256 mb of ram. The kernel uses 2mb of that and lets say we give the 256-4 the end mark so that 4mb is left for whatever is on the end. Is this sufficent? And how do I setup paging like this? Sorry if I sound a little lost. :) Anyways, my bootloader is isolinux if that helps. Thanks, Matt

Re: Paging Questions

Posted: Sat Apr 13, 2013 12:04 pm
by Opcode
You will need a good book from the library. The only thing I will tell you about this book is that you'll find it by looking at the sixty-ninth index card in the library's main index system, and going to Chapter 4. So you go down to the local library and start your search. You start by going through the index system and finding this entry and reading it. On this card it gives you the location on the bookshelf where to find a copy of the Intel® 64 and IA-32 Architectures Software Developer’s Manual. The librarian will make sure you're allowed to even look at this book, and may not give a **** if you bring it back all scribbled over. If it's not on the self it's probably in the back storage room.

Anyway, assuming you are going to want the first 256 library cards pointing to the first bookshelf locations you'll need the following:

4KB PML4 Table with entry #0 pointing to a PDPT table
4KB PDPT Table with entry #0 pointing to a PD table
4KB PD Table with the first 128 entries pointing to the first 128. 2mb page frames

Re: Paging Questions

Posted: Sat Apr 13, 2013 12:07 pm
by Brendan
Hi,
PearOs wrote:I don't need that, I just want to use all the memory past the 'EndCode' mark for allocation of data. But the system uses some of that memory for ACPI and other stuff I believe. So my main question is, with paging how do I setup pages so that after the 2mb mark its page up until the last section of usable memory. So lets say I have 256 mb of ram. The kernel uses 2mb of that and lets say we give the 256-4 the end mark so that 4mb is left for whatever is on the end. Is this sufficent? And how do I setup paging like this? Sorry if I sound a little lost. :) Anyways, my bootloader is isolinux if that helps. Thanks, Matt
It sounds like you want to map all physical pages of usable RAM into the virtual address space, so that you end up with a single large contiguous area that contains all usable RAM just above the end of the kernel.

This should be relatively easy. You'd need code to parse the memory map (to find all physical pages of usable RAM) and a function that maps a new page of RAM at the current "top of memory mapping in virtual address space".

For example:

Code: Select all

    for each entry in the memory map:
        if(entry->type == USABLE_RAM) {
             start = (entry->start_address + 4095) & ~0xFFF;      // Align to next page boundary
             end = (entry->start_address + entry->size) & ~0xFFF; // Align to previous page boundary
             while(start < end) {
                 add_page_to_mapping(start);
             }
        }
    }
And:

Code: Select all

void add_page_to_mapping(physical_address) {
    if(PDPT is needed at end_of_mapping) {
       use new physical page as PDPT;
    } else if(PD is needed at end_of_mapping) {
       use new physical page as PD;
    } else if(PT is needed at end_of_mapping) {
       use new physical page as PT;
    } else {
       create entry in existing page table to map physical page at end_of_mapping;
       end_of_mapping += 4096;
    }
}

Cheers,

Brendan

Re: Paging Questions

Posted: Sat Apr 13, 2013 12:13 pm
by Combuster
s/4011/4095/ (as in, it should be 1 << 12 - 1) :wink:

Re: Paging Questions

Posted: Sat Apr 13, 2013 12:21 pm
by PearOs
Brendan wrote:Hi,
PearOs wrote:I don't need that, I just want to use all the memory past the 'EndCode' mark for allocation of data. But the system uses some of that memory for ACPI and other stuff I believe. So my main question is, with paging how do I setup pages so that after the 2mb mark its page up until the last section of usable memory. So lets say I have 256 mb of ram. The kernel uses 2mb of that and lets say we give the 256-4 the end mark so that 4mb is left for whatever is on the end. Is this sufficent? And how do I setup paging like this? Sorry if I sound a little lost. :) Anyways, my bootloader is isolinux if that helps. Thanks, Matt
It sounds like you want to map all physical pages of usable RAM into the virtual address space, so that you end up with a single large contiguous area that contains all usable RAM just above the end of the kernel.

This should be relatively easy. You'd need code to parse the memory map (to find all physical pages of usable RAM) and a function that maps a new page of RAM at the current "top of memory mapping in virtual address space".

For example:

Code: Select all

    for each entry in the memory map:
        if(entry->type == USABLE_RAM) {
             start = (entry->start_address + 4011) & ~0xFFF;      // Align to next page boundary
             end = (entry->start_address + entry->size) & ~0xFFF; // Align to previous page boundary
             while(start < end) {
                 add_page_to_mapping(start);
             }
        }
    }
And:

Code: Select all

void add_page_to_mapping(physical_address) {
    if(PDPT is needed at end_of_mapping) {
       use new physical page as PDPT;
    } else if(PD is needed at end_of_mapping) {
       use new physical page as PD;
    } else if(PT is needed at end_of_mapping) {
       use new physical page as PT;
    } else {
       create entry in existing page table to map physical page at end_of_mapping;
       end_of_mapping += 4096;
    }
}

Cheers,

Brendan
Thanks guys, Ok so I am going to go work on reading the physical memory mappings. I believe this is stored somewhere in my mutliboot flags. I think I know where. And ill find all of the physical pages and then ill ask some paging questions once that's done. Thanks guys, Matt

This is very helpful!

Re: Paging Questions

Posted: Sat Apr 13, 2013 12:23 pm
by Brendan
Hi,
Combuster wrote:s/4011/4095/ (as in, it should be 1 << 12 - 1) :wink:
Doh - fixed :oops:


Cheers,

Brendan

Re: Paging Questions

Posted: Sun Apr 14, 2013 2:15 am
by PearOs
Hey guys, so I spent all day trying to understand paging. And I finally figured it out, but sadly my code to map from 0x0 to 4mb didn't work. When I tried to set the CR3 register in long mode it raised a page fault which makes no sense but ok. So here is my kernel's intilizer for paging that works and is what is currently active in my long mode:

Code: Select all

mov edi, 0x1000    ; Set the destination index to 0x1000.
mov cr3, edi       ; Set control register 3 to the destination index.
xor eax, eax       ; Nullify the A-register.
mov ecx, 4096      ; Set the C-register to 4096.
rep stosd          ; Clear the memory.
mov edi, cr3       ; Set the destination index to control register 3.

mov DWORD [edi], 0x2003      ; Set the double word at the destination index to 0x2003.
add edi, 0x1000              ; Add 0x1000 to the destination index.
mov DWORD [edi], 0x3003      ; Set the double word at the destination index to 0x3003.
add edi, 0x1000              ; Add 0x1000 to the destination index.
mov DWORD [edi], 0x4003      ; Set the double word at the destination index to 0x4003.
add edi, 0x1000              ; Add 0x1000 to the destination index.

mov ebx, 0000003          ; Set the B-register to 0x00000003.
mov ecx, 512              ; Set the C-register to 512.
.SetEntry:
mov DWORD [edi], ebx         ; Set the double word at the destination index to the B-register.
add ebx, 0x1000              ; Add 0x1000 to the B-register.
add edi, 8                   ; Add eight to the destination index.
loop .SetEntry               ; Set the next entry.

mov eax, cr4                 ; Set the A-register to control register 4.
or eax, 1 << 5               ; Set the PAE-bit, which is the 6th bit (bit 5).
mov cr4, eax                 ; Set control register 4 to the A-register.
There's some other part's, but you get the idea. So my issue mainly is that the stupid 32bit paging that is setup above, starts at 4k and runs through my multi boot structure and stuff. So I am tying to create a page setup with the first 4mb paged using variables in nasm. This part runs in 64bit mode, because I want to swap out page tables as a test basically, and for fun. But my paging code isn't working as I said above. Now one thing to consider is that the 'Bit_Set' function simply set the bit of my choice to what I would like in a number. It comes in handy. So here is my Paging code:

Code: Select all

;The root table that contains 512 Page Table Directory Pointer Table's
PML4T: TIMES 4096 db 0
;The first Page Directory Pointer Table that contains 512 Page Directory Table's
PDPT_0: TIMES 4096 db 0
;The first Page Directory Table that contains 512 Page Table's
PDT_0: TIMES 4096 db 0
;The first Page Table that contains 512 Page's
PT_0: TIMES 4096 db 0 ;2Mb mapped
;The seond Page Table that contains 512 Page's
PT_1: TIMES 4096 db 0 ;2Mb mapped
;A total of 20kb of data.

;Each table has 512 64bit value entry's.

;Setups paging in 32bit mode.
Paging_Setup:
;First we want to set the first entry in the PLM4T to PDPT_0
mov rax, PDPT_0
push rax
push 64
push 1 ;On, Present
call Bit_Set
push 63
push 1 ;On, Read/Write
call Bit_Set
pop rax
mov dword[PML4T], eax
;Now set the first entry in PDPT_0 to PDT_0
mov rax, PDT_0
push rax
push 64
push 1 ;On, Present
call Bit_Set
push 63
push 1 ;On, Read/Write
call Bit_Set
pop rax
mov qword[PDPT_0], rax
;Now set the first entry in PDT_0 to PT_0
mov rax, PT_0
push rax
push 64
push 1 ;On, Present
call Bit_Set
push 63
push 1 ;On, Read/Write
call Bit_Set
pop rax
mov qword[PDT_0], rax
;Now the last thing to do is set the last entry in the PDT_0 to itself
mov rax, PDT_0
push rax
push 64
push 1 ;On, Present
call Bit_Set
push 63
push 1 ;On, Read/Write
call Bit_Set
pop rax
mov dword[PDT_0+4088], eax
;Now map all pages from 0 to 2mb
;So PT_0 contains 512 4k entrys which is 512*4096 = 2MB
;So lets start our mapping process.
mov rax, 0 ;Start address
mov rbx, PT_0 ;Start
mov rcx, PT_0 ;End
add rcx, 4096 ;End
;-------------------------------------------------------------------------------
.loop_0:
cmp rbx, rcx
jge .done_0
; Now we want to move the current address
; into the page table, but we first have to set the present and read/write flags.
push rax ;Save the address
push rbx ;Save ths start
push rcx ;Save the end
push rax
push 64
push 1 ;Present
call Bit_Set
push 63
push 1 ;Read/Write
call Bit_Set
pop rdx
pop rcx ;End
pop rbx ;Start
pop rax ;Address
; Now we will move the new address with the flags set into the page table
mov dword[rbx], edx
; Now add 0x1000:4096 to the address
add rax, 0x1000
add rbx, 8 ;Entry size
; Just loop back now
jmp .loop_0
.done_0:
; -------------------------------------------------------------------------------
mov rbx, PT_1 ;Start
mov rcx, PT_1 ;End
add rcx, 4096 ;End
.loop_1:
cmp rbx, rcx
jge .done_1
; Now we want to move the current address
; into the page table, but we first have to set the present and read/write flags.
push rax ;Save the address
push rbx ;Save ths start
push rcx ;Save the end
push rax
push 64
push 1 ;Present
call Bit_Set
push 63
push 1 ;Read/Write
call Bit_Set
pop rdx
pop rcx ;End
pop rbx ;Start
pop rax ;Address
; Now we will move the new address with the flags set into the page table
mov dword[rbx], edx
; Now add 0x1000:4096 to the address
add rax, 0x1000
add rbx, 8 ;Entry size
; Just loop back now
jmp .loop_1
.done_1:
;Now set the PML4T to the CR3 register to swap page tables
;this might kill the whole system...
mov rax, PML4T
mov cr3, rax
jmp os_hang

Now you'll notice that in one part of the code up there I map PDT_0 to itself because this page here: http://www.rohitab.com/discuss/topic/31 ... directory/

was talking about a issue where you couldn't edit the page directory tables unless you mapped it to itself in the last entry. I tried my code with that done, and without it. I think my logic somewhere is flawed.

Some of that is a eye sore. Anyways, sorry for asking so much of you guys. I commented the code as much as possible when
I was writing it so that if it failed you guys could follow my logic some. Thanks, Matt

Re: Paging Questions

Posted: Sun Apr 14, 2013 3:30 pm
by PearOs
No worries. I finally figured out paging and got it working. I was able to Map 1GB to my OS so that I can use 1GB if I have it installed. So that's all I needed. Thanks guys, - Matt

Re: Paging Questions

Posted: Mon Apr 15, 2013 5:46 am
by gusc
Here's my approach on mapping all the memory, but it's in C. Also I do map all the reserved regions as well, so that I can read ACPI tables from them or access video buffer.