OSDev.org

I know this will sound foolish & noobish, but I couldn't really find anything useful about it.
I successfully finished FAT32 bootloader (well, depends on the prospective...I guess really completed is only part that loads next stage located within FAT32 by filename provided) for my USB, but now I am stuck with memory mapping- I read the articles about it here, I wrote the function to make it, but now I keep asking myself what the heck should I do with reserved memory that it returns?

I mean: Should I be doing some special privileges (ring 0 only or something) with it in GDT? Or should I just simply make "usable data" and only prevent access to it for anything other than kernel (in case the need arrives) within functions for dedicating memory to other processes? Or should I do both? (or is it only up to me in the end?)

A little off topic here:
I also know I should get myself better familiar with memory addressing first, but I still decided to ask about it here, since I am already creating this topic. I'd like to hear some more explaination on memory mapping (making higher half kernel) and for memory addressing within GDT (once it is created) - don't worry, I will do a lot more research on it, it is just that I am not a native english speaker, so I have difficulties finding correct keywords for searching (and understanding some of stuff written in wiki)... I'd just like to "remove" my kernel from 1MB memory (to leave it free...You never know when you'll need to jump back there, since I am planning on operating within 64bit mode mostly - no VMode there).

Also another (simple) question (for which I might know the answer already, but being sure is breing sure): About running 32bit programms within 64bit mode. Do I have to switch to compatibility mode? (within 64bit mode you CAN access 32bit registers, so I guess I wouldn't really have to do that, switch to compatibility mode, I mean, right? Only other bits, non 32bit ones of 64bit registers, gets zeroed?)

Sincerely.

Hi,

makuc9 wrote:I know this will sound foolish & noobish, but I couldn't really find anything useful about it.
I successfully finished FAT32 bootloader (well, depends on the prospective...I guess really completed is only part that loads next stage located within FAT32 by filename provided) for my USB, but now I am stuck with memory mapping- I read the articles about it here, I wrote the function to make it, but now I keep asking myself what the heck should I do with reserved memory that it returns?

I mean: Should I be doing some special privileges (ring 0 only or something) with it in GDT? Or should I just simply make "usable data" and only prevent access to it for anything other than kernel (in case the need arrives) within functions for dedicating memory to other processes? Or should I do both? (or is it only up to me in the end?)

In general, the 'reserved' parts of the memory map are areas that the OS shouldn't touch and never needs to access. However, during boot an OS may need to search for a few different tables (e.g. ACPI tables, SM BIOS tables, MP specification tables) and these searches typically include searching the BIOS area (which is reported as 'reserved'). For this reason, I'd search for these tables and copy them to normal usable RAM in the boot code somewhere (e.g. before paging is started) so that the rest of the OS can treat the tables like normal data (and so that the boot code can reclaim any 'ACPI reclaimable' areas before passing control to the rest of the OS, to avoid the need for the rest of the OS to worry about it).

makuc9 wrote:A little off topic here:
I also know I should get myself better familiar with memory addressing first, but I still decided to ask about it here, since I am already creating this topic. I'd like to hear some more explaination on memory mapping (making higher half kernel) and for memory addressing within GDT (once it is created) - don't worry, I will do a lot more research on it, it is just that I am not a native english speaker, so I have difficulties finding correct keywords for searching (and understanding some of stuff written in wiki)... I'd just like to "remove" my kernel from 1MB memory (to leave it free...You never know when you'll need to jump back there, since I am planning on operating within 64bit mode mostly - no VMode there).

This is what paging is for - it lets you map any physical pages (at any physical address) into the virtual address space anywhere. For example, the kernel might be at the physical address 0x00100000, and paging might map it to the virtual address 0xFFFFFFFF80000000.

makuc9 wrote:Also another (simple) question (for which I might know the answer already, but being sure is breing sure): About running 32bit programms within 64bit mode. Do I have to switch to compatibility mode? (within 64bit mode you CAN access 32bit registers, so I guess I wouldn't really have to do that, switch to compatibility mode, I mean, right? Only other bits, non 32bit ones of 64bit registers, gets zeroed?).

In protected mode, the GDT/LDT entry for a code descriptor has a flag to select 16-bit or 32-bit; and this allows you to run 16-bit code or 32-bit code in protected mode.

In long mode, the GDT/LDT entry for a code descriptor has 2 flags to select 16-bit, 32-bit or 64-bit; and this allows you to run 16-bit code, 32-bit code or 64-bit code in long mode. However, 64-bit code behaves a little differently (e.g. most segment bases and limits are ignored, some instructions are deprecated and used as prefixes, etc) so CPU manufacturers like to call it "the 64-bit sub-mode of long mode", and call 16-bit and 32-bit code in long mode "the compatibility sub-mode of long mode".

You can't (easily) run 32-bit programs in the 64-bit sub-mode of long mode; but you can run 16-bit and 32-bit programs in the compatibility sub-mode of long mode.

Note: For all of the above, "16-bit code" means 16-bit code designed for protected mode segmentation and protection rules; and does not include real mode code designed for real mode segmentation and real mode's complete lack of protection rules. You can run real mode code (safely) in protected mode using "the virtual8086 sub-mode of protected mode", but there isn't any "virtual8086 sub-mode of long mode" so you can't run real mode code in long mode at all.

However; a lot of the opcodes for 32-bit instructions have the same meaning in 64-code code, so if 32-bit code is very carefully designed (e.g. doesn't make assumptions about stack alignment or offsets from ESP, doesn't use instructions like PUSHAD/POPAD, doesn't use smaller versions of INC/DEC, etc.) it would be possible to have code that will run in 32-bit protected mode and 32-bit long mode and 64-bit long mode. You'd have to write this code manually in assembly (while being extremely cautious) though, as there are no tools that will help (no compiler will generate "mixed mode" code, and no assembler will warn you when you've used a normal 32-bit instruction that doesn't behave the same in 64-bit code). Of course this would be difficult and mostly pointless (easier to just write normal 32-bit code, or normal 64-bit code).


Cheers,

Brendan

Thanks Brendan, this was helpful. However one question still remains. Should I prevent access to those reserved memory using GDT, or should I simply not include it within GDT (this should mean that LMODE can't access it, right?)? Or should I only create a function that when accessing memory simply "skipes" those areas?

And Brendan? I've been looking at the link from one of your posts http://bcos.hopto.org/temp/init.html. I'd just like to know where are those instructions defined, like "CALL_BOOT_API". (i did not try to assemble it yet, because I doubt that nasm has any of those predefined, but I might be wrong). And how much memory is it supposed to map? (I'd like to switch to 64bit mode directly, but the more I think about it, more obvious it is to me that I should go to PMODE first, to make things easier...)

Sincerely,
Armin.

Hi,

makuc9 wrote:Thanks Brendan, this was helpful. However one question still remains. Should I prevent access to those reserved memory using GDT, or should I simply not include it within GDT (this should mean that LMODE can't access it, right?)? Or should I only create a function that when accessing memory simply "skipes" those areas?

Normally you'd have a physical memory manager that keeps track of all the free pages of usable RAM; and (during boot) you'd initialise the physical memory manager by adding any pages in from any areas in the memory map where "area type = usable RAM". Then you'd have a virtual memory manager that uses the physical memory manager to allocate physical pages (which can't include "system" or "reserved" areas) and maps them wherever you want them in the virtual address space. In this way, it's impossible for pages in reserved/system areas to end up in any virtual address space.

For the GDT; typically all code and all data descriptors are set to "base = 0, limit = max." so that you can forget about them. In long mode you don't have a choice (the CPU ignores the actual values anyway).

makuc9 wrote:And Brendan? I've been looking at the link from one of your posts http://bcos.hopto.org/temp/init.html. I'd just like to know where are those instructions defined, like "CALL_BOOT_API". (i did not try to assemble it yet, because I doubt that nasm has any of those predefined, but I might be wrong). And how much memory is it supposed to map? (I'd like to switch to 64bit mode directly, but the more I think about it, more obvious it is to me that I should go to PMODE first, to make things easier...)

I was working on some boot code and wondering if it's possible to go directly from real mode to long mode; so I slapped an ugly hack into the middle of the boot code to test it out. Things like "CALL_BOOT_API" and "MARK" were just macros I'd been using for the rest of the boot code that made it a little easier - it won't assemble without fixing them. The "MARK" was something I was using for debugging and can be simply deleted. The "CALL_BOOT_API" macro called an API that was provided by the previous stage, and looks like it was only used for memory management (e.g. allocating free physical pages).

Mostly, it's just example code (rather than code that was designed to be cut & pasted).



Cheers,

Brendan

I never planned on copypasting it ... I've only been reading comments (and steps) and than I was wondering what those macros were supposed to be there for.
So this helped a lot. Thank you. And for all other explaination.

Brendan wrote:For the GDT; typically all code and all data descriptors are set to "base = 0, limit = max." so that you can forget about them. In long mode you don't have a choice (the CPU ignores the actual values anyway).

Do you mean I should also ignore privileges and set them all to ring 0? What do you mean by CPU ignores the actual values? What am I missing? (now i am starting to think that i should go back to the very beginning...)

You should go back to the manual, it describes it in detail.

tl;dr: RTFM

Hi,

makuc9 wrote:

Brendan wrote:For the GDT; typically all code and all data descriptors are set to "base = 0, limit = max." so that you can forget about them. In long mode you don't have a choice (the CPU ignores the actual values anyway).

Do you mean I should also ignore privileges and set them all to ring 0? What do you mean by CPU ignores the actual values? What am I missing? (now i am starting to think that i should go back to the very beginning...)

A segment register has 4 parts:

• A visible value (e.g. the value that would be copied into AX if you did "mov ax,ds").
• A hidden base address. This is loaded from the GDT/LDT entry in protected mode or long mode, and calculated from the visible value in real mode (e.g. "base_address = (value << 4) & 0xFFFF0"). It's ignored in the 64-bit sub-mode of long mode for all segment registers except for FS and GS. Note: FS and GS are "special" because most OSs use them for fast access to thread local storage and/or CPU local storage.
• A hidden limit. This is loaded from the GDT/LDT entry in protected mode or long mode, and never changed in real mode (default is 64 KiB unless it's changed in protected mode). It's ignored in the 64-bit sub-mode of long mode for all segment registers (including FS and GS).
• A hidden set of attributes. This is loaded from the GDT/LDT entry in protected mode or long mode and determines the segment's characteristics - if it can be read/written/executed; if the default size is 16-bit, 32-bit or 64-bit; etc.

A code or data descriptor in the GDT/LDT has all of this information, and also has a few additional flags (DPL, present, accessed) that are only used when a segment register is loaded.

All of these things are used in the 64-bit sub-mode of long mode, except for the base address (most of the time) and limit (always).


Cheers,

Brendan

One (two) more question (everything you've posted so far Brendan is highly appreciated and I apologize for not replying & thanking earlier, but I did read them and started doing some more on-topic research on my own). Anyway, don't take this as a stupid question!

I made a create memory map function using BIOS call ax=e820h and now I want to know something for sure (because it seems very weird to me): Is it possible for memory map to contain (don't laugh) 59000 entries (it is not a typo)?

I tried it on 3 machines.
1st testing machine was VirtualBox and it returned 5 entries (Base Memory set to 512MB).
2nd testing machine was my laptop PC, which returned those 59000 entries (and it is not a typo!) (has 4GB RAM).
3rd testing machine was my desktop PC, which returned 3 entries (a little older, but has 4GB RAM). I tried on this one only because of 2nd machine's weird results (because it was first real-hardware, so I was unsure).

I still have to implement a system to eliminate no-text, zero length entries, but I doubted that it would be necessary (now I think I should). Could this be the case? (I couldn't really analyse all printed entries, because monitor kept refreshing.)

Is this a valid memory map (I know I should remove duplicate entries)? (It is from VirtualBox testing machine).

If it is:
What does this FFFF word mean? Does it mean ACPI 3.0 Extended Attributes are set (because it is kind of weird. I did not imagine it like that from the wiki description)? Or does it simply states end of memory map? (it is last entry, after all)
Or am I reading it from the wrong side? (memory is written as is, byte by byte from specified location in MM creation) Should I be reading 0x0000000000000020 as the first base address?

But than again, doesn't 0x00000002 means "Protected" memory?

Hope that doesn't sound too noobish (well, I kind of am, but I still always try my best to research things before asking here).

Sincerely.

makuc9 wrote:

This map looks fairly arbitrary (i.e. not correct).
The memory map descriptor has only 3 fields - 64 bit base, 64 bit size and type.
(Edit: ah, my bad, there's also 4 bytes of extended attributes)

You may need to print the little endian data in a different order.

Hi,

makuc9 wrote:I made a create memory map function using BIOS call ax=e820h and now I want to know something for sure (because it seems very weird to me): Is it possible for memory map to contain (don't laugh) 59000 entries (it is not a typo)?

In theory, there's no fixed limit on the number of entries that might be returned. In practice it's rare to have more than 30 of them (and if I remember right Linux has space reserved for 256 entries and won't handle more than that).

makuc9 wrote:Is this a valid memory map (I know I should remove duplicate entries)? (It is from VirtualBox testing machine).

No, it can't be valid. For 80x86 there's an architectural maximum physical address size of 52-bit (and most CPUs don't support that - e.g. only supporting 48-bit physical addresses). This means that it's impossible for an area to have a starting address or a size (or an ending address) that doesn't fit in 52 bits (like 0x7EFB61C300000000).

It's very likely you've got a bug somewhere.


Cheers,

Brendan

I am reading qword from right to left, if that's what you mean. And if wiki (detecting memory) is correct, each memory map entry consists of 24 bytes (20 in most cases, but 14 are to be sure).

But is the whole map supposed to be read backwards? (meaning I splited qwords wrongly - the last 2 dwords are a single qword)

Brendan wrote:No, it can't be valid. For 80x86 there's an architectural maximum physical address size of 52-bit (and most CPUs don't support that - e.g. only supporting 48-bit physical addresses). This means that it's impossible for an area to have a starting address or a size (or an ending address) that doesn't fit in 52 bits (like 0x7EFB61C300000000).

But did you read qword backward? I did not translate anything yet... Or am I maybe supposed to read everything backward? Besides, how can values be wrong, if BIOS writes it? I only increased DI register for 24 bytes (as recommended in wiki)...

But did you read qword backward?

Regardless of which byte you'd try to start at, you'll be crossing the 52-bit limit. Also, if you can't write or bother to fix a simple printing function, then how would we believe you can write a complex one?

There's at least three bugs here, of which one is apparently carrying coder laziness disease . Go fix them or come with a better description.

Ok, sorry, I will fix the printing procedure. But I still do not know wether to read it all backward or only qwords... Besides, it is not that hard to change the direction flag ...

makuc9 wrote:Ok, sorry, I will fix the printing procedure. But I still do not know wether to read it all backward or only qwords...

Are you familiar with concept of LITTLE ENDIAN architecture (of which all Intel processors in use are)?

Lower bytes of a value will be stored in the memory first, higher bytes next.

Also, don't randomly increase DI by 24, the function description in wiki clearly says:

CL will contain the number of bytes actually stored at ES:DI (probably 20).

Find yourself an order for the first 8 bytes of the second row such that it fits in the mentioned 48, or even 52 bits. Then if that creates a working value, your printing function should correspond to that. You'll notice that's impossible.


Anyway, I'll give you the benefit of the doubt and get my crystal ball. It's rather cloudy but I think it says "stacktrace".