Info for Newbies + My OS

[Therx]

In an attempt to not confuse my self and give up. This time I'm writing everything down properly. Althogh there is hardly any info about my OS planing yet the research sections have got a fair bit of info on things like the GDT, IDT, multiboot header, PICs, paging etc.

I must admit some of it is copied from relevant technical docs (don't worry I have referenced to them) but I've collected the useful stuff and made it easier to read. May be worth a look if you're just starting and want easy to find info.

The project is here

That was badly worded but there we go :-X

Just trying to help.

Pete

[Tim]

Nice page . Documentation is slow to write, and annoying to keep up to date, but great to read 6 months later when you've forgotten what you were thinking when you wrote a certain piece of code.

Mind if I pick a few holes in it?

Although drivers will be ?separate? from the kernel, they must be compiled in with the
kernel (i.e. not loaded at runtime).

I would have done the opposite. A server environment is one where you specifically don't want to reboot. All sorts of technologies have arisen to avoid the need to reboot a server: hot-pluggable RAID and PCI come to mind. It's one thing to have to reboot in order to install a driver; it's another to have to recompile the kernel, and then reboot. In a server environment you need to be able to replace both hardware and software without affecting operations.

None of the Microsoft designed systems come near to being an ideal server as they are all based on the same internals as the desktop OS which has to support games etc.

Microsoft's current operating systems (Windows 2000, XP and Server 2003) all stem from the original Windows NT, which was specifically designed as a server/top-end workstation OS. NT has only really had games support since Windows 2000, when DirectX was made generally available on the NT platform.

They also all incorporate a complicated GUI which all requires a high performance PC.

I won't argue with that -- I've got Longhorn installed and it's slow . Luckily you can turn enough of the gizmos off until it looks like Windows 2000 again, and you can actually use it.

No memory should fall in a black hole like it does in windows.

What's that supposed to mean?

Virtual Memory Support (paging) allowing up to 4GB of RAM.

Why not provide for PAE support, at least in the future? That's useful for really big server, with up to 64GB of RAM.

Oh, and I think the Multiboot header is aligned to 4 bytes, not 4KB as you've written.

[Therx]

No memory should fall in a black hole like it does in windows.

What's that supposed to mean?

I suppose its more on the app side but I find with Windows (admittably 98) that after the PCs been on a few hours and I've been doing image editing or something it will eventually freeze when I try to put something on the clipboard. I haven't even got lots of programs in the background. However if I restart the same operation works fine. to my mind this must mean that somewhere along the line memory's being allocated but not freed. Although this is strictly an application problem I want my OS to automatically free memory if the thread using it ends.

Why not provide for PAE support, at least in the future? That's useful for really big server, with up to 64GB of RAM.

I've just had a look at this in the Intel Manual but it seems very complicated and I can't understand how you can have a 32bit address space which convert to a 36bit physical address space. Maybe you could explain?

In a server environment you need to be able to replace both hardware and software without affecting operations.

Point taken. I need to think this over. If drivers are loadable then I'd have to implement disk access quicker. And then you get the classic debate of which drivers should be inbuilt and which loaded etc. Maybe I could have all drivers that might be needed compiled in and then they could be loaded and unloaded at runtime.

Oh, and I think the Multiboot header is aligned to 4 bytes, not 4KB as you've written.

Opps :-[

Thanks for the feedback.

Pete

[Tim]

Although this is strictly an application problem I want my OS to automatically free memory if the thread using it ends.

OK, if you're referring to Windows 98, then I fully understand. This is a problem with the way the Windows 95 series is implemented, not with the Windows OS as a whole. I've not used Windows 95, 98 or Me in years so at first I can't understand points like this.

I've just had a look at this in the Intel Manual but it seems very complicated and I can't understand how you can have a 32bit address space which convert to a 36bit physical address space. Maybe you could explain?

Through paging. I haven't looked at it myself, but I think page table entries double in size, and you have an extra 'page directory directory' at the top.

Point taken. I need to think this over. If drivers are loadable then I'd have to implement disk access quicker. And then you get the classic debate of which drivers should be inbuilt and which loaded etc. Maybe I could have all drivers that might be needed compiled in and then they could be loaded and unloaded at runtime.

Yes, this is a traditional chicken-and-egg scenario. In general, you need to load the OS before the OS has started. You're already doing that with the kernel, so it's common to load the boot drivers alongside the kernel. For example, Mobius load the kernel, the kernel's config file, a storage device driver (ATA or floppy) and a file system driver.

[Therx]

Through paging. I haven't looked at it myself, but I think page table entries double in size, and you have an extra 'page directory directory' at the top.

Yeh I found it eventually. Its a shame that you have to double the size of all page entries and then the spare 28bits are 'reserved' and have to be set to 0 rather than being free for information such as the owner of the memory etc.

I think I'll give this a miss for the moment. But I must so its quite cool as my HD's only 10GB (5 years old) and if there was the maximum memory installed I could load it all into the RAM (6 times if I wanted) and then only copy it back at shutdown.

Pete

[Tim]

I don't think I'll be implementing PAE any time soon, either (well, I will if someone buys me 64GB of RAM I can use for testing). But it's worth bearing in mind. Any Mobius kernel functions that refer to a physical memory address use the addr_t type instead of uint32_t or void*. On versions of the kernel that support PAE, addr_t can be 64 bits wide; on others, it can be 32 bits.

[Robert Lee]

32 bit EIP + 16 bit CS = 48 bit address space.

Most PMode OSs don't use segments to extent the address space, instead opting for a flat model. But in theory, it is possible to have a 281,474,976,710,656 (48-bit) byte address space. At least, that is my understanding. But then, what the heck do I know? :o)

-Robert

[Therx]

32 bit EIP + 16 bit CS = 48 bit address space.

??? doesn't CS point to the segment selector. If you're suggesting setting the segment base to FFFFFFFFh and then using the offset. I think that the CPU will automatically loop it round to 0.

About the PAE. Just had a thought. Having 4 page dirs would be quite nice. You could then have the first dir (1st gb) for the kernel the 2nd for drivers and the 3rd + 4th for apps. This would mean on a task switch you just changed 2 pointers in the pointer dir and everything would be remapped.

Pete

[Tim]

Robert: No, the address space is still 32 bits. Each segment just refers to an area within the same address space. Recall that segments can overlap.

Pete: There's still only one CR3 register, and still only 32 bits of linear address space. The PTEs are wider but the page tables themselves are still 4KB in size. This means you need an extra level of translation, which I believe is the four-element page directory directory (can't remember the official name). CR3 points to this; each element of this points to a page directory; and each element of each page directory points to a page table.

[Therx]

What I meant was each element of the "Page-directory-pointer-table" could be used for a section of the OS (eg kernel, drivers, apps and something else) can then if you had a directory for each app you could change one section without the others. CR3 would never change (although it may be nessacery to set it to its self to clear the cache).

Would it matter if the page-directories overlapped ie:-
Dir1 maps areas 0->0.5GB and 1.5GB->2GB
Dir2 maps area 0.5GB->1.5GB
etc.

Also could the memory info provided by GRUB (or another multiboot loader) detect up to 64GB or just 4GB. Shame I don't have a machine with > 4GB to try this out on....

Pete

[Tim]

What I meant was each element of the "Page-directory-pointer-table" could be used for a section of the OS (eg kernel, drivers, apps and something else) can then if you had a directory for each app you could change one section without the others. CR3 would never change (although it may be nessacery to set it to its self to clear the cache).

You could do, but you could do this without PAE, too. With the three-level scheme used with PAE, each middle level directory covers 1GB of address space.

Would it matter if the page-directories overlapped ie:-

The page directories can't overlap.

OK, I've looked up the relevant section in the Intel manual (section 3.8 if you're playing at home). With PAE, we have:

• Page directory pointer table: Four entries of 64 bits each. Each entry points to...
• Page directory: 512 entries of 64 bits each. Each entry points to...
• Page table: 512 entries of 64 bits each. Each entry points to a page within the 36-bit physical address space

Bits 63-36 of each PDE and PTE are reserved and must be set to zero. The bottom 20 bits of each have the same meaning as they do without PAE enabled.

Also could the memory info provided by GRUB (or another multiboot loader) detect up to 64GB or just 4GB.

The Multiboot standard allows for 64-bit addresses in the memory map structures, as do the BIOS memory map services.

[mr. xsism]

hey, that's a nice little collection of data strucutres. How long did it take you to write that up? Looks very nice and your project is actually profesional looking. Quite different than what i am use to seeing in the hobby OS communities. Interesting nonetheless.

I am glad you are planning ahead. i had my own study OS that got me pretty far pretty fast with my current project. But i didn't plan much into tasking and i am having a hard time with understand how it all will flow. I am making progress, just really slow progress.

I am thinking of dumping my current project and planning all the way to application loading this time. But I have gotten so far. But not far enough. Time will tell I suppose.

Kindly,
mr. x

[Pype.Clicker]

i'd just like to bring to your attention that parts of this work is quoting from the Intel Manuals, and that this needs to be explicitly mentionned, if you want to avoid troubles with Copyright Lawyers

[mr. xsism]

yes, you have a resources section, but you left out the quiess from intel. Even the boldness looks the same. So give some credit for the big guys.

[Veldriss]

Is it possible to download the source code off it anywere?
Looks very intressting what you are doing here.