Architecture library / libia32

[mkruk]

Hello everyone,

I've been reading this community's resources (i.e. in the wiki and these forums) for quite some time now and since this is my first post, I want to start by thanking everyone who contributes to this community. The wiki and the forums are really great resources for os development!

Now about the topic, a couple of days ago I've had the idea to write a "library" for basic IA-32 functions (segments/segmentation, hardware task switching, paging, etc.) that can be used by hobby os developers to compile into the kernel or as a "launch pad" for the own implementation.

I guess that might be interesting for those who are new to os development, however I'm unsure if a library like that is actually necessary for the following reasons:

• the Intel manuals are pretty straight-forward (in my opinion). From this PoV the lib is not really necessary
• os developers should write their own implementation from scratch and with the manuals, otherwise they run the risk of being spoonfed or using copy-pasted code
• writing an own implementation and running into problems will result in a better understanding of the architecture
• it seems to me that the IA-32 architecture is not really the most popular choice among hobby os developers

Please tell me if you think my idea is worth the effort or if I should rather spend my time on other projects

Regards, mkruk

[NickJohnson]

A better name for what you're trying to write is a hardware abstraction layer (HAL). Portable kernel designs make a distinction between architecture-dependent and architecture-independent things in this way (not to say that many kernels are portable). Depending on your kernel design, the architecture-dependent pieces can make up most of the kernel (l4, etc.), or be less than a tenth of it (Linux, etc.). I'm not sure how useful it will be to other people, mostly because of the general not-invented-here attitude that pervades osdev, but if you do write a well designed HAL, you will at the very least have a kernel framework for yourself that is better structured than most. It's a win-win situation: you either succeed, or fail and re-label it a kernel.

Also, IA-32 is, afaik, the most popular osdev architecture at the moment; x86-64 is gradually replacing it, but even real mode x86 hasn't been eclipsed entirely.

[mkruk]

A better name for what you're trying to write is a hardware abstraction layer (HAL). Portable kernel designs make a distinction between architecture-dependent and architecture-independent things in this way (not to say that many kernels are portable). Depending on your kernel design, the architecture-dependent pieces can make up most of the kernel (l4, etc.), or be less than a tenth of it (Linux, etc.). I'm not sure how useful it will be to other people, mostly because of the general not-invented-here attitude that pervades osdev, but if you do write a well designed HAL, you will at the very least have a kernel framework for yourself that is better structured than most. It's a win-win situation: you either succeed, or fail and re-label it a kernel.

Also, IA-32 is, afaik, the most popular osdev architecture at the moment; x86-64 is gradually replacing it, but even real mode x86 hasn't been eclipsed entirely.

No, I did not intend to write a hardware abstraction layer. Well, at least I did not mean to abstract from the IA-32 architecture. What I wanted to do is merely write a set of functions that can be used to manage the IA-32 specific structures (segment descriptors, and so forth), which means that the user would still have to know about the mechanics that lie behind those structures.

Basically, the idea was that a developer could declare for instance the GDT as array of n segment descriptors and tell the library where the GDT is and what it's size is. Now if the developer wants to create a TSS descriptor, he could just call the appropriate function and the function would return the segment selector of the descriptor.
And in a similar manner I wanted to implement functions for all the other IA-32 management structures.

But then again, it kind of misses the point of hobby os development, i.e. developing one's own code...

[NickJohnson]

The problem is that those things are not very hard to set up: most things, like setting up the gdt and idt and tss and paging, would only take a handful of lines of C and assembly. The thing that is tricky about segmentation and paging and hardware-assisted task switching (which is slower than doing it manually, btw) is the concept and side effects, not the semantics. Of course, this is different with hardware drivers, which frequently have strange semantics - the processor is nice in comparison - but are also out of the scope of a library like this.

[gerryg400]

The problem is that those things are not very hard to set up: most things, like setting up the gdt and idt and tss and paging, would only take a handful of lines of C and assembly. The thing that is tricky about segmentation and paging and hardware-assisted task switching (which is slower than doing it manually, btw) is the concept and side effects, not the semantics. Of course, this is different with hardware drivers, which frequently have strange semantics - the processor is nice in comparison - but are also out of the scope of a library like this.

You're correct, but sometimes it is difficult to know how to take the first steps. In the early days you could look at the vsta, linux or minix code for hints. Now those code bases appear too complex (or in one case are just rubbish design) Perhaps a simple, well-commented reference/implementation that people could begin with to understand how the concepts can be applied.

Having said that, there is a lot of information on the wiki already that appears to be ignored .....

- gerryg400

[NickJohnson]

But the Intel manuals and the wiki provide more than enough sample code and documentation for those basic operations. If you simply concatenated all of the wiki sample code, you would (ignoring all the error messages), get something pretty close to the OP's idea. And many tutorials exist that have this sort of thing: JamesM's tutorial goes through all of the steps to get the GDT, IDT, and paging to work and at the end gives you a zipfile of (albeit copyrighted) sample code.

[mkruk]

Yep, those are the reasons why I'm unsure if it all makes sense. Many of the aspects are already covered in the wiki articles and the library would be just another implementation for something that can be easily derived from the manuals.
But the goal would be to provide an implementation of everything that is covered in the Intel manuals, which would later also cover APIC, multiple processor management, etc. which are topics that I think are not covered too well by existing sources

[xenos]

I think the general idea is not too bad. There are some things which are frequently used, such as dealing with segment / control / debug registers, MSRs, APIC, TSC, performance counters... These could easily be covered by such a library, and it would certainly be some pretty piece of work.

Of course most of these things can be implemented just by a few lines of C / ASM code, and most OS developers have already done that at least once. In fact, my own kernel design uses a library named "libia32" containing exactly these things (and another one called "libx86" for things that are common between 32 and 64 bit modes, and finally there is "libamd64"). The only drawback I can see so far is that one would have to fix some interface for these library functions. For example, my kernel (and my libia32) is written in C++ and the functions look like Apic::Enable(), while a pure C programmer would certainly prefer something like apic_enable(). But that's probably a minor issue.

[qw]

I think it would make most sense if the "library" consists mostly of inline functions or function-like macro's. It may be very handy to have functions like make_descriptor(base, limit, etc.) and load_gdt(address, limit) that you can rely on, though they may be relatively small.

[mkruk]

so, I guess I'll just go ahead and write the library. It'll be useful, even if no one else uses it since my operating system doesn't have proper APIC support yet

it might take a while though since I'm working simultaneously on other projects (graduation, my job, my os, etc.) .. I hope nobody will mind if I just bump this thread when I'm done

[~]

I am currently trying to make such a thing, you can see it here:

http://126.sytes.net/projects/x86/OS/LowEST/

I call it LowEST (Low Executive System Tests) and is intended to provide an interface to access all hardware components from a DOS operating system by calling .COM programs and seeing back the true results on the hardware.

Currently has basic but well-defined functionality (but that's the idea), and later, for things like paging, memory management algorithms, etc., I plan to emulate them so that the programmer can appreciate step by step how they really work.

You can see then that the intention is to document the machine behavior and also provide fundamental snippets so basic that copy-pasting them or not would yield the same result when understood, but that provide a functional preview to save the programmer the risk of failed implementations by first seeing the true effects and only after understanding them, decide on designs.