Newbie welcomes small theory assistance

[Nekroze]

ok I've been reading OSdev wiki and a few other sources for over a month now to get the general theory down and learn the flow of this significantly different river.

i am a long time programmer but have been using C/C++ for a very short time but i have always picked up syntax and style rather fast, so long as i have the means to put what i am trying to learn into practice and see feedback on ways i can manipulate the code.

now the main thing i I'm inquiring is the general style and methods that need be used for a kernel.

i have gotten to grips with the main kernel loop partially, but that is covered enough to learn about in the wiki, the things i need to know about is in what i need to pre-assemble in order to support things latter.

for instance i will need to run executable files of an as yet undecided format. but i don't know what i would have to do to even cover the theory of this working in myos. i have not found anything on this topic so far.

also something i cannot grasp completely is memory management in C/C++ (oh my os will be in C++) i have read many many things on the topic however i do not understand how without referring to the physical memory address like in a few assembly things i have seen, it can actually locate and read/write specific places in the memory.

i also am having difficulties finding relevant beginner and a tad higher learning resources in assembly. for instance i can find many things that explain how to do stuff at a pre-requisite level of which i have yet to attain. i did manage to track down an eBook version for "The Art of Assembly" apparently a great book to learn from however it uses some different compiler and syntax as compared to what i believe NASM uses which is my preferred assembler so far just because it is very very common. the book is sub-titled the HLA Version and as such it has many differences that even the first chapter is enough to warn me off of it.

for now i think this is all of the nudging i need to be facing my target.

thank you in advanced and sorry for being long winded, lol.

sign,
Nekroze

[Combuster]

Given that there are so many questions on so many unrelated fields, I think its best that you just start somewhere and see how far you can get. You will probably encounter a lot of common problems, but then you'll know what they are and what to search for. OS Development is just too large a subject to get a 100% complete design (even mentally) before you start with implementing.

There are a number of tutorials to get you started: [1] [2] [3]

Of course, if you have anything specific, feel free to ask.

[AndrewAPrice]

Hi and welcome to the forums!

It's best if you start with a goal (you want a kernel that can load and run 2 programs) before actually building a more serious project otherwise you'll end up with bloatware.

i have gotten to grips with the main kernel loop partially, but that is covered enough to learn about in the wiki, the things i need to know about is in what i need to pre-assemble in order to support things latter.

Most kernel's don't actually have a loop (a few monolithic ones may have). Once initialised and running programs a kernel is pretty much event-based. The timer interrupt is an event that tells the kernel to switch to the next thread, a system call is an event that tells the kernel to do a particular task, etc. The closest thing to a 'loop' is halting the CPU when there is no awake thread to switch into.

for instance i will need to run executable files of an as yet undecided format. but i don't know what i would have to do to even cover the theory of this working in myos. i have not found anything on this topic so far.

There is a quite a bit on it, but it's spread out quite a bit. An executable file contains executable code with some sort of header saying what virtual address the code should be copied into. Loading a program is a matter of creating an address space, opening the file, loop through the headers/sections in the file, copying the executable code from each section into memory, then create a new thread at the program's 'entry point' (either hardcoded or specified in the file's header).

also something i cannot grasp completely is memory management in C/C++ (oh my os will be in C++) i have read many many things on the topic however i do not understand how without referring to the physical memory address like in a few assembly things i have seen, it can actually locate and read/write specific places in the memory.

You should look into paging. In most kernels each program exists within it's own 'virtual address space'. The virtual address space is described by the page directory/tables that you create. The kernel then loads the physical address of the page directory into the register CR2 (general applications cannot modify CR2 due to ring protection, but that's a more advance topic), then when code tries to access a memory location (e.g. via 'mov' in assembly) the CPU will translate every address through the page directory to get the physical address. This is done by the CPU automatically, transparently, and fast.

[midir]

I applaud your patience, reading documentation for over a month in advance. There comes a point though, it's better to get some code written down and running. You'll learn faster and enjoy it more if you've got a live kernel to play with.

Executable formats:
This is not something you need to worry about for *ages*. A kernel can be quite interesting simply running a shell that's part of it. By the time you're far enough to load additional programs, you'll know enough about executable formats to answer your own question. (And the exe format of the kernel itself is less of an issue.)

Memory management:
Hmmm. There could be many different things you're asking there. I *hope* I'm explaining the right thing, but all you need to start with are trivial stack and static memory.

Stack memory (stores function return addresses and function-local variables) can be as simple as one assembly instruction that sets the stack pointer (CPU register) to point to (the *top* of) some spot you pick for it. C++ doesn't need to know where this address is in advance (at compile time) since everything on the stack is relative to the stack pointer.

Statically allocated memory is where things are loaded at fixed addresses decided during compilation & linking. In C++ this applies to static variables inside a function, and all variables at file scope (whether attributed "static" or not -- at file scope, "static" means something quite different). You are correct to ask -- how can it actually locate stuff without referring to the physical addresses? Well when you compile a source file and its included headers ("translation unit") to an object file, it contains code, but all the addresses of fixed things like statically allocated data and function addresses are blank. The object file includes names of functions and variables ("symbols") that might be shared between files, and how big they are, but not where they go.

The linker's job is to decide where they all go, relative to some base address(es) you decide via the linker script or command line arguments. In a simple program one can let the compiler call the linker itself and choose default load addresses. In a kernel it's better to separate compilation/linking so you can get finer control. (Hint: If your linker has the feature to generate a map file, turn it on! It's extremely useful!)

Now provided your linker and boot loader agree on the base address things should be loaded at, all addresses of function calls and statically allocated data will just work.

Finally there's dynamically allocated memory (the likes of malloc, free, new, delete, etc.). These things allocate memory from (or put it back to) a "heap" (a data structure). For this to work you have to create a heap and give it some memory pages to work with (so that it has something to dish out on request). It's quite complex, but fortunately not vital for ages.

i did manage to track down an eBook version for "The Art of Assembly" apparently a great book to learn from however it uses some different compiler and syntax as compared to what i believe NASM uses

I don't know the book but the most important aspects of understanding assembly are common to all syntaxes and to many similar architectures, so don't worry. (Most of my understanding of assembly comes from not from x86 but the Super Nintendo.) But yes, every assembler has unique directives and syntax for the non-instruction things, and generic books are probably unhelpful for this. Here is the NASM manual: http://sourceforge.net/projects/nasm/fi ... p/download.
I think it has an x86 reference in one of the appendices too.

Hope this helps.

[qw]

i did manage to track down an eBook version for "The Art of Assembly" apparently a great book to learn from however it uses some different compiler and syntax as compared to what i believe NASM uses which is my preferred assembler so far just because it is very very common. the book is sub-titled the HLA Version and as such it has many differences that even the first chapter is enough to warn me off of it.

Yes, Randall Hyde wrote a great assembly tutorial book for MS-DOS programming. Unfortunately, in the Linux and Win32 books he is using his self invented High Level Assembly which hardly looks like assembly. You need his HLA Assembler to compile the examples.

[Colonel Kernel]

The kernel then loads the physical address of the page directory into the register CR2 (general applications cannot modify CR2 due to ring protection, but that's a more advance topic)

That's CR3... I don't mean to be picky, I just don't want the OP to get confused.

[Nekroze]

Thanks very much guys, doubly so to you midir.

thanks for all of the info and it has help in some understanding. i will procede to try and just make a kernel that can do some basic text output, then i will figure out keyboard input and make a basic CLI and move on from there, i think i still have a bit more reading to do on memory management, mainly of actual source code for simple versions just going to hunt some down today.

thanks again,
Nekroze