Your OS design

Colonel Kernel · Post by **Colonel Kernel** » Sun Mar 04, 2007 10:48 pm

bubach wrote:even if i would try to reuse code with diffrent types of calling conventions, it's not really much of a problem in asm, just use the method any comments for that function tells you to.

It's great if someone took the time to write those comments, but it takes a human to write them and a human to read, understand, and apply them. You don't get any sort of automated help from your development tools. Like I said, it takes self-discipline. This is the point I was trying to make.

Brendan · Post by **Brendan** » Sun Mar 04, 2007 11:32 pm

Hi,

Colonel Kernel wrote:I guess what I'm trying to say is that writing reusable asm requires far more self-discipline. Any development process requiring that much self-discipline is inherently not scalable.

IMHO you can deliberately write reuseable, or you can deliberately write highly specialised code. Sometimes these are mutually exclusive and sometimes they aren't.

I always try to write highly specialised code. Sometimes I can cut & paste older code, but in general I can't. Usually (for me) there's larger changes, and it's these larger changes that cause the rewriting.

For an example, let me compare my last code to the current code - they're both mostly similar (modular) micro-kernel designs and the only main difference is that the newest code is designed to support headless systems and diskless systems. I'll focus on all the boot code because neither of them got much further than that (so far).

First, there's the boot loaders. The 1440 KB floppy boot loader started as the old code, but I added code to make the PC speaker beep when there's an error (for headless systems) and some code to detect if a VGA card is present (to automatically force a "headless boot" if there's no VGA). The Bochs ROM boot loader and the PXE/network boot loader are both entirely new code (nothing similar in the previous version).

Next is the "Boot Manager". About 80% of it is re-used code while the other 20% is was rewritten due to different physical memory usage and different keyboard/video methods (it used to directly access keyboard and video, but this was split into a seperate module).

There's also a "fake" boot manager that decompresses boot images, and a utility to compress the boot images (written in portable C code). The algorithm I was using to compress the boot images was too slow (especially for large boot images), so I rewrote this utility to use a different algorithm (now it uses more RAM, but compresses much faster). When I rewrote the compression utility I didn't care about making the output compatible with the old version, so when in the end I rewrote the OSs decompression module too.

Then there's the boot user interface code. There's 2 versions of this - one for VT100 over the serial port (for headless systems) and the other for keyboard/VGA. The VT100 code is entirely new. The keyboard/VGA code was mostly rewritten because I changed the kernel log format (stripped out the colour control codes and replaced them with bold and highlight codes, because VT100 can't reliably handle colour) and improved the way it works for better performance (instead of scrolling each time the cursor reaches the bottom of the screen, it pre-checks the new text and does one big scroll, then draws the new text without caring about hitting the bottom of the screen - end result is a lot less time spent scrolling when several lines are added at once).

The physical memory manager boot module is used to detect the system memory map and dynamically allocate pages before the kernel is started. The last version took into account page colour and NUMA domain, so that any code that allocates a page got the best possible physical page for it's purposes. Because of the complexity of it I had an "allocate a page" function in the physical memory manager that was added to the boot API, and the physical memory manager had to remain in memory (like a TSR, if anyone remembers DOS programming). I decided this wasn't such a good idea - the boot code was "32-bit only" and couldn't handle pages above 4 GB, it tended to impose it's data structures on the kernel's own physical memory manager (which was meant to be easily changeable), and because it was the only boot module that needed to stay in RAM after it had done it's thing it made the boot manager messier.

The new version uses a much simpler approach - one bit per page, without caring about page colours or NUMA domains. This means the physical memory manager used during boot is radically different (different function parameters, different data structures, etc), and meant changing all the code that uses the physical memory manager during boot (mainly code that sets up paging for the kernel) so it allocates it's own pages directly from the "free page bitmap" instead of using the boot API. It also means that the kernel will need to be able to optimise the physical pages it's using (something like a "kernel memory optimiser" running in the background that exchanges the physical pages the kernel is using for better physical pages).

The "CPU detection" module was mostly cut & paste, but I added code to detect "CPU forgery" and code to handle the errata for a lot more CPUs.

Then there's a module that handles NUMA domain information. This was mostly cut & paste, but the old code detected areas of the physical address space that are reserved for hot plug RAM. This got removed as it has nothing to do with NUMA domains and belongs in the physical memory manager boot module (it's unfortunate that the ACPI SRAT table contains a lot of NUMA information *and* the hot plug RAM information - the old code parsed this table once, while for the new code the same table is parsed twice).

Overall, I'd estimate about 20% of the code was reused, 40% was rewritten due to change, 10% was rewritten for no good reason, and the remaining 30% is entirely new. If I was working in C (or any other HLL) and if I had deliberately written the original code to be reusable (which I didn't), it really wouldn't have made much difference (I'd still rewrite a little bit for no good reason, that's just the way I do things).

Cheers,

Brendan

AndrewAPrice · Post by **AndrewAPrice** » Mon Mar 12, 2007 5:46 pm

What do people think about something like http://fabrice.bellard.free.fr/tcc/tccboot.html?

Dex · Post by **Dex** » Mon Mar 12, 2007 8:40 pm

Cool, people use tcc to code C program for DexOS, which seems to work fine.

AndrewAPrice · Post by **AndrewAPrice** » Tue Mar 13, 2007 2:42 am

No, I mean compiling the kernel at runtime.

Combuster · Post by **Combuster** » Tue Mar 13, 2007 3:54 am

IMHO its a waste of time. What tcc must do every boot can be done by gcc just once.

urxae · Post by **urxae** » Tue Mar 13, 2007 5:26 am

Combuster wrote:IMHO its a waste of time. What tcc must do every boot can be done by gcc just once.

I agree, though this might be useful on a livecd, if tcc performs CPU-specific optimizations (eg it uses new CPU features that can't be counted on to be present in all current CPUs, but happen to be present in the current one). A single gcc run can't do that, since it won't know which CPUs will be booting from the CD.

It could also be useful for Linux (kernel and/or driver) developers, at least for their test systems.

But for a normal install it doesn't make much sense, since the harddisk (which is presumably where the kernel is stored

) is mutable, so if you're going to compile your kernel specifically for that CPU you might as well do so once and store it on disk.

AndrewAPrice · Post by **AndrewAPrice** » Tue Mar 20, 2007 12:24 am

How do people layout their OS directory structure?

Mine is (my OS uses \ instead of the Unix /):
\System\
kernel.bin
Configure\
system
users.cfg
...
\Users\
Superuser\
Applications\
Source\
Include\
Kernel\
Drivers\
Servers\
Boot\
Libraries\
\Applications\
ProgramName\
\Devices\

When requesting a file from \Applications\ProgramName\ (such as a config file), my API says a program should first search in \Users\--\Applications\ProgramName\ for user-specific files.

ehird · Post by **ehird** » Wed Mar 21, 2007 9:34 am

The FS design I've got right now is pretty basic. It's POSIX-style, but incompatible.

Code: Select all

bin/
    -- Programs. Duh.
    local/
        -- Sysadmin-compiled programs. Put here for easy removal.
lib/
    -- Shared libraries.
    local/
        -- Same as /bin/local.
var/
    -- Data. System-wide configuration, log files, etc.
    log/
        -- Logs.
    cfg/
        -- Program configuration, except OS-wide. User-specific settings override these.
    src/
        -- Source of any kind! Sysadmins should dump them here before compiling.
    local/
        -- Guess. Of course, contains log,cfg,src.
usr/
    -- User directories.
    <usrname>/
        -- User's  home directory. Whatever they want. Optionally:
        var,lib,bin/...
            -- User-installed {anything}. For when they're not sysadmin.
boot/
    -- Grub, kernel, etc.
sys/
    -- System related stuff.
    var,lib,bin/...
        -- Core system binaries, libraries, and data (/sys/var/src contains system sources).
dev/
    -- Devices. :)

I plan stuff too far in advance

pcmattman · Post by **pcmattman** » Wed Mar 21, 2007 3:49 pm

All my FS is so far is:

Code: Select all

/boot
     -- grub stages, kernel and config file for grub
/
    -- the ( < 512 byte) file that stores user login info, storage for anything else

As you can see, it's really simple

. Mainly because ATM my file i/o drivers only read consecutive sectors (so a fragmented file means serious problems). I'm still trying to figure out how on earth to do the whole FAT12 'fat table' thing (properly).

Once I do get it:

Code: Select all

/
    -- root dir
/boot
    -- grub etc...
/sam
    -- user login data, unavailable for access from OS
/usr
    -- user folders
/bin
    -- shell programs
/help
    -- help system files
/tmp
    -- system temporary files
/dev
    -- wrappers for devices

My plan is to emulate a typical ext2 filesystem on a FAT disk. Once I get file i/o working (Somehow) I'll implement this.

Brynet-Inc · Post by **Brynet-Inc** » Wed Mar 21, 2007 7:42 pm

Good luck with permissions

pcmattman · Post by **pcmattman** » Wed Mar 21, 2007 8:26 pm

As I said, I'm a long way from getting it to work

Only thing between me and trying now is getting FAT16 onto a floppy...

Edit: And as they say, better to aim for the sky and hit a tree then aim for a tree and hit a rock.

ehird · Post by **ehird** » Fri Mar 23, 2007 11:35 am

Just don't take the __.__linux file route, for the love of all things holy. (*has visions of religion topic*)

pcmattman · Post by **pcmattman** » Sat Mar 24, 2007 9:22 pm

Hmmm.. Just got it working

Now, to figure out permissions

~ · Post by ~ » Mon Apr 02, 2007 11:15 am

I'm currently just taking projects from others to try to learn something, leave it well explained and extract the interesting bits from there. Hopefuly it will reach a point of didactic value for the ones that document and organize it (currently only me) and the ones that will consume that information.

So, I'm currently "designing" better methods and techniques to more easily crack down the complexities of understanding and so...

Maybe somebody will donate me so I can keep up with that work