Kernel Directory Structure

Muneer · Post by **Muneer** » Tue Sep 04, 2012 4:22 am

Hi,

It has been about 9 months since I last came here. I am planning to reenter Os development in my spare time. Now after having ported newlib,gcc,nasm,binutils I am now thinking of restructuring my kernel sources, especially the directory structure. This may involve rewrite of much of the kernel code. So I was thinking about how to laydown my kernel directory structure. I have come up with a few design of my own but my mind is fresh and perhaps I may be overlooking something or some directory design decisions. So what do you think of it and what recommendations do you have.

Code: Select all



Kernel directory structure
-------------------------------------

  src
  |
  +---+ x86
      |
      +----+ docs
      |
      |
      +----+ include
      |
      |
      +----+ kernel libs
      |    |
      |    +------+ malloc
      |    |
      |    +------+ free
      |    |
      |    +------+ printf et al.
      |
      +----+ bootloader
      |    |
      |    +------+ USB fat32 bootloader 
      |    |
      |    +------+ CDROM El-Torito bootloader
      |    
      |
      +----+ kernel
      |    |
      |    +------+ hardware specific functions
      |    |      |
      |    |      +------+ asm functions
      |    |      |
      |    |      +------+ c functions
      |    |
      |    +------+ hardware specific code functionality
      |    |      |
      |    |      +------+ GDT and IDT interface, interrupt and exception handling
      |    |      |
      |    |      +------+ paging management
      |    |      |
      |    |      +------+ pic and apic
      |    |      |
      |    |      +------+ process dispatcher and context switch management
      |    |      |
      |    |      +------+ multiprocessing
      |    |      |
      |    |      +------+ tss implementation
      |    |      |
      |    |      +------+ v86 implementation
      |    |      |
      |    |      +------+ PCI scanning
      |    |
      |    +------+ initialisation code
      |    |
      |    +------+ main kernel code
      |    |
      |    +------+ memory management interface
      |    |
      |    +------+ executable interface
      |    |      |
      |    |      +------+ elf loader
      |    |      |
      |    |      +------+ dll loading
      |    |
      |    +------+ module management
      |    |
      |    +------+ access control and permissions
      |    |
      |    +------+ network
      |    |      |
      |    |      +------+ TCP/IP UDP implementation
      |    |      |
      |    |      +------+ network management
      |    |
      |    +------+ resource management
      |    |
      |    +------+ scheduler
      |    |      |
      |    |      +------+ process
      |    |      |
      |    |      +------+ threads
      |    |      |
      |    |      +------+ multitasking
      |    |
      |    +------+ stdin stdout stderr interface
      |    |
      |    +------+ system calls
      |    |
      |    +------+ file interface
      |    |
      |    +------+ driver interface
      |    |      |
      |    |      +------+ native interface
      |    |      |
      |    |      +------+ EDI interface
      |    |
      |    +------+ inter process communication
      |    |      |
      |    |      +------+ pipes
      |    |      |
      |    |      +------+ sockets
      |    |      |
      |    |      +------+ signal
      |    |      |
      |    |      +------+ semaphores
      |    |      |
      |    |      +------+ mutex
      |    |      |
      |    |      +------+ shared memory
      |    |
      +----+ drivers
      |    |
      |    +----+ device drivers
      |    |    |
      |    |    +------+ hard disk
      |    |    |      |
      |    |    |      +------+ ATA driver
      |    |    |      |
      |    |    |      +------+ SATA driver
      |    |    |      |
      |    |    |      +------+ SCSI driver
      |    |    |
      |    |    +------+ network driver
      |    |    |      |
      |    |    |      +------+ 8023 driver
      |    |    |
      |    |    +------+ usb driver
      |    |    |      |
      |    |    |      +------+ EHCI driver
      |    |    |      |
      |    |    |      +------+ OHCI driver
      |    |    |      |
      |    |    |      +------+ UHCI driver
      |    |    |
      |    |    +------+ mouse driver
      |    |    |
      |    |    +------+ keyboard driver
      |    |
      |    +----+ other drivers
      |    |    |
      |    |    +------+ filesystem driver
      |    |    |      |
      |    |    |      +------+ vfs driver
      |    |    |      |
      |    |    |      +------+ fat32 driver
      |    |    |      |
      |    |    |      +------+ ext2 driver
      |    |    |      |
      |    |    |      +------+ ntfs driver
      |    |    |
      |    |    +------+ screen driver
      |    |    |      |
      |    |    |      +------+ vga driver
      |    |    |      |
      |    |    |      +------+ console driver
      |    |    |      |
      |    |    |      +------+ vesa driver
      |    |    |
      |    |    |
      |    |    |
      |    |    |
      |    |    |
      ^^^^^^^^^^^

Kevin · Post by **Kevin** » Tue Sep 04, 2012 4:24 am

Having an x86 directory, but none for platform independent code seems wrong.

Muneer · Post by **Muneer** » Tue Sep 04, 2012 4:31 am

Hi,

I meant the platform dependant codes to go in the sections
bootloader,
hardware specific functions,
hardware specific code functionality.

I did this because I thought (correct me if I am wrong) having a platform specific code on a seperate directory can isolate platform dependant code but this might lead to untidy arrangement of kernel code that depends on these platform dependant code.

Kevin · Post by **Kevin** » Tue Sep 04, 2012 4:39 am

What would the structure look like when you introduce a second architecture? Would you have the same thing duplicated in src/new-arch/?

Maybe you should just have the platform specific directory at a different level. In tyndur, there's kernel/src/... for independent code, and kernel/src/arch/i386/... for i386 specific code (as opposed to the x86/kernel/... in your draft).

Muneer · Post by **Muneer** » Tue Sep 04, 2012 5:01 am

When a new architecture is added, duplicating the code was what I was after. But I always knew that it was the wrong way. But I dont know somehow I felt it was good. But while reading your post I knew beforehand what you are trying to convey and after reading yours I somehow have a different feeling. Yeah I should restructure the directory to include

/src/all the independant code
/src/arch/x86
/src/arch/x64
and move
bootloader,
hardware specific functions,
hardware specific code functionality
to /src/arch/x86 ( as x86 is what I am presently interested in).

Any other suggestions and possible pitfalls.

Muneer · Post by **Muneer** » Tue Sep 04, 2012 5:18 am

I have changed to

Code: Select all

  +---+ docs
  |
  +---+ src
      |
      +----+ include
      |
      +----+ arch
      |    |
      |    +------+ x86
      |    |      |
      |    |      +------+ bootloader
      |    |	  |	 |
      |    |	  |	 +------+ USB fat32 bootloader 
      |    |      |	 |
      |    | 	  |	 +------+ CDROM El-Torito bootloader
      |    |      |
      |    |      +------+ hardware specific functions
      |    |      |	 |
      |    |      |	 +------+ asm functions
      |    |      |	 |
      |    |      |	 +------+ c functions
      |    |      |
      |    |      +------+ hardware specific code functionality
      |    |      	 |
      |    |      	 +------+ GDT and IDT interface, interrupt and exception handling
      |    |      	 |	 
      |    |      	 +------+ paging management
      |    |      	 |
      |    |      	 +------+ pic and apic
      |    |      	 |
      |    |      	 +------+ process dispatcher and context switch management
      |    |      	 |
      |    |      	 +------+ multiprocessing
      |    |      	 |
      |    |      	 +------+ tss implementation
      |    |      	 |
      |    |      	 +------+ v86 implementation
      |    |      	 |
      |    |      	 +------+ PCI scanning
      |	   |
      |    +------+ x64
      |	   |
      |    +------+ arm
      |
      |    
      |
      +----+ kernel libs
      |    |
      |    +------+ malloc
      |    |
      |    +------+ free
      |    |
      |    +------+ printf et al.
      |
      |    
      |
      +----+ kernel
      |    |
      |    +------+ initialisation code
      |    |
      |    +------+ main kernel code
      |    |
      |    +------+ memory management interface
      |    |
      |    +------+ executable interface
      |    |      |
      |    |      +------+ elf loader
      |    |      |
      |    |      +------+ dll loading
      |    |
      |    +------+ module management
      |    |
      |    +------+ access control and permissions
      |    |
      |    +------+ network
      |    |      |
      |    |      +------+ TCP/IP UDP implementation
      |    |      |
      |    |      +------+ network management
      |    |
      |    +------+ resource management
      |    |
      |    +------+ scheduler
      |    |      |
      |    |      +------+ process
      |    |      |
      |    |      +------+ threads
      |    |      |
      |    |      +------+ multitasking
      |    |
      |    +------+ stdin stdout stderr interface
      |    |
      |    +------+ system calls
      |    |
      |    +------+ file interface
      |    |
      |    +------+ driver interface
      |    |      |
      |    |      +------+ native interface
      |    |      |
      |    |      +------+ EDI interface
      |    |
      |    +------+ inter process communication
      |    |      |
      |    |      +------+ pipes
      |    |      |
      |    |      +------+ sockets
      |    |      |
      |    |      +------+ signal
      |    |      |
      |    |      +------+ semaphores
      |    |      |
      |    |      +------+ mutex
      |    |      |
      |    |      +------+ shared memory
      |    |
      +----+ drivers
      |    |
      |    +----+ device drivers
      |    |    |
      |    |    +------+ hard disk
      |    |    |      |
      |    |    |      +------+ ATA driver
      |    |    |      |
      |    |    |      +------+ SATA driver
      |    |    |      |
      |    |    |      +------+ SCSI driver
      |    |    |
      |    |    +------+ network driver
      |    |    |      |
      |    |    |      +------+ 8023 driver
      |    |    |
      |    |    +------+ usb driver
      |    |    |      |
      |    |    |      +------+ EHCI driver
      |    |    |      |
      |    |    |      +------+ OHCI driver
      |    |    |      |
      |    |    |      +------+ UHCI driver
      |    |    |
      |    |    +------+ mouse driver
      |    |    |
      |    |    +------+ keyboard driver
      |    |
      |    +----+ other drivers
      |    |    |
      |    |    +------+ filesystem driver
      |    |    |      |
      |    |    |      +------+ vfs driver
      |    |    |      |
      |    |    |      +------+ fat32 driver
      |    |    |      |
      |    |    |      +------+ ext2 driver
      |    |    |      |
      |    |    |      +------+ ntfs driver
      |    |    |
      |    |    +------+ screen driver
      |    |    |      |
      |    |    |      +------+ vga driver
      |    |    |      |
      |    |    |      +------+ console driver
      |    |    |      |
      |    |    |      +------+ vesa driver
      |    |    |
      |    |    |
      |    |    |
      |    |    |
      |    |    |
      ^^^^^^^^^^^

Jezze · Post by **Jezze** » Tue Sep 04, 2012 5:50 am

I have also recently been changing my directory layout back and forth. One thing that always annoys me is the placement of the include folder.

First I had the include folder in a seperate folder as you have, but then I thought, well the include folder should be a subfolder of whatever it is part of so i had:

lib/include/
lib/src/

kernel/include/
kernel/src/

arch/x86/include/
arch/x86/src/

Somehow it felt right to see each subfolder from the root as it's own project in a sense.... now the only problem was that if say in the kernel/arch/x86/src/cpu.c file I wan't to include both from kernel/include/kernel.h and lib/include/string.h I get a really terrible problem. The beginning of a file would have this:

#include <string.h>
#include <kernel.h>

Now I don't know by looking at the code from what include folder I got string.h and kernel.h.

The only way I've found to make this more clear is to have one include folder and put subfolders under them like this:

include/lib/string.h
include/kernel/kernel.h

which would become:

#include <lib/string.h>
#include <kernel/kernel.h>

But now the include folder lives it's own life on the side of everything else...

It's a catch 22.

Combuster · Post by **Combuster** » Tue Sep 04, 2012 6:41 am

Jezze wrote:One thing that always annoys me is the placement of the include folder.

I ended up settling on the following mechanism

Code: Select all

include/common
include/arch-x86
include/arch-m68k
include/x86-pc

for global stuff and local compinent/include subdirectories for internal header files. That way, you can easily have platform specifics override their generic counterparts, at architectural and machine specific levels.

It's a bit overkill though when only the pc target is really under scrutiny, but it helps with keeping the real work clean from all the helloworld fragments that happen to be out there at the same time.

Muneer · Post by **Muneer** » Tue Sep 04, 2012 8:09 am

I was really in doubt about what to do with the include directory. At first I even thought that no include directory should be there and to have all the required header files to be in the same directory as their source files and to include as #include "abc.h" instead of <abc.h>. But then seeing that there will be some global header files to be included from many places it gets really messy having to enter the full path.

So I currently thought to have an include folder that will contain the global header files . This include folder will again have subdirectories as needed and I must have to refer to them with relative paths.
so /include/abc.h will be included as #include <abc.h> and /include/arch/x86/abc.h will be included as #include <arch/x86/abc.h>.
also inorder to get rid of the long paths which can get tedious for typing and memorising and lookup, I am planning to write /include/locations.h> which will have (though I havent thought about how to implement them)
#define platform_specific arch

so

Inorder to include platform specific abc.h
#include <locations.h> // for almost all platform specific header files
#include <platform_specific/abc.h> // pass the required architecture as env variable while building.

Inorder to include platform independant abc.h
#include <abc.h>

And header files that need not be global can reside in the appropriate directory where the source file resides i.e for
/src/kernel/abc.c
we can have /src/kernel/abc.h
And if we want to include /src/kernel/abc.h then
we can specify it as #include "abc.h" instead of #include <abc.h> or #include <platform_specific/abc.h> from /src/kernel/abc.c.

I havent thought more about any possible problems with this approach.

bluemoon · Post by **bluemoon** » Tue Sep 04, 2012 11:18 am

My directory structure:

Code: Select all


   |-obsoleted                (obsoleted stuff)
   |-project                  (xcode project files)
   |---bluemoon.xcworkspace
   |---drivers.xcodeproj
   |---kernel.xcodeproj
   |---libgloss.xcodeproj
   |---testapp.xcodeproj
   |-reference                (self explained)
   |---amd
   |---elf
   |---intel
   |-screenshot               (self explained)
   |-test                     (test image and scripts)
   |-usr                      (base directory for self-hosting)
   |---bin                    (binary for arch independent)
   |-----i586                 (binary for i586)
   |-----x86_64               (binary for x86_64)
   |---include                (include file, platform independent)
   |-----i586                 (include file for i586, should not be included directly)
   |-----x86_64               (include file for x86_64, should not be included directly)
   |---lib                    (library file base directory, should be empty)
   |-----i586                 (library file for i586)
   |-----x86_64               (library file for x86_64)
   |---libc                   (base for compiled libc, --prefix)
   |-----i586-elf             (libc for i586)
   |-------include
   |-------lib
   |-----x86_64-elf           (libc for x86_64)
   |-------include
   |-------lib
   |---share                  (random stuff like README and logo)
   |---src
   |-----boot                 (stages of boot loaders)
   |-----drivers              (drivers, ./configure --arch=x86_64; make; make install to sync into test image)
   |-------bga
   |-------pci
   |-------rtl8139
   |-----kernel               (kernel, ./configure --arch=x86_64; make; make install to sync into test image)
   |-------arch
   |---------i586
   |---------x86_64
   |-------ddk                (interface exposed to drivers, drivers are compiled with -I$kernel/ddk)
   |-----libgloss             (libc glue, platform independent, ./configure --arch=x86_64; make; make install to put into /usr/lib/$arch)
   |-------i586               (libc glue for i586)
   |-------x86_64             (libc glue for x86_64)
   |-----mkdiskimage          (utility to make blank disk image)
   |-----mkinitrd             (utility to assemble initrd)
   |-----newlib               (newlib)
   |-------build-i586         (temp build directory)
   |-------build-x86_64       (temp build directory)
   |-------src                (new lib code from cvs)
   |-----testapp              (test app, platform independent)
   |---------i586             (test app, i586 specific / currently empty)
   |---------x86_64           (test app, x86_64 specific / currently empty)

All drivers must be arch independent - the ddk should warp all arch specific stuff.
All source folder works the same way: configure to select a "currently working" target arch, and make:

Code: Select all

./configure --arch=i586|x86_64
make && make install

Kevin · Post by **Kevin** » Wed Sep 05, 2012 2:06 am

Jezze wrote:Somehow it felt right to see each subfolder from the root as it's own project in a sense.... now the only problem was that if say in the kernel/arch/x86/src/cpu.c file I wan't to include both from kernel/include/kernel.h and lib/include/string.h I get a really terrible problem. The beginning of a file would have this:

#include <string.h>
#include <kernel.h>

Now I don't know by looking at the code from what include folder I got string.h and kernel.h.

But does it really matter? I've never had a problem with this and I like this certainly more than having to specify a path everywhere.

Our build system picks up include/ and lib/ directories on each level of the directory structure between the root and the source file. This way in each directory you "inherit" all of the headers and library object files from the parent directory, and you can add new local ones. So for a typical source file we have like three or four possible places for include files - but their position is rarely surprising, because you just know at which level it belongs.

bluemoon · Post by **bluemoon** » Wed Sep 05, 2012 12:40 pm

Jezze wrote:Somehow it felt right to see each subfolder from the root as it's own project in a sense.... now the only problem was that if say in the kernel/arch/x86/src/cpu.c file I wan't to include both from kernel/include/kernel.h and lib/include/string.h I get a really terrible problem. The beginning of a file would have this:

#include <string.h>
#include <kernel.h>

Now I don't know by looking at the code from what include folder I got string.h and kernel.h.

To address this issue, I have a clear defined area for includes:

libc is within default include path
- /usr/libc/$arch/include/ #include <stdio.h>
standard user-level include (3rd party packages) is within default include path
- /usr/include/ #include <zlib.h>
kernel DDK (interface for driver development), kernel_path is added to search path for building drivers
- $kernel_path/ddk/ #include <ddk/driver.h>

other kernel files are not exposed, so you will not see #include <kernel.h> except by the kernel itself.

neon · Post by **neon** » Wed Sep 05, 2012 2:13 pm

Hello,

So I currently thought to have an include folder that will contain the global header files . This include folder will again have subdirectories as needed and I must have to refer to them with relative paths.

Having a single include directory separate from the actual source is a good idea as it helps clearly define interfaces of different projects. I also encourage architecture-specific code to be in directories on a per-project bases though in order to avoid duplication of common non-architecture dependent code and to encourage a modular design. The architecture-specific code can be "compiled out" and "compiled in" based on build settings.

For example, our general directory structure looks like this:

Code: Select all

src
  --os
     --app
     --boot
     --ex
     --inc -- provided for an example
        --boot
           --library API
           --i386
              --i386 specific headers
     --lib  -- provided for an example
        --boot
           --sources
           --i386
              --sources
        --sdk
        ...etc...
     --util
  ...etc...

A common lib directory contains the source files for static libraries; the inc directory contains include files for different APIs (like the crt, udi and boot). By having a separate lib and include directory the build process becomes simpler as we can specify a base library path and include path for all parts of the system that use them. We don't have to worry about the i386 headers at all, only the sources which are only compiled in when _x86 is defined.

Platform specific (or firmware specific) headers are only used by the source that implements them. Due to this connection, they can include the headers directly -- for example, #include <boot/i386/vbe.h>. The sources implement an abstract firmware and platform independent (so-called) well-defined interface by non-architecture dependent sources or APIs so platform independent code do not need to include any platform-specific headers.

OSDev.org

Kernel Directory Structure

Kernel Directory Structure

Re: Kernel Directory Structure

Re: Kernel Directory Structure

Re: Kernel Directory Structure

Re: Kernel Directory Structure

Re: Kernel Directory Structure

Re: Kernel Directory Structure

Re: Kernel Directory Structure

Re: Kernel Directory Structure

Re: Kernel Directory Structure

Re: Kernel Directory Structure

Re: Kernel Directory Structure

Re: Kernel Directory Structure