What does your build process look like?

[max]

Heyho!

As there are quite a lot approaches to building a kernel, applications and whatever is necessary to form your OS, I'd like to open this thread so that everyone can write a little about their build process. I think it's quite interesting to see what other people do and what steps their process involves, and maybe we can share ideas and discuss the methods.

So I'm going to start out!

About the design: My kernel is a microkernel, and everything runs in userspace, launched by now from the ramdisk. I'm building the parts using my buildtool Capri. As toolchain I'm using a OS-specific toolchain, the setup instructions can be found here (thats how you build it yourself, I'll add downloadable binaries one day).

To the build process itself, my projects folder structures (a little simplified) looks like this:

Code: Select all

/GhostKernel          (the kernel itself)
	/src-api           (global stuff that the API needs, like systemcall definition headers)
	/src-kernel        (actual kernel sources)
	/src-loader        (loader sources)
	/src-shared        (stuff thats shared between kernel and loader)
	/ramdisk
	/iso               (contains GRUB files)
/GhostAPI             (kernel API wrapper userspace library)
/GhostLibrary         (high-level userspace library)
/GhostApps            (repository with all applications)
	/GhostApp-XX1
	/GhostApp-XX2

So each project has a buildscript with the tasks "clean" and "all". This is the process of building the entire OS:

Code: Select all

1. API library
  a) compiles it's sources, including /GhostKernel/src-api and creates the archive "libghostapi.a"
  b) compiles the crts
  c) copies all headers from /GhostKernel/src-api and all its own headers to the toolchains "include" directory
  d) copies "libghostapi.a" and the crt*.o's to the toolchains "lib" directory
2. GhostLibrary, this is just a normal build process creating the archive "libghostlib.a"
3. The applications are built, resulting binaries are copied to the /GhostKernel/ramdisk folder
4. The kernel is built:
  a) compiles the sources from src-loader, including src-shared, creating the loader binary
  b) compiles the sources from src-kernel, including src-shared and src-api, creating the kernel binary
  c) builds the ramdisk
  d) packs the kernel, loader and ramdisk to an .iso with GRUB
5. Optionally, launches the .iso in QEMU

So when building the entire thing (I'm using Eclipse), I can just hit "clean" and "build" on each project and everything is done. I'm curious to see what steps you do to come from your sources to an .iso! Feel free to leave comments, too

Greets,
Max

[bluemoon]

My build process is:

Code: Select all

# Checkout & build newlib, do it once
cd usr/src/newlib
./checkout.sh
./build32.sh
./build64.sh
cd ../../..

# create test image, it also build necessary tools in host environment.
cd test
./mkimage.sh
cd ..

# mount the image to host
...

# build kernel and install to image
cd usr/src/libgloss
./configure --arch=x86_64
make && make install
cd ../kernel
./configure --arch=x86_64
make && make install
cd ../../..

# build all drivers
cd usr/src/driver
./configure --arch=x86_64
make
cd ../../../

# build test app
cd usr/src/testapp
./configure --arch=x86_64
make

# pack drivers and test app (and other files) into initrd
./mkinitrd initrd.txt

# Launch VM
cd test
./run64.sh bga

There is also an "make all" to automate the whole process (assume the image is mounted).

[neon]

Hello,

Our source tree and build environment has not changed for awhile.

/env/ - Build environment
/os/app - Applications
/os/boot - Boot loader
/os/drv - Drivers
/os/ex - Executive
/os/inc - Includes
/os/lib - Static libraries
/os/slib - Shared libraries
/os/util - Utilities

Strict naming conventions and directory paths are used to define system utilities and applications. Every component is defined within its own folder. For example, /os/boot/nboot/ contains the source files for the boot loader. /os/lib/nbl/ contains the source for the boot library. /os/util contains most of the software currently, it is where all the system utilities are at. For example, /os/util/ndbg and /os/util/nbuild. Although we plan to rewrite the memory manager, it will be an external server so will be at /os/drv/mm/ separate from the kernel. In addition, we stick by the standard 8.3 naming convention and all system utilities and programs start with a lowercase n.

We use make files but most of the work is handled by the build environment, nbuild, and nmake. We just run nbuild on the root directory to target a specific architecture and it builds anything that has changed since the last build and in the correct order to avoid dependency issues. Dependency issues are resolved by a special dirs file in all directories to control what libraries and utilities to build and in what order. The system is guaranteed to build everything in the correct order with any project specific configuration.

So to build just involved running nbuild on the source tree. The make files for each program is greatly simplified. For example, this is the makefile for nbuild.

Code: Select all

!undef DBGMSG
SYSTEM = SYSTEM_NEPTUNE
TARGETNAME = nbuild.exe

SOURCES = build.c \
	dirs.c \
	file.c \
	list.c \
	status.c

!include $(NEPTUNE_ENV)\makefile.def

$(NEPTUNE_ENV)\makefile.def is part of the build environment and hides the tool chain specific details.

[twrl]

I'm using the new gn tool from Chromium, so my build system is generally

Code: Select all

tools/gn gen out
ninja -C out

There are some shell scripts to tidy things up, clean out file trees, automate launching it in qemu and things like that, but basically everything revolves around those two commands.

[mallard]

For the toolchain, I have a script that downloads the tarballs, extracts them, applies my patches (by overwriting files with my copies) and builds them.

For the OS, the good old-fashioned recursive make, combined with a few shell scripts. Three top-level directories; user, modules and kernel, built in that order (as the kernel's initfs drags some of the modules and userspace programs into the kernel via xxd).

I've also got a disk image that's extracted from a .gz (if it doesn't exist) and has files copied into it with mtools. mtools is really bad for this sort of thing, since it's heavily config-driven (you can't specify many options on the command line), so I have to write out a config file (which violates the principle that the build should be self-contained) and is designed for floppy images, so I have to add "@@1M" on the image file name.

Anybody know any half-decent tools that allow me to create, partition, format and copy files to/from a disk image without requiring root (no build process should require root)? I used to use parted to partition/format, but formatting is a deprecated feature and it crashes on some of my systems. I'm half tempted just to write a better tool for the job (I could easily adapt the MBR and FAT code from the OS).

[no92]

I think we're finally getting into an age in which not everybody uses nasm, gcc and make to build a UNIX-clone. The OSDev forums seem to have evolved in the past year (I started reading forum threads way before I actually joined). I think it's a good sign.

So, back to the thread: I'm currently developing my own build system. I kindof liked make, but it often gets to messy, mainly because of 3 reasons: you need shell snippets everywhere, it's partly ugly syntax and the way you have to use variables. As there's no other build system that implements my ideas, so I decided to write my own. The one coming closest is probably max's Capri, but it's more an OO-language than a build system, at least in my opinion.

Before starting with my own build system, I used shell scripts and Makefiles (which relied on shell scripts partly, too).

[Wajideu]

I use autoconf/automake, and my build tree is:

Code: Select all

/Gidjit
 +- /Apps
 |   '- /Terminal
 +- /OS
 |   +- /Drivers
 |   '- /Kernel
 |       '- /Boot
 '- /Root

The /Root directory is what is written to the disk. It somewhat mirrors the structure of the filesystem layout I came up with, where /OS gets installed in /Root/System and /Apps gets installed in /Roots/Users/Shared/Applications.

Code: Select all

/Root
 +- /System
 |   +- /Devices
 |   +- /Drivers
 |   +- /Kernel
 |   |   '- /Boot
 |   '- /Locale
 |       '- /en-US
 '- /Users
     '- /Shared
         +- /Applications
         |   '- /Appname
         |       +- /Assets
         |       '- /Data
         |           +- /Cache
         |           '- /Temp
         +- /Documents
         +- /Music
         +- /Pictures
         '- /Videos

The idea behind this is that user mode applications will only have write privileges to their own /Assets and /Data directories, and the Users/*/Documents,Music,Pictures,Videos directory. Basically I'm just trying to prevent the problem Windows has where ignoranus programmers can't make up their mind to store their application data in one of the 3 AppData directories, ProgramData, the working directory, or some other random-@$$ place; then leave a bunch of crap on your pc when they're removed.

[no92]

Well, for that, I like the OS X attempt: it stores all assets inside the application, which is, in fact, only a directory. And there's an additional place in the Filesystem in the "~/Library/Application Support/<app name>" to store data, when needed. There are strict guidelines by Apple which prevent most apps from creating a mess in the filesystem, as it in M$-Windows.

[Wajideu]

Well, for that, I like the OS X attempt: it stores all assets inside the application, which is, in fact, only a directory. And there's an additional place in the Filesystem in the "~/Library/Application Support/<app name>" to store data, when needed. There are strict guidelines by Apple which prevent most apps from creating a mess in the filesystem, as it in M$-Windows.

The sandboxing that Mac uses is pretty much what I'm going for; except it's slightly more privileged in that apps at least have read-access to files outside of their own directory. Which is useful for things like importing bookmarks or contacts from other applications.

[embryo]

I'm currently developing my own build system.

It's interesting to understand the pros and cons of a custom build system. C programmers use make files, Java programmers use Ant scripts and so on, but why they all do not use the language they are programing in? Just because of some libraries related to make? But what prevents a programmer from using those libraries from C? Or is it scripting appeal when you just hit any text editor and get your script ready? But there is always some preferable editor, then why not to use your preferred IDE instead? And beside of scripting pros there is the problem of very non standard OS code, when it is required to build a "not so simple" program for image build process - why not to use your preferred language instead of murky scripts?

And of course, my system image is built by the program, written in Java (no scripts at all).

[Dju]

Hello everyone.

Well, my build process takes place in a Windows environment.

My workspace use a Visual Studio solution, containing several projects :

• CRT : Basic stuff to handle MSVC compiler
  HAL : Basic stuff to deal with CPU, PIC, PIT, ...
  System : My own framework, to replace stdlib. It contains the headers to define my user mode SDK.
  Kernel : The kernel and device drivers
  Log : A C# project to display serial output
  API : A C# project to generate the SDK layer. It generates a file in System to map C++ objects to interrupts, and a file in Kernel to handle these interrupts and call the actual code.
  Test : A test usermode application, using my SDK

When I click or the Run button in Visual Studio, here is what happens :

• Build
  • The CRT and HAL are built
    The SDK is generated
    The Kernel and System part are built
    The test application is built, using CRT and System
    The C# tools are built
  Deploy & run
  • The virtual machine is stopped, if running
    Generated files are moved in the virtual hard drive
    The Log tool, to read serial ouput is started
    The virtual machine is started

The virtual machine is VMware Player. It bundles a GDB stub I use to debug low level behaviors.

I keep a physical image of the virtual hard drive in my solution folder. Since the content of this folder is copied in the virtual hard drive, I just have to add files in this folder to deploy them in the virtual machine.
Here is the structure of my file system :

Code: Select all

/System
 |- /Storage  // This is where I mount external storage
 |- /Boot  // It contains Bootloader and Menu files, from Grub
 |- /Fonts  // Contains fonts used in my UI (actually Open and Consolas with my own format)
 |- /Images  // Contains graphic resources used in my UI (for theming, also with my own format)
 |- Kernel  // The actual Kernel executable
 |- Kernel.db  // The SQLite database containing devices data (for PCI support)
Apps
 |- Test  // The test app, to test usermode and my SDK
 |- Editor  // Future structure I'll use
     |- Name
Users  // Future structure I'll use
 |- Name
     |- Documents
     |- Pictures
     |- Videos
     |- Music
     |- Downloads

I know some people do not like Visual Studio, and generally osdeving in a Windows environment, but I find it very convenient to develop. Concerning MSVC compatibility, I have absolutely no external dependencies. I developped my CRT from scratch, now featuring RTTI support.

Actually, I use PE file format for my Kernel and my Apps, and Serial output to debug.
The most common iteration takes about 2-3 seconds to run, fast enough to develop quickly.

[no92]

@embry: I asked myself the question too. I came to the conclusion that normal programming languages aren't suitable for things like build system scripts. I'd be more comfortable with writing my scripts with Make than with C, which would feel really weird. It's hard to adapt a normal programming language to serve the purpose of a specialized one. Just my POV, tell me if you don't agree with me.

[mallard]

why they all do not use the language they are programing in?

If your "build scripts" are written in a compiled language, what compiles these scripts? Sounds like a bit of a chicken-and-egg problem...

Just because of some libraries related to make?

Not really. A Makefile is just a list of targets with dependencies and rules for building said targets. Saying "all .c files can be used to build .o files through this command" is a simple Makefile rule. In C. Saying "kernel.bin requires these .o files and is linked by this command" is another simple rule. In a Makefile, you might have something like:

Code: Select all

KERNEL_DEPS :=  boot.o scheduler.o filesystem.o #etc...

kernel.bin: $(KERNEL_DEPS)
    i686-someos-ld $(KERNEL_DEPS) -o $@ $(LD_OPTIONS)

.c.o: %.c
    i686-someos-gcc $< -o $@ $(GCC_OPTIONS)

.s.o: %.s
    i686-someos-gas $< -o $@ $(GAS_OPTIONS)

I'm not sure how you'd capture those sorts of things in a straightforward C syntax. You'd end up with something like:

Code: Select all

char *kernel_deps[]={"boot.o", "scheduler.o", "filesystem.o" /* etc */, NULL };

void buid_object(const char *filename){
    char new_filename_buffer[MAX_PATH];
    if(file_exists(change_file_extension(filename, "c", new_filename_buffer, MAX_PATH))) compile_with_gcc(new_filename_buffer , GCC_OPTIONS);
    if(file_exists(change_file_extension(filename, "s", new_filename_buffer, MAX_PATH))) assmemble_with_gas(new_filename_buffer, GAS_OPTIONS);
   /* ... */
}

void build_kernel(){
    for(int i=0; kernel_deps[i]; ++i){
         build_object(kernel_deps[i]);
    }
    link_with_ld(kernel_deps, LD_OPTIONS);
}

And that's without even trying to only rebuild what's actually changed (as Make does). It quickly gets pretty unwieldy. GCC even has features like the "-M" option to help with dependency tracking in Makefile-based build systems, which you'd have to do manually with another build system.

But what prevents a programmer from using those libraries from C? Or is it scripting appeal when you just hit any text editor and get your script ready? But there is always some preferable editor, then why not to use your preferred IDE instead? And beside of scripting pros there is the problem of very non standard OS code, when it is required to build a "not so simple" program for image build process - why not to use your preferred language instead of murky scripts?

What do you mean by "very non standard OS code"? Apart from passing some unusual options to the compiler (things like -nostdlib), there's nothing fundamentally different to building an OS kernel to building any other computer program. Also, why are scripts any more "murky" than any other form of programming language?

And of course, my system image is built by the program, written in Java (no scripts at all).

So how do you deal with the "build system needs to be compiled by the build system" chicken-and-egg? How do you properly describe the dependencies of each target file? How do you ensure that you only (re)build what's actually necessary?

[twrl]

I used ninja directly as my build system for a while, which I was generally very happy with, but it became unwieldy as I had more objects. I didn't like cmake, so I looked at various other things. I did start tinkering with using coffeescript for a custom build system, basically just extending cake to have some handy functions for building large native apps, until I hit on gn.

It's interesting seeing everyone's source tree layout. I have

Code: Select all

build
disks
 +- esp
docs
external
modules
 +- system
 |   +- acpica
 |   +- hal-x64-uefi
 |   +- kernel
 +- servers
 |   +- sysbase  [i]Root process server[/i]
 |   +- namespace  [i]Namespace service (kinda like vfs)[/i]
 |   +- vgui  [i]Vector graphics subsystem[/i]
 |   +- commonhw  [i]User mode drivers for common hardware[/i]
 +- userlibs
 |   +- libpk. [i]kernel interface and minimal libc[/i]
 |   +- libpk+  [i]ditto, with C++ goodies[/i]
 +- apps
     +- shell
out
tools

Or at least, I'm in the middle of a big refactor but that's what I've almost got.

[Wajideu]

I like what dju has going on. The one thing that deterred me a bit from using MSVS was that I felt like I needed something a bit more recursive. I should probably rethink my project structure.