How do you compile your OS?

[Candy]

Open question.

I'm working on a build tool - think make / cmake / ... replacement - that tries to be powerful enough to easily express very complex projects. In looking for examples I thought of this place and figured - OSes are pretty much the most complex build setup you could want.

So, after that opener question, what kind of things do you want from a build tool / would make your OS much easier to compile / more reliable / faster ?

[edit] This is not a hollow question. I've got it recompiling my OS, fully including building a bootable 4GB disk image with my own FS on it, in 0.75 seconds. On a laptop.

[AndrewAPrice]

Make is overkill for my simple needs. I have a bunch of shell scripts.

kernel/build.sh - builds the kernel
build.sh - builds everything and creates a bootable CD image
run.sh - builds everything, creates an image, boots it in QEMU

I've thought about using Perl scripts instead. Only rebuild an object file if it's source file changes, or rebuild everything if a header file changes.

[thomasloven]

Make is overkill for my simple needs.

There's no task too trivial for make.
I usually do things like:

Code: Select all

$ mkdir hello
$ cd hello
$ echo '#include <stdio.h>\n int main() { printf("Hello, world!\\n"); return 0;}' > main.c
$ make main
cc    main.c    -o main
$ ./main
Hello, world!
$

Not to get off topic though:
I use a combination of make and shell scripts to compile my os.
I use make rules to check if my toolchain exists, and if it doesn't a shell script downloads and compiles it. This part could definitely be sped up...
Perhaps a built in package management, which checks common repositories for third party libraries based on #include tags found in the source?

My biggest problem with make is the weird function syntax and the forced hard tabs.
Just a thought: Perhaps a different approach could be a collection of tools that could plug in to shell scripts which fills the space between shell and make - such as dependency tree generation?

[embryo]

what kind of things do you want from a build tool / would make your OS much easier to compile / more reliable / faster ?

I want one environment for osdeving and building. Because it is really convenient. Next I want it to be simple and flexible. The simple and flexible system usually has some good architecture, so I wand a good architecture. And finally I want it to be mature. It means I want a set of options rich enough to be able to accomplish really tricky tasks.

Same environment is usually achieved by having many translations of the same algorithms. Good architecture is not as simple as many translations, but it can serve for good for long time and for many translations. The maturity is partially achieved by selecting most usable operations of different build systems to represent a really good set of options. And next part of maturity is just time in service. The longer the system serves - the bigger set of options it is asked to include and at some point it can contain the set that satisfies a lot of users.

[max]

I've actually developed a build script lately, named Capri. It's similar to C and Java allows you to define projects that contain tasks & dependencies, import other scripts and use native functions to access stuff. This is a small example that builds all sources in src that end with c or cpp to bin.

Code: Select all

project MyKernel {

    CXX = "i686-ghost-g++";
    CFLAGS = "-whatever";

    system windows {
        CXX = "i686-sucky-cygwin-gcc";
    }

    task all depends check {
        Compilation.executeForChanged("src", {".c", ".cpp"}, "{CXX} {CFLAGS} $file$");
    }

    task check {
        if(!File.isFolder("bin")) {
            clean();
        }
    }

    task clean {
        File.cleanFolder("bin");
    }

}

You can write huge scripts that do a lot of stuff, or simple small things like this one that use the support script.
It's in the alpha/beta phase, and not very many people where interested the last time so I had no real motivation to publish it yet. ^^

[b.zaar]

I'd like something mixed between a makefile and a shell script, more like a structured function.

Code: Select all

out all(kernel.sys, initrd)

out kernel.sys(mbhead.o, kernel.o)
{
    // linking instructions. All variables are private to this block  

}

out initrd(ata.drv, fbdev.drv, ext2fs.drv)
{
    // Shell commands and more private variables
}

All external commands are handled in a single shell so you could cd to a directory and run the rest of the function from there. Also most commands should be handled by the build program so that it's not relying on a certain shell being installed, kind of like an internal bash environment.

Well max posted before I finished but yeah capri is similar but i don't think is quite the same.

[max]

I'd like something mixed between a makefile and a shell script, more like a structured function.

Code: Select all

out all(kernel.sys, initrd)

out kernel.sys(mbhead.o, kernel.o)
{
    // linking instructions. All variables are private to this block  

}

out initrd(ata.drv, fbdev.drv, ext2fs.drv)
{
    // Shell commands and more private variables
}

All external commands are handled in a single shell so you could cd to a directory and run the rest of the function from there. Also most commands should be handled by the build program so that it's not relying on a certain shell being installed, kind of like an internal bash environment.

Well max posted before I finished but yeah capri is similar but i don't think is quite the same.

You can do this with capri. You can use the functions "System.changeDirectory" and "System.getCurrentDirectory" to navigate through folders, determine which tool to use for which system and stuff, and simple call them from within the dir you changed to by using "System.execute"

[Candy]

I've actually developed a build script lately, named Capri. It's similar to C and Java allows you to define projects that contain tasks & dependencies, import other scripts and use native functions to access stuff. This is a small example that builds all sources in src that end with c or cpp to bin.

Code: Select all

project MyKernel {

    CXX = "i686-ghost-g++";
    CFLAGS = "-whatever";

    system windows {
        CXX = "i686-sucky-cygwin-gcc";
    }

    task all depends check {
        Compilation.executeForChanged("src", {".c", ".cpp"}, "{CXX} {CFLAGS} $file$");
    }

    task check {
        if(!File.isFolder("bin")) {
            clean();
        }
    }

    task clean {
        File.cleanFolder("bin");
    }

}

You can write huge scripts that do a lot of stuff, or simple small things like this one that use the support script.
It's in the alpha/beta phase, and not very many people where interested the last time so I had no real motivation to publish it yet. ^^

I think I can write that in my current tool as

Code: Select all

CXX = "i686-ghost-g++"
CFLAGS = "-whatever"

(.*)/src/.*\.(?:c|cpp) => \1.exe
  $(CXX) $(CFLAGS) -o $@ $^

At least, if you compile without first making object files. If you do use the intermediate step (not sure from your script):

Code: Select all

CXX = "i686-ghost-g++"
CFLAGS = "-whatever"

(.*)/src/(.*)\.(?:c|cpp) => \1/obj/\2.o
  $(CXX) $(CFLAGS) -o $@ -c $^
(.*)/obj/.*\.o => \1.exe
  $(CXX) $(CFLAGS) -o $@ $^

I'd like something mixed between a makefile and a shell script, more like a structured function.

...

All external commands are handled in a single shell so you could cd to a directory and run the rest of the function from there. Also most commands should be handled by the build program so that it's not relying on a certain shell being installed, kind of like an internal bash environment.

Well max posted before I finished but yeah capri is similar but i don't think is quite the same.

Sounds like a good idea. Right now I'm only allowing oneliners, but of arbitrary complexity. It would be fairly easy to extend this to a full-blown scripting language / bash script and would simplify my own build even more (and I thought it was simple already!).

My biggest problem with make is the weird function syntax and the forced hard tabs.
Just a thought: Perhaps a different approach could be a collection of tools that could plug in to shell scripts which fills the space between shell and make - such as dependency tree generation?

I've dropped the hard tabs (because it's a pretty stupid idea). For dependency logic I've included a function to do transitive closures over a replacement. It's pretty specific though, so I'm not sure about its general applicability. It does a repetitive replacement of a pattern by a replacement pattern and then filters out the last unique entry of each. This gets you an ordered list of direct and indirect dependencies (if no cycles). Would that help you in your problem?

[sortie]

I make the distinction between the high level logic build system and the build system of the operating system components.

For the operating system components such as the kernel, libc, utilities and so on, I use Make. This is a good tool for the job as files are obviously derived from each other and can be built in parallel. Therhave is good room for configuration by the advanced user through standard and ccomponent-local environment variables.

I use GNU make and I hate it. It is too poorly written (please seriously read its source codexif you use it) and is too compatible with awful systems, meaning its source code is dumbed down in troublesome manners. I am currently as a low priority project writing my own Make that is much cleaner, is compatible with POSIX, and will new useful features the way I want them.

The top level build system is responsible for building the entire operating system and the ports. This is high level logic and is better suited for shell scripts than Make, as files are not obviously derived from each other. I implement this with make rather than shell scripts for historical reasons. This layer is meant to set up a basic system root, then cross compile each component and port into it. It should then add configuration to the system root and make a bootable image in accordance with the user's wishes.

Your building system should be designed such that building on your OS itself is the primary and simplest platform. Cross compiling is always a secondary platform, make it simple and no simpler. For instance, my makefiles always build the code for the local operating system, even if it doesn't make sense, as many of my programs don't work on Linux. The cross user must explicitly request cross compilation.

[Combuster]

I use makefiles, with a fair share of repetition because there output directory is an hierarchy of its own, and there needs to be a mkdir for every directory which all the compile rules for that directory have to depend on.

What's also very useful is a way to reliably pick one possible source for an object file. As in, pick the most applicable of { generic/tri.c, x86_mmx/tri.asm, neon/tri.asm } to make tri.o, and a somewhat more-legible-than-perl way of specifying all .c files turned to .o files, perhaps macro-ish, would be very welcome.

Though, I won't quit using make if the other utility can't do massive parallism.

[max]

@Candy ah okay, your tool is more about writing smaller scripts but allowing complexity. Mine is more about readibility and comfort

Though, I won't quit using make if the other utility can't do massive parallism.

capri can! you can use "concurrent <statement>" to start anything in an extra thread. for example

Code: Select all

threadId = concurrent massiveCalculation(251, 836.51);
// do other stuff in the meantime
// once you need an result:
result = join threadId;

hehe now ill stop advertising i promise

[seuti]

My project is only a few source files big so I just use a single Makefile.

[Candy]

For the operating system components such as the kernel, libc, utilities and so on, I use Make. This is a good tool for the job as files are obviously derived from each other and can be built in parallel. Therhave is good room for configuration by the advanced user through standard and ccomponent-local environment variables.

I use GNU make and I hate it. It is too poorly written (please seriously read its source codexif you use it) and is too compatible with awful systems, meaning its source code is dumbed down in troublesome manners. I am currently as a low priority project writing my own Make that is much cleaner, is compatible with POSIX, and will new useful features the way I want them.

The top level build system is responsible for building the entire operating system and the ports. This is high level logic and is better suited for shell scripts than Make, as files are not obviously derived from each other. I implement this with make rather than shell scripts for historical reasons. This layer is meant to set up a basic system root, then cross compile each component and port into it. It should then add configuration to the system root and make a bootable image in accordance with the user's wishes.

Your building system should be designed such that building on your OS itself is the primary and simplest platform. Cross compiling is always a secondary platform, make it simple and no simpler. For instance, my makefiles always build the code for the local operating system, even if it doesn't make sense, as many of my programs don't work on Linux. The cross user must explicitly request cross compilation.

I've set it up to allow you to target all at the same time. You can specify which options exist in the build line and then use it to determine how to build a certain target. Cross compilation for N targets should be trivial... example below with Combuster as example.

I use makefiles, with a fair share of repetition because there output directory is an hierarchy of its own, and there needs to be a mkdir for every directory which all the compile rules for that directory have to depend on.

That's one of the initial design goals - it's intended for *big* projects that have N output formats/styles in a hierarchy of output folders. Heck, one of the biggest projects I'm testing the design on has 5 major build flavors, with 2 subflavors and 3 sub-sub-flavors, for a total of 31 build configurations (those 5*3*2, plus your host). It's got a format specifically for doing the "compile this source to N outputs based on this" that Make seems to be lacking (IMO):

Code: Select all

(.*)\.cpp => Out/$(OS:Linux CombusterOS)/$(BUILDTYPE:Debug Release)/$(MODE:kernel user)/\1.o
  $($(OS)-CXX) $(CCFLAGS-$(BUILDTYPE)-$(MODE)) -c -o $@ $^

What's also very useful is a way to reliably pick one possible source for an object file. As in, pick the most applicable of { generic/tri.c, x86_mmx/tri.asm, neon/tri.asm } to make tri.o, and a somewhat more-legible-than-perl way of specifying all .c files turned to .o files, perhaps macro-ish, would be very welcome.

More legible than perl is pretty hard, if you still want the power a regex can provide. If you have concrete suggestions though, let me know

Selecting the most applicable is pretty hard to do. Switching between x86_mmx and neon is easy, if they match your build configs - just compile the asm file to the relevant output dir's obj location and it'll be taken into account for linking.

Though, I won't quit using make if the other utility can't do massive parallism.

-j<number of cores> is implied, but you can tune it down if you want to.

@Candy ah okay, your tool is more about writing smaller scripts but allowing complexity. Mine is more about readibility and comfort

Though, I won't quit using make if the other utility can't do massive parallism.

capri can! you can use "concurrent <statement>" to start anything in an extra thread. for example

Code: Select all

threadId = concurrent massiveCalculation(251, 836.51);
// do other stuff in the meantime
// once you need an result:
result = join threadId;

hehe now ill stop advertising i promise

Feel free to keep advertising Max, I'm not here to sell a tool, I'm here to find the problem space and multiple solutions. My tool is about having the ability to specify what you want to build efficiently (without needless repetition - its base concept is Don't Repeat Yourself). Heck, for the big project I referred to near the start the rule file is already around 600 lines, mostly due to illogical include paths and dependency information. I really don't need that to balloon into a few thousand of lines.

[Combuster]

I noticed I should have proofread that post. It was horrible.

More legible than perl is pretty hard, if you still want the power a regex can provide.

Akin to that difference between using C++ and manually writing out all exception checking. The point was, several cases like all files with extension X, or convert extension X in the input to Y in the output are extremely common patterns, and you might want to have defaults for that.
The reason I suggested macros as well is because regexes are typically write-once-read-never constructs. If you have some sort of a macro system you can make composing regexes easier, and you can convey the meaning in the process, making stuff actually maintainable. You probably noticed I'm not a fan of extensive use of perl regular expressions.

Selecting the most applicable is pretty hard to do. Switching between x86_mmx and neon is easy, if they match your build configs - just compile the asm file to the relevant output dir's obj location and it'll be taken into account for linking.

I think the most appropriate example here is that you have a C library with platform-specific parts. (in my case there are 5 of those constructs). You can just compile the thing with the just-return-an-error stubs and have a C library configured to go on every platform, but you want to override those key parts with more meaningful implementations whenever you can support it. In similar vain, you might want to rewrite those default (soft)float math.h functions using the relevant assembly oneliners for the architecture in question.
And of course, you don't want to write out each particular combination by hand.

If you want that to work consistently, you will need to have a way to deal with the case where there's a rule to make X from Y and a rule to make X from Z, and that you can define strict precedence to one in particular.

-j<number of cores> is implied, but you can tune it down if you want to.

Just sanity checking there. Max' explicit system is one particular reason for me not to use it. Probably never because it's a severe regression compared to Make. On a sidenote though, cores+1 is often recommended in order to have spare work for a core to do while it's otherwise designated processes is performing I/O instead.

[b.zaar]

@Candy

How far along are you in the design and code? Is this an open source thing I can contribute to?
I have a few ideas for the syntax which makes it a mix between Bash, C and Make that I'd like #included if you are looking for help.