How do you compile your OS?

[Candy]

@Candy

How far along are you in the design and code? Is this an open source thing I can contribute to?
I have a few ideas for the syntax which makes it a mix between Bash, C and Make that I'd like #included if you are looking for help.

I've got it running reliably. Thing is, I've also tried to port the codebase at work to it (which is a 60000-file codebase) including dependency ordering and that took a while, so I technically worked on it during company time. I still need to get in writing that I can opensource it (but I have already asked without putting it in writing, and my boss is OK with it).

Here's my OS' rulefile:

Code: Select all

arm-CXX-user=~/cross/bin/arm-none-eabi-g++ -std=c++11
arm-CXX-kernel=~/cross/bin/x86_64-pc-elf-g++ -std=c++11 -fno-exceptions -fno-rtti
arm-AR=~/cross/bin/arm-none-eabi-ar
arm-LD=~/cross/bin/arm-none-eabi-ld
x86-CXX-kernel=~/cross/bin/x86_64-pc-elf-g++ -std=c++11 -fno-exceptions -fno-rtti -mcmodel=large
x86-CXX-user=~/cross/bin/x86_64-pc-elf-g++ -std=c++11
x86-CC-kernel=~/cross/bin/x86_64-pc-elf-gcc -std=gnu99 -mcmodel=large
x86-CC-user=~/cross/bin/x86_64-pc-elf-gcc -std=gnu99
x86-AR=~/cross/bin/x86_64-pc-elf-ar
x86-LD=~/cross/bin/x86_64-pc-elf-ld
x86-ASM=~/cross/bin/yasm

host-AR=ar
host-CXX-user=g++ -std=c++11
host-CC-user=gcc -D_FILE_OFFSET_BITS=64 -std=gnu99

kernel-DEPS=libs/libcpp libs/libos
loader-DEPS=
tools/fusedriver-DEPS=libs/libstarfs
libs/libstarfs-DEPS=
libs/libcpp-DEPS=
libs/libos-DEPS=
libs/libz-DEPS=libs/libcpp
libs/libpng-DEPS=libs/libz
tools/mkfs-DEPS=libs/libstarfs
tools/fusedriver-ADDITIONAL-LIBS=-lfuse -lulockmgr
tools/mkfs-ADDITIONAL-LIBS=
COMPILER-c=CC
COMPILER-cpp=CXX

.*\.c.*>>(.*)/src/(.*)\.(c|cpp) => out/$(ARCH:x86 arm host)/$(CONFIG:user kernel)/obj/\1/\2.o
  $($(ARCH)-$(COMPILER-\3)-$(CONFIG)) $(subst (.*),-I@1/include,$(rep_subst (.*),@1 $(@1-DEPS),\1)) -c -o $@ $^

.*\.S>>(.*)\.(arm|x86)\.S => out/\2/$(CONFIG:user kernel)/obj/\1.o
  $(\2-CXX-$(CONFIG)) -c -o $@ $^

boot/structure.asm => out/x86/kernel/bin/bootsector.bin
  $(x86-ASM) -o $@ $^

#todo: remove x86 in next line and make generic. Also fix arm build.
out/x86/kernel/bin/.* out/host/bin/tools/mkfs.elf out/host/bin/tools/fusedriver.elf => disk.img
  ./makeimage.sh

out/([^/]*)/kernel/obj/kernel/.*\.o => out/\1/kernel/bin/kernel.elf out/\1/kernel/bin/kernel.elf.map
  $(\1-LD) -o $@ -T kernel/link/\1.ld $^ -z max-page-size=0x1000 -Map [email protected]

out/([^/]*)/([^/]*)/obj/(libs/[^/]*)/.*\.o => out/\1/\2/lib/\3.a
  $(\1-AR) rcs $@ $^

out/x86/kernel/obj/loader/.*\.o => out/x86/kernel/bin/loader.bin out/x86/kernel/bin/loader.bin.map
  $(x86-LD) -o $@ -T loader/link.ld $^ -Map [email protected]

out/host/user/obj/(tools/[^/]*)/.*\.o $(subst (.*),out/host/user/lib/@1.a,$(\1-DEPS)) => out/host/bin/\1.elf
  $(host-CC-user) -o $@ $^ $(\1-ADDITIONAL-LIBS)

disk.img => all
  touch all

So anything you see in there actually works as it looks like it would, and there's a few details that I'm not using in there that also work. I've also ported HippoMocks to it (another side project) and it now has 8 builds integrated in a single file, instead of 8 builds on a CI server with variables and makefiles. The rulefile for AtlantisOS replaces 15 makefiles and about 4x as much code, size-wise.

Combuster: All three sound like a great idea. WRT the syntax, do you have a concrete proposal?

@Candy

How far along are you in the design and code? Is this an open source thing I can contribute to?
I have a few ideas for the syntax which makes it a mix between Bash, C and Make that I'd like #included if you are looking for help.

I'm looking for people thinking along first and foremost - I'm honestly fresh out of ideas and used to its syntax, and I have trouble finding complex projects that need solutions to not fix it to the projects I can find. If you have syntax ideas I'm all ears - I dislike regexes as they are hard to read, but they're hard to beat for recognizability and compactness. It is already between Make and Bash, so what'd you want to add / change?

[b.zaar]

Yeah you're looking like you've pretty much finished and it looks like make with regex target & dependencies. It's nice and compact...

Not quite script enough for my ideas though.

[Candy]

Yeah you're looking like you've pretty much finished and it looks like make with regex target & dependencies. It's nice and compact...

Not quite script enough for my ideas though.

What do I need to change for you to be able to use it? What kind of things do you want to be able to do that you can't do with this? What in it makes it not "script enough"?

[b.zaar]

Well it looks like a makefile in your example as far as the shell commands go. I guess if I played with it for a few projects I'd notice the differences.

My ideal build tool would have a C feel to it in the shell build commands. I could make up an example build script but I can't guarantee it'd be the way anyone else would want to build their projects. Just my own wish list for a build system.

[Combuster]

Combuster: All three sound like a great idea. WRT the syntax, do you have a concrete proposal?

The problem with defining a syntax is that it needs to be logical, and that it should cooperate with the regex parsing - which I assume will be part of a library somewhere.

Since the '!' symbol is not a part of any sane filename, nor a regex standard, you could probably introduce a !macroname(...) syntax and have it parsed with minimal additional complexity on the part of the parser - if you can edit it which is one reason why I initially left out suggestions for syntax. Since they should be purely functional as far as I'm concerned, so you can just perform substitution until all (nested) macros are gone and you end up with the "base" syntax again.

As for priority, you want that to work as expected regardless of the order in which definitions are going to be defined. So you might want to prefix a rule with a numerical priority. Possibly set it to zero by default and allow negative priorities to hint that this should not be used unless there's no other way. Possibly add a warning as well when multiple rules of the same priority could have been used.

My ideal build tool would have a C feel to it in the shell build commands.

Many people are not actually used to think their code in the opposite way. Make basically feels like a instance of functional programming. If you're not used to see code written back-to-front, it will always feel awkward. However by restructuring the other way around you get free advantages you'd have serious troubles pulling off when it comes to C-style syntax. That way of thinking makes people uncomfortable about using make in the first place.

One annoying thing about Make I just thought of is that the moment you do more advanced shell things (like if's, do's and for's) you have to either stick it completely on one line rendering it mostly illegible, or you have to append those backslashes at the end of every line, which is error-prone. But it's not that much of an issue though.

[b.zaar]

My ideal build tool would have a C feel to it in the shell build commands.

Many people are not actually used to think their code in the opposite way. Make basically feels like a instance of functional programming. If you're not used to see code written back-to-front, it will always feel awkward. However by restructuring the other way around you get free advantages you'd have serious troubles pulling off when it comes to C-style syntax. That way of thinking makes people uncomfortable about using make in the first place.

I like the idea that make works in an event driven way. This file needs building then run these commands, this one doesn't so do nothing. It's a functional language where the return value is the target and the arguments are the dependencies.

My major annoyance is all variables are assigned at the time of reading the makefiles which makes it feel unnatural to any other language. This is a real problem if you are trying to use non-recursive make as recursive make keeps it's variables private. I'm sure somehow you could make all the target and dependencies fixed when reading the files but assign the variables for directories, sources or compiler and assembler options on a module by module basis.

One annoying thing about Make I just thought of is that the moment you do more advanced shell things (like if's, do's and for's) you have to either stick it completely on one line rendering it mostly illegible, or you have to append those backslashes at the end of every line, which is error-prone. But it's not that much of an issue though.

This is something that could be fixed if everything is first treated as an internal command and you must shell out to run commands. This would hopefully limit the use on the shell as a scripting engine and only use it for build commands.

I'd like to work on a make project but not starting from the GNU source, so if there's another open source version out there I'd be interested in having a look. Most of my ideas work inside of the makefile structure so if you don't use the extensions then the makefile goes through the regular build process of make.

[elderK]

I use Make with a little "iterative" makefile framework I wrote.

I'm looking forward to seeing what use I can find for the newly added integration with Guile Scheme.

[Combuster]

Any luck with your employer yet, Candy?