What does your build process look like?

[Combuster]

I'm not sure how you'd capture those sorts of things in a straightforward C syntax. You'd end up with something like:

Code: Select all

char *kernel_deps[]={"boot.o", "scheduler.o", "filesystem.o" /* etc */, NULL };

void buid_object(const char *filename){
    char new_filename_buffer[MAX_PATH];
    if(file_exists(change_file_extension(filename, "c", new_filename_buffer, MAX_PATH))) compile_with_gcc(new_filename_buffer , GCC_OPTIONS);
    if(file_exists(change_file_extension(filename, "s", new_filename_buffer, MAX_PATH))) assmemble_with_gas(new_filename_buffer, GAS_OPTIONS);
   /* ... */
}

Plus that passing strings in this way demonstrates exactly why you can't use C for this.

[max]

... murky scripts?

Because if you use for example C, you have to recompile it everytime you make changes, and that sucks. Also you want to have it platform-independent, what forces you to add nasty macros & recompile it on each platform everytime you make changes.

To the IDE point: Capri for example has an Eclipse plugin. So I can edit it in my preferred editor, have it platform independent, and never have to recompile it.

Pic related, Capri in it's natural environment

[embryo]

I'd be more comfortable with writing my scripts with Make than with C, which would feel really weird.

Isn't it just a matter of habit?

It's hard to adapt a normal programming language to serve the purpose of a specialized one.

What kind of specialization do you mean? Is it all about the same C code, which is called from scripts? Why not to call the code from C?

[embryo]

If your "build scripts" are written in a compiled language, what compiles these scripts? Sounds like a bit of a chicken-and-egg problem...

The problem was solved long ago. It's the same as writing compiler in C and next using it for compilation of C code.

A Makefile is just a list of targets with dependencies and rules for building said targets.

So, it's just a list of parameters which are sent to some C code, isn't it? And the problem is in the syntax of the parameter representation? But it is possible to represent all the parameters as a part of C code, why not to do it? Then instead of something like "$(KERNEL_DEPS)" we would have "dependencies" variable (or ok, the style C programmers like - KERNEL_DEPS).

Saying "all .c files can be used to build .o files through this command" is a simple Makefile rule.

What prevents us from writing the rule in form of a function call? Like this:

Code: Select all

buildObjectFiles(srcDirectory,outputDirectory);

Code: Select all

char *kernel_deps[]={"boot.o", "scheduler.o", "filesystem.o" /* etc */, NULL };

void buid_object(const char *filename){
    char new_filename_buffer[MAX_PATH];
    if(file_exists(change_file_extension(filename, "c", new_filename_buffer, MAX_PATH))) compile_with_gcc(new_filename_buffer , GCC_OPTIONS);
    if(file_exists(change_file_extension(filename, "s", new_filename_buffer, MAX_PATH))) assmemble_with_gas(new_filename_buffer, GAS_OPTIONS);
   /* ... */
}

void build_kernel(){
    for(int i=0; kernel_deps[i]; ++i){
         build_object(kernel_deps[i]);
    }
    link_with_ld(kernel_deps, LD_OPTIONS);
}

The code above can be much shorter. Like this:

Code: Select all

char *kernel_deps_dir="/some/directory/path";
buid_objects(kernel_deps_dir);

And all checks like "if file_exists()" are placed within the function. Also there is the build loop.

And that's without even trying to only rebuild what's actually changed (as Make does).

Such test is easily incorporated in the function above. It is programmed once and then can be used infinite number of times.

GCC even has features like the "-M" option to help with dependency tracking in Makefile-based build systems, which you'd have to do manually with another build system.

The reason for the -M option is the script addiction. Would there no scripts, then the option -M never happen to exist.

What do you mean by "very non standard OS code"?

At least the system image has non standard executable format. And systems like Java OS have much more non standard options.

Also, why are scripts any more "murky" than any other form of programming language?

Because of this:

Code: Select all

.c.o: %.c
    i686-someos-gcc $< -o $@ $(GCC_OPTIONS)

Of course, in the end a programmer gets used to such syntax, but at least there is obvious special sign overuse and command line syntax addiction.

How do you properly describe the dependencies of each target file?

Fortunately, I use Java. The Java compiler gets class path as an input and locates all dependencies automatically. Is it too hard to implement in C?

How do you ensure that you only (re)build what's actually necessary?

Currently I do not ensure this (my builds are not so frequent). But the idea of checking file modification time is still ready to be used in my code, it's just a matter of concentration on it.

[mallard]

If your "build scripts" are written in a compiled language, what compiles these scripts? Sounds like a bit of a chicken-and-egg problem...

The problem was solved long ago. It's the same as writing compiler in C and next using it for compilation of C code.

So, you build the first verison of the build system with another pre-existing build system and then "port" the build system over to itself. (At least that's how programming languages become "self-hosting"; you write an initial compiler in a pre-existing language and then port it to itself.) Does that mean there are two set of build system source code in your repository?

A Makefile is just a list of targets with dependencies and rules for building said targets.

So, it's just a list of parameters which are sent to some C code, isn't it?

All program code/scripts/whatever end up as CPU instructions and data somehow. That doesn't mean we should write everything in assembly (or directly in machine code with a hex editor). We have higher-level languages to reduce the cognitive burden, improve portability and separate concerns.

And the problem is in the syntax of the parameter representation? But it is possible to represent all the parameters as a part of C code, why not to do it? Then instead of something like "$(KERNEL_DEPS)" we would have "dependencies" variable (or ok, the style C programmers like - KERNEL_DEPS).

Saying "all .c files can be used to build .o files through this command" is a simple Makefile rule.

What prevents us from writing the rule in form of a function call? Like this:

Code: Select all

buildObjectFiles(srcDirectory,outputDirectory);

You've still got to define how the files are built, with which compiler, options, etc. Most OS kernels are written in multiple languages (i.e. assembly plus a higer level language) which means you need to have different rules for different kinds of source files. Your "buildObjectFiles" funtion would need to call something resembling my "build_object" function. (My "compile_with_gcc" type functions are imagined as thin wrappers over "execve"/"system".)

C isn't very good at this sort of thing. Things like string handling and variable-length arrays are awkward. It quickly becomes apparent that a DSL (domain-specific-language) would simplify things. Makefiles are one such DSL.

Code: Select all

/* Quoted code removed for brevity. */

The code above can be much shorter. Like this:

Code: Select all

char *kernel_deps_dir="/some/directory/path";
buid_objects(kernel_deps_dir);

And all checks like "if file_exists()" are placed within the function. Also there is the build loop.

Except that build_objects would be something like "list_files_in_directory" followed by a "for" loop containing "build_object". All you've done is slightly refactor the "build_kernel" function.

Also, not all build instructions are as simple as compiling a source file. What about getting the VCS commit ID into the code:

Code: Select all

buildid.c: dummy.file
	./buildid.sh

buildid.sh:

Code: Select all

COMMIT=$(git --no-pager log -1 --pretty=format:%h)
echo "char *kernel_buildid=\"$COMMIT\";" > buildid.c

Or building a HDD image with the latest builds of drivers/applications (too long to paste here).

And that's without even trying to only rebuild what's actually changed (as Make does).

Such test is easily incorporated in the function above. It is programmed once and then can be used infinite number of times.

GCC even has features like the "-M" option to help with dependency tracking in Makefile-based build systems, which you'd have to do manually with another build system.

The reason for the -M option is the script addiction. Would there no scripts, then the option -M never happen to exist.

The -M option exists because build systems need to know which other files (header files in C-style languages) are used to build a particular source file, so that when one of the headers is modified, the source is rebuilt. Without something like -M, the build system needs to parse the source files to get that information. It makes much more sense to have the compiler output this information, rather than have two language parsers (one in the compiler, the other in the build system). It also means that the build system doesn't need any special knowledge of the language being used. Any build system that supports "differential" builds, rather than just the dumb/slow build-everything-every-time method needs some way of doing this, if it's going to work with a language that supports something equivalent to "#include".

What do you mean by "very non standard OS code"?

At least the system image has non standard executable format. And systems like Java OS have much more non standard options.

Things like executable formats are generally little more than extra parameters to the linker (or maybe an extra objcopy step to convert to a flat binary) as far as the build system is concerned. There's nothing "special" about building an OS to a properly designed build system.

Also, why are scripts any more "murky" than any other form of programming language?

Because of this:

Code: Select all

.c.o: %.c
    i686-someos-gcc $< -o $@ $(GCC_OPTIONS)

Of course, in the end a programmer gets used to such syntax, but at least there is obvious special sign overuse and command line syntax addiction.

It's just code. If all you're familiar with is Java, than something declarative rather than imperative and using a different syntax might be a little difficult to follow at first, but the same could be said of someone familiar with Makefiles learning Java for the first time. Different languages have different purposes, different strengths and weaknesses. There is no "one language to rule them all" and there never will be.

How do you properly describe the dependencies of each target file?

Fortunately, I use Java. The Java compiler gets class path as an input and locates all dependencies automatically. Is it too hard to implement in C?

That's exactly what the -M option does.

How do you ensure that you only (re)build what's actually necessary?

Currently I do not ensure this (my builds are not so frequent). But the idea of checking file modification time is still ready to be used in my code, it's just a matter of concentration on it.

It's not as simple at just checking the modification times (see the discussion of the "-M" option above). It's my understanding that, in Java, if you use "import static" anywhere, you'll need to rebuild the importers every time the importee is modified, just like with "#include" in C-style languages (I know "#include" is a bit primitive, but that's a criticism of the language, not the build system.), so you need some way of extracting that sort of dependency information. You also need to ensure that dependents are built after their dependencies. Make (and every other build system; I believe Ant is common for Java) already has this logic built-in, why re-implement it (unless that's one of the goals of your project - in which case, it's far more work than you seem to realise)?

[Brendan]

Hi,

C isn't very good at this sort of thing. Things like string handling and variable-length arrays are awkward. It quickly becomes apparent that a DSL (domain-specific-language) would simplify things. Makefiles are one such DSL.

This is a common mistake. The hassle of doing (e.g.) string manipulation in C can easily be less hassle than learning a different language, maintaining code in a different language and making the project depend on additional tools to support that different language.


Cheers,

Brendan

[mallard]

The hassle of doing (e.g.) string manipulation in C can easily be less hassle than learning a different language, maintaining code in a different language and making the project depend on additional tools to support that different language.

If you want to be a somewhat "accomplished" programmer, you've got to learn more than one language. Once you've learnt a few, picking up the basics of another one only takes a few hours. Building a complete build system in C takes far more than a few hours. "Make" is a standard tool that any serious development system should have pre-installed. Even if you use some other build system, having a working knowledge of Makefiles (and shell scripting) is a very useful skill.

[bluemoon]

For me, I'd stick with Makefile (or other build system in the future), and not re-inventing my own.
As long as gmake does the job, I should focus on the OS development.

As for string manipulation in C, I think every OS developer should be capable of doing it, and do it very well.
but a reliable build system is much more than string manipulation, it involve years of testing which any programmer cannot shorten.

[Wajideu]

I've gotten into the habit of using autoconf/automake/libtool for most of my stuff recently. It's nowhere near as complicated as it was in the past. The best resource I found for getting started was this.

My only gripes are that it creates an autom4te.cache directory, the build process takes longer than make and doesn't cache the settings it detects, and you have you re-run autoreconf every time you add a new source directory. But that's a small price to pay for a cleaner project structure. I won't bash anyone for wanting to create their own build system though. I've tried working with alternatives like CMake and SCons, but I just couldn't get used to them. Neither was as flexible as autotools and CMake makes your source directory look like crap. Would be nice if there was an xml-based build system.


EDIT:
Here's a pastebin that puts into perspective what I had in mind for an xml-based build system. Pretty much the same functionality as autotools and make, but in a format that is simpler to parse and can be serialized.

[embryo]

mallard

It seems that most important reason of your script support is explained in the following:

Make ... already has this logic built-in, why re-implement it?

But if there was a C based build system and it had all Make's logic implemented, then it seems you would have no viable ground for objections.

And now about less important things (as I see it).

So, you build the first verison of the build system with another pre-existing build system and then "port" the build system over to itself. (At least that's how programming languages become "self-hosting"; you write an initial compiler in a pre-existing language and then port it to itself.) Does that mean there are two set of build system source code in your repository?

No. There should be just one set. The first set, which was used to build the second, now became a history, it is obsolete now and can be placed in some museum archives. And developers should care about the second set only.

We have higher-level languages to reduce the cognitive burden, improve portability and separate concerns.

Yes, but why we should use two languages instead of one? We have one preferred language and one legacy language for making builds. Why ever should we be bothered with the legacy language and it's quirks? Of course, we can learn the legacy stuff, we even can find many similarities with another legacy systems like Unix's command line language and other cryptic systems. But what useful we can get from the mastering of legacy (and as I see it - obsolete) systems? Why not to extract some useful knowledge without every day efforts of using the obsolete system?

You've still got to define how the files are built, with which compiler, options, etc. Most OS kernels are written in multiple languages (i.e. assembly plus a higer level language) which means you need to have different rules for different kinds of source files. Your "buildObjectFiles" funtion would need to call something resembling my "build_object" function. (My "compile_with_gcc" type functions are imagined as thin wrappers over "execve"/"system".)

The question "how" is very easy to answer to - just the same way it was done in Make's libraries. It is still possible to call C based libraries (that Make uses) from C based program. Just reuse all those "how to", but in a bit more preferred manner (by using preferred language). And all options can be passed to the library code as plain parameters.

But instead we have a layer of complexity over the preferred language. The layer requires us to get acknowledged with some new concepts, with new syntax, with new environment requirements, and after all - it binds us to the unix-style operating system just to be able to run it.

C isn't very good at this sort of thing. Things like string handling and variable-length arrays are awkward. It quickly becomes apparent that a DSL (domain-specific-language) would simplify things.

Even in C we can isolate some awkward stuff using library functions, for example. The functions should be written once and then all we have to do is just provide them with useful parameters. Is it so hard to achieve?

the build system doesn't need any special knowledge of the language being used.

If we use many languages, then it makes some sense. But do we really use many languages? Most often we use just one. And to reuse the build system functionality we can make some libraries and call them from different languages. Why not to go this way?

Things like executable formats are generally little more than extra parameters to the linker (or maybe an extra objcopy step to convert to a flat binary) as far as the build system is concerned.

And what about system image layout? First should be 512 byte of bootloader, then should go some file system stuff, then some files, one of the files should be our kernel in flat binary form (or may be not), somewhere else should be some system data structures and helper binaries. Is there some useful stuff in make scripts that places it all in a proper order? And if it is, then to what extent it is defined in libraries (to be called from preferred language)?

It's my understanding that, in Java, if you use "import static" anywhere, you'll need to rebuild the importers every time the importee is modified, just like with "#include" in C-style languages

If class names and signature of methods are not changed then it is possible to not touch the importee. I suppose something similar should be actual for C libraries, but instead of names (class or method) there should be relative addresses of functions and structures exposed to the external world.

But however, if it is about code dependencies graph and we need for some reason to decouple the graph creation from image build process then what prevents us from doing this using preferred language? It's again just a matter of preference and not a matter of some additional complexity. Even more - the complexity of build layer language is greater than the complexity of build system made using preferred language.

[Combuster]

But if there was a C based build system and it had all Make's logic implemented, then it seems you would have no viable ground for objections.

Actually, such a thing already exists... wait, why doesn't that argument make any sense now?

[embryo]

Would be nice if there was an xml-based build system.

It actually is. But for Java (I do not know about C analogs). However, underneath of the xml based system we can find the same calls to the Java libraries, that implement some standard stuff like compiling, cleaning, iterating, archiving and so on.

May be an xml solution can be seen as a more declarative way of expressing what we need to do, but what prevents us to write our preferred language code in a more declarative manner? May be here plays the different style of learning things. When a programmer learns C or Java he most often thinks about the program in a procedural style, but when he learns xml the style of thinking is changed to something more declarative. Then it is just a matter of our habits - we can't switch our mind to the declarative mode when writing C or Java programs, but it makes our programs less intuitive and less clear.

[embryo]

Actually, such a thing already exists... wait, why doesn't that argument make any sense now?

Is it really make no sense? Last update was in October 2013.

[Owen]

Actually, such a thing already exists... wait, why doesn't that argument make any sense now?

Is it really make no sense? Last update was in October 2013.

Perhaps this should be taken as indication that GNU Make is a piece of battle tested and reliable software, not needing regular changes?

[twrl]

Actually, such a thing already exists... wait, why doesn't that argument make any sense now?

Is it really make no sense? Last update was in October 2013.

I can think of plenty of things to criticize make for, but stability is not one of them. Stability is generally a good thing.