OSDev.org

Posted: **Wed Sep 03, 2014 12:46 am**

mallard wrote:I'm not sure how you'd capture those sorts of things in a straightforward C syntax. You'd end up with something like:

Code: Select all

char *kernel_deps[]={"boot.o", "scheduler.o", "filesystem.o" /* etc */, NULL };

void buid_object(const char *filename){
    char new_filename_buffer[MAX_PATH];
    if(file_exists(change_file_extension(filename, "c", new_filename_buffer, MAX_PATH))) compile_with_gcc(new_filename_buffer , GCC_OPTIONS);
    if(file_exists(change_file_extension(filename, "s", new_filename_buffer, MAX_PATH))) assmemble_with_gas(new_filename_buffer, GAS_OPTIONS);
   /* ... */
}

Plus that passing strings in this way demonstrates exactly why you can't use C for this.

Posted: **Wed Sep 03, 2014 1:00 am**

embryo wrote:... murky scripts?

Because if you use for example C, you have to recompile it everytime you make changes, and that sucks. Also you want to have it platform-independent, what forces you to add nasty macros & recompile it on each platform everytime you make changes.

To the IDE point: Capri for example has an Eclipse plugin. So I can edit it in my preferred editor, have it platform independent, and never have to recompile it.

Pic related, Capri in it's natural environment

Posted: **Wed Sep 03, 2014 1:05 am**

no92 wrote:I'd be more comfortable with writing my scripts with Make than with C, which would feel really weird.

Isn't it just a matter of habit?

no92 wrote:It's hard to adapt a normal programming language to serve the purpose of a specialized one.

What kind of specialization do you mean? Is it all about the same C code, which is called from scripts? Why not to call the code from C?

Posted: **Wed Sep 03, 2014 1:38 am**

mallard wrote:If your "build scripts" are written in a compiled language, what compiles these scripts? Sounds like a bit of a chicken-and-egg problem...

The problem was solved long ago. It's the same as writing compiler in C and next using it for compilation of C code.

mallard wrote:A Makefile is just a list of targets with dependencies and rules for building said targets.

So, it's just a list of parameters which are sent to some C code, isn't it? And the problem is in the syntax of the parameter representation? But it is possible to represent all the parameters as a part of C code, why not to do it? Then instead of something like "$(KERNEL_DEPS)" we would have "dependencies" variable (or ok, the style C programmers like - KERNEL_DEPS).

mallard wrote:Saying "all .c files can be used to build .o files through this command" is a simple Makefile rule.

What prevents us from writing the rule in form of a function call? Like this:

Code: Select all

buildObjectFiles(srcDirectory,outputDirectory);

mallard wrote:

Code: Select all

char *kernel_deps[]={"boot.o", "scheduler.o", "filesystem.o" /* etc */, NULL };

void buid_object(const char *filename){
    char new_filename_buffer[MAX_PATH];
    if(file_exists(change_file_extension(filename, "c", new_filename_buffer, MAX_PATH))) compile_with_gcc(new_filename_buffer , GCC_OPTIONS);
    if(file_exists(change_file_extension(filename, "s", new_filename_buffer, MAX_PATH))) assmemble_with_gas(new_filename_buffer, GAS_OPTIONS);
   /* ... */
}

void build_kernel(){
    for(int i=0; kernel_deps[i]; ++i){
         build_object(kernel_deps[i]);
    }
    link_with_ld(kernel_deps, LD_OPTIONS);
}

The code above can be much shorter. Like this:

Code: Select all

char *kernel_deps_dir="/some/directory/path";
buid_objects(kernel_deps_dir);

And all checks like "if file_exists()" are placed within the function. Also there is the build loop.

mallard wrote:And that's without even trying to only rebuild what's actually changed (as Make does).

Such test is easily incorporated in the function above. It is programmed once and then can be used infinite number of times.

mallard wrote:GCC even has features like the "-M" option to help with dependency tracking in Makefile-based build systems, which you'd have to do manually with another build system.

The reason for the -M option is the script addiction. Would there no scripts, then the option -M never happen to exist.

mallard wrote:What do you mean by "very non standard OS code"?

At least the system image has non standard executable format. And systems like Java OS have much more non standard options.

mallard wrote:Also, why are scripts any more "murky" than any other form of programming language?

Because of this:

Code: Select all

.c.o: %.c
    i686-someos-gcc $< -o $@ $(GCC_OPTIONS)

Of course, in the end a programmer gets used to such syntax, but at least there is obvious special sign overuse and command line syntax addiction.

mallard wrote:How do you properly describe the dependencies of each target file?

Fortunately, I use Java. The Java compiler gets class path as an input and locates all dependencies automatically. Is it too hard to implement in C?

mallard wrote:How do you ensure that you only (re)build what's actually necessary?

Currently I do not ensure this (my builds are not so frequent). But the idea of checking file modification time is still ready to be used in my code, it's just a matter of concentration on it.

Posted: **Wed Sep 03, 2014 2:49 am**

embryo wrote:
mallard wrote:If your "build scripts" are written in a compiled language, what compiles these scripts? Sounds like a bit of a chicken-and-egg problem...
The problem was solved long ago. It's the same as writing compiler in C and next using it for compilation of C code.

So, you build the first verison of the build system with another pre-existing build system and then "port" the build system over to itself. (At least that's how programming languages become "self-hosting"; you write an initial compiler in a pre-existing language and then port it to itself.) Does that mean there are two set of build system source code in your repository?

embryo wrote:
mallard wrote:A Makefile is just a list of targets with dependencies and rules for building said targets.
So, it's just a list of parameters which are sent to some C code, isn't it?

All program code/scripts/whatever end up as CPU instructions and data somehow. That doesn't mean we should write everything in assembly (or directly in machine code with a hex editor). We have higher-level languages to reduce the cognitive burden, improve portability and separate concerns.

embryo wrote: And the problem is in the syntax of the parameter representation? But it is possible to represent all the parameters as a part of C code, why not to do it? Then instead of something like "$(KERNEL_DEPS)" we would have "dependencies" variable (or ok, the style C programmers like - KERNEL_DEPS).
mallard wrote:Saying "all .c files can be used to build .o files through this command" is a simple Makefile rule.
What prevents us from writing the rule in form of a function call? Like this:
Code: Select all
buildObjectFiles(srcDirectory,outputDirectory);

You've still got to define how the files are built, with which compiler, options, etc. Most OS kernels are written in multiple languages (i.e. assembly plus a higer level language) which means you need to have different rules for different kinds of source files. Your "buildObjectFiles" funtion would need to call something resembling my "build_object" function. (My "compile_with_gcc" type functions are imagined as thin wrappers over "execve"/"system".)

C isn't very good at this sort of thing. Things like string handling and variable-length arrays are awkward. It quickly becomes apparent that a DSL (domain-specific-language) would simplify things. Makefiles are one such DSL.

embryo wrote:
mallard wrote:
Code: Select all
/* Quoted code removed for brevity. */
The code above can be much shorter. Like this:
Code: Select all
char *kernel_deps_dir="/some/directory/path";
buid_objects(kernel_deps_dir);
And all checks like "if file_exists()" are placed within the function. Also there is the build loop.

Except that build_objects would be something like "list_files_in_directory" followed by a "for" loop containing "build_object". All you've done is slightly refactor the "build_kernel" function.

Also, not all build instructions are as simple as compiling a source file. What about getting the VCS commit ID into the code:

Code: Select all

buildid.c: dummy.file
	./buildid.sh

buildid.sh:

Code: Select all

COMMIT=$(git --no-pager log -1 --pretty=format:%h)
echo "char *kernel_buildid=\"$COMMIT\";" > buildid.c

Or building a HDD image with the latest builds of drivers/applications (too long to paste here).

embryo wrote:
mallard wrote:And that's without even trying to only rebuild what's actually changed (as Make does).
Such test is easily incorporated in the function above. It is programmed once and then can be used infinite number of times.
mallard wrote:GCC even has features like the "-M" option to help with dependency tracking in Makefile-based build systems, which you'd have to do manually with another build system.
The reason for the -M option is the script addiction. Would there no scripts, then the option -M never happen to exist.

The -M option exists because build systems need to know which other files (header files in C-style languages) are used to build a particular source file, so that when one of the headers is modified, the source is rebuilt. Without something like -M, the build system needs to parse the source files to get that information. It makes much more sense to have the compiler output this information, rather than have two language parsers (one in the compiler, the other in the build system). It also means that the build system doesn't need any special knowledge of the language being used. Any build system that supports "differential" builds, rather than just the dumb/slow build-everything-every-time method needs some way of doing this, if it's going to work with a language that supports something equivalent to "#include".

embryo wrote:
mallard wrote:What do you mean by "very non standard OS code"?
At least the system image has non standard executable format. And systems like Java OS have much more non standard options.

Things like executable formats are generally little more than extra parameters to the linker (or maybe an extra objcopy step to convert to a flat binary) as far as the build system is concerned. There's nothing "special" about building an OS to a properly designed build system.

embryo wrote:
mallard wrote:Also, why are scripts any more "murky" than any other form of programming language?
Because of this:
Code: Select all
.c.o: %.c
    i686-someos-gcc $< -o $@ $(GCC_OPTIONS)
Of course, in the end a programmer gets used to such syntax, but at least there is obvious special sign overuse and command line syntax addiction.

It's just code. If all you're familiar with is Java, than something declarative rather than imperative and using a different syntax might be a little difficult to follow at first, but the same could be said of someone familiar with Makefiles learning Java for the first time. Different languages have different purposes, different strengths and weaknesses. There is no "one language to rule them all" and there never will be.

embryo wrote:
mallard wrote:How do you properly describe the dependencies of each target file?
Fortunately, I use Java. The Java compiler gets class path as an input and locates all dependencies automatically. Is it too hard to implement in C?

That's exactly what the -M option does.

embryo wrote:
mallard wrote:How do you ensure that you only (re)build what's actually necessary?
Currently I do not ensure this (my builds are not so frequent). But the idea of checking file modification time is still ready to be used in my code, it's just a matter of concentration on it.

It's not as simple at just checking the modification times (see the discussion of the "-M" option above). It's my understanding that, in Java, if you use "import static" anywhere, you'll need to rebuild the importers every time the importee is modified, just like with "#include" in C-style languages (I know "#include" is a bit primitive, but that's a criticism of the language, not the build system.), so you need some way of extracting that sort of dependency information. You also need to ensure that dependents are built after their dependencies. Make (and every other build system; I believe Ant is common for Java) already has this logic built-in, why re-implement it (unless that's one of the goals of your project - in which case, it's far more work than you seem to realise)?

Posted: **Wed Sep 03, 2014 3:57 am**

Hi,

mallard wrote:C isn't very good at this sort of thing. Things like string handling and variable-length arrays are awkward. It quickly becomes apparent that a DSL (domain-specific-language) would simplify things. Makefiles are one such DSL.

This is a common mistake. The hassle of doing (e.g.) string manipulation in C can easily be less hassle than learning a different language, maintaining code in a different language and making the project depend on additional tools to support that different language.

Cheers,

Brendan

Posted: **Wed Sep 03, 2014 4:17 am**

Brendan wrote:The hassle of doing (e.g.) string manipulation in C can easily be less hassle than learning a different language, maintaining code in a different language and making the project depend on additional tools to support that different language.

If you want to be a somewhat "accomplished" programmer, you've got to learn more than one language. Once you've learnt a few, picking up the basics of another one only takes a few hours. Building a complete build system in C takes far more than a few hours. "Make" is a standard tool that any serious development system should have pre-installed. Even if you use some other build system, having a working knowledge of Makefiles (and shell scripting) is a very useful skill.

Posted: **Wed Sep 03, 2014 5:46 am**

For me, I'd stick with Makefile (or other build system in the future), and not re-inventing my own.
As long as gmake does the job, I should focus on the OS development.

As for string manipulation in C, I think every OS developer should be capable of doing it, and do it very well.
but a reliable build system is much more than string manipulation, it involve years of testing which any programmer cannot shorten.

Posted: **Wed Sep 03, 2014 11:53 am**

I've gotten into the habit of using autoconf/automake/libtool for most of my stuff recently. It's nowhere near as complicated as it was in the past. The best resource I found for getting started was this.

My only gripes are that it creates an autom4te.cache directory, the build process takes longer than make and doesn't cache the settings it detects, and you have you re-run autoreconf every time you add a new source directory. But that's a small price to pay for a cleaner project structure. I won't bash anyone for wanting to create their own build system though. I've tried working with alternatives like CMake and SCons, but I just couldn't get used to them. Neither was as flexible as autotools and CMake makes your source directory look like crap. Would be nice if there was an xml-based build system.

EDIT:
Here's a pastebin that puts into perspective what I had in mind for an xml-based build system. Pretty much the same functionality as autotools and make, but in a format that is simpler to parse and can be serialized.

Posted: **Thu Sep 04, 2014 1:49 am**

mallard

It seems that most important reason of your script support is explained in the following:

mallard wrote:Make ... already has this logic built-in, why re-implement it?

But if there was a C based build system and it had all Make's logic implemented, then it seems you would have no viable ground for objections.

And now about less important things (as I see it).

mallard wrote:So, you build the first verison of the build system with another pre-existing build system and then "port" the build system over to itself. (At least that's how programming languages become "self-hosting"; you write an initial compiler in a pre-existing language and then port it to itself.) Does that mean there are two set of build system source code in your repository?

No. There should be just one set. The first set, which was used to build the second, now became a history, it is obsolete now and can be placed in some museum archives. And developers should care about the second set only.

mallard wrote:We have higher-level languages to reduce the cognitive burden, improve portability and separate concerns.

Yes, but why we should use two languages instead of one? We have one preferred language and one legacy language for making builds. Why ever should we be bothered with the legacy language and it's quirks? Of course, we can learn the legacy stuff, we even can find many similarities with another legacy systems like Unix's command line language and other cryptic systems. But what useful we can get from the mastering of legacy (and as I see it - obsolete) systems? Why not to extract some useful knowledge without every day efforts of using the obsolete system?

mallard wrote:You've still got to define how the files are built, with which compiler, options, etc. Most OS kernels are written in multiple languages (i.e. assembly plus a higer level language) which means you need to have different rules for different kinds of source files. Your "buildObjectFiles" funtion would need to call something resembling my "build_object" function. (My "compile_with_gcc" type functions are imagined as thin wrappers over "execve"/"system".)

The question "how" is very easy to answer to - just the same way it was done in Make's libraries. It is still possible to call C based libraries (that Make uses) from C based program. Just reuse all those "how to", but in a bit more preferred manner (by using preferred language). And all options can be passed to the library code as plain parameters.

But instead we have a layer of complexity over the preferred language. The layer requires us to get acknowledged with some new concepts, with new syntax, with new environment requirements, and after all - it binds us to the unix-style operating system just to be able to run it.

mallard wrote:C isn't very good at this sort of thing. Things like string handling and variable-length arrays are awkward. It quickly becomes apparent that a DSL (domain-specific-language) would simplify things.

Even in C we can isolate some awkward stuff using library functions, for example. The functions should be written once and then all we have to do is just provide them with useful parameters. Is it so hard to achieve?

mallard wrote:the build system doesn't need any special knowledge of the language being used.

If we use many languages, then it makes some sense. But do we really use many languages? Most often we use just one. And to reuse the build system functionality we can make some libraries and call them from different languages. Why not to go this way?

mallard wrote:Things like executable formats are generally little more than extra parameters to the linker (or maybe an extra objcopy step to convert to a flat binary) as far as the build system is concerned.

And what about system image layout? First should be 512 byte of bootloader, then should go some file system stuff, then some files, one of the files should be our kernel in flat binary form (or may be not), somewhere else should be some system data structures and helper binaries. Is there some useful stuff in make scripts that places it all in a proper order? And if it is, then to what extent it is defined in libraries (to be called from preferred language)?

mallard wrote:It's my understanding that, in Java, if you use "import static" anywhere, you'll need to rebuild the importers every time the importee is modified, just like with "#include" in C-style languages

If class names and signature of methods are not changed then it is possible to not touch the importee. I suppose something similar should be actual for C libraries, but instead of names (class or method) there should be relative addresses of functions and structures exposed to the external world.

But however, if it is about code dependencies graph and we need for some reason to decouple the graph creation from image build process then what prevents us from doing this using preferred language? It's again just a matter of preference and not a matter of some additional complexity. Even more - the complexity of build layer language is greater than the complexity of build system made using preferred language.

Posted: **Thu Sep 04, 2014 1:55 am**

But if there was a C based build system and it had all Make's logic implemented, then it seems you would have no viable ground for objections.

Actually, such a thing already exists... wait, why doesn't that argument make any sense now?

Posted: **Thu Sep 04, 2014 2:01 am**

DaemonR wrote:Would be nice if there was an xml-based build system.

It actually is. But for Java (I do not know about C analogs). However, underneath of the xml based system we can find the same calls to the Java libraries, that implement some standard stuff like compiling, cleaning, iterating, archiving and so on.

May be an xml solution can be seen as a more declarative way of expressing what we need to do, but what prevents us to write our preferred language code in a more declarative manner? May be here plays the different style of learning things. When a programmer learns C or Java he most often thinks about the program in a procedural style, but when he learns xml the style of thinking is changed to something more declarative. Then it is just a matter of our habits - we can't switch our mind to the declarative mode when writing C or Java programs, but it makes our programs less intuitive and less clear.

Posted: **Thu Sep 04, 2014 2:03 am**

Combuster wrote:Actually, such a thing already exists... wait, why doesn't that argument make any sense now?

Is it really make no sense? Last update was in October 2013.

Posted: **Thu Sep 04, 2014 11:38 am**

embryo wrote:
Combuster wrote:Actually, such a thing already exists... wait, why doesn't that argument make any sense now?
Is it really make no sense? Last update was in October 2013.

Perhaps this should be taken as indication that GNU Make is a piece of battle tested and reliable software, not needing regular changes?

Posted: **Thu Sep 04, 2014 11:52 am**

embryo wrote:
Combuster wrote:Actually, such a thing already exists... wait, why doesn't that argument make any sense now?
Is it really make no sense? Last update was in October 2013.

I can think of plenty of things to criticize make for, but stability is not one of them. Stability is generally a good thing.

OSDev.org

What does your build process look like?

Re: What does your build process look like?

Re: What does your build process look like?

Re: What does your build process look like?

Re: What does your build process look like?

Re: What does your build process look like?

Re: What does your build process look like?

Re: What does your build process look like?

Re: What does your build process look like?

Re: What does your build process look like?

Re: What does your build process look like?

Re: What does your build process look like?

Re: What does your build process look like?

Re: What does your build process look like?

Re: What does your build process look like?

Re: What does your build process look like?