Page 1 of 2

A new build system?

Posted: Thu Jun 18, 2009 11:00 am
by earlz
Well, recently I've had some "fun" trying to get my makefiles to work with both GNU make and BSD make(which there are actually many versions for) and I just got generally fed up with the make syntax.

So, I am wanting to make a new build system, even if it's so I can compile my own stuff easy(if it gets that far lol)
My requirements are this: No dependencies(only LibC), Made in either C or C++(probably C), lightweight: as in, I don't have to have configuration files everywhere(either on my system or in the project directory), Powerful enough to handle big projects easily, Powerful enough that it won't one day have a million extensions making things incompatible(as in, powerful enough that extensions aren't needed in most cases).

The closest thing I've seen to such a system is Cmake, but I consider it to be a little messy and plus it has dependencies. You must use make to use the makefiles it generates and such.

One big problem with make(I am referring to POSIX make as it is the standard GNU and BSD both share) is it's difficult(if not impossible) to just straight compile all files of a certain type in one directory to another directory. such as src/*.c to obj/*.o Cause no one actually wants to manually add to a makefile each time you make a new source file.

Well, I came up with this syntax of how things can work. I know it isn't quite consistent yet and such, but tell me what you think. (Also, I'm aiming for easy to parse and one pass/interpreted line-by-line)

Code: Select all

compile src/test1.c to obj/test1.o if out of date:

compile($src,$trg){
	exec([$CC $CFLAGS -c $src -o $trg]);
};

default(){
	if older("obj/test1.o","src/test1.c") { //if test1.o is older than test1.c
		compile("src/test1.c","obj/test1.o");
	} else {
		exec("echo \"Target is up to date\" ");
	}
}

..continuation to also link obj/test1.o to test1

link($objs,$trg){
	exec([$CC $LDFLAGS -o $trg $objs]);
}

default(){
	string $objs;
	if older("obj/test1.o","src/test1.c") { //if test1.o is older than test1.c
		compile("src/test1.c","obj/test1.o");
	} else {
		exec("echo \"Target is up to date\" ");
	}
	$objs=concat($objs,"obj/test1.o ");
	$objs=concat($objs,"objs/other_objects.o ");
	link($objs,"bin/test1");
	
}

complicated example to compile all .c files in src/ to .o files in objs/
and then link them all(only if out of date though) (using compile and link from before)
for example, it contains files: main.c test.c foo.c no.f


default(){
	string $cfiles;
	string $objfiles;
	cd("src/");
	$cfiles=file_glob("*.c"); //this is so we get main.c not src/main.c
	cd("..");
	for_each($cfiles," ") = $this_file{ //for each element in cfiles seperated by " ". Put current element in this_file
		string $this_obj;
		$this_obj=glob_replace($this_file,"*.c","*.o"); //replace main.c with main.o, *.c with *.o
		$this_obj=concat("objs/",$this_obj); //make it so it's objs/main.o
		$this_file=concat("src/",$this_file); //make it so it's src/main.c
		if(older($this_obj,$this_file)){
			compile($this_file,$this_obj);
		}
		$objfiles=concat($objfiles,$this_obj);
	}
	link($objfiles,"bin/a.out");
}

Note, that there will be an optional common actions include for things like compile() and link()

And I don't wish for it to be so powerful you could make an application for, but I would like something possibly dynamic. Also, I'm not sure I want to support anything but string variables.

edit:
My main goal with this though is to make it so things are much easier to maintain and much easier to understand to an outsider.(as in, anyone who knows C, Java, C++, etc have an idea of what is going on in this)

I want to take the black magic out of build systems.

Re: A new build system?

Posted: Thu Jun 18, 2009 11:22 am
by Solar
Declaring a dislike with the make's (declarative) syntax, and proposing a procedural (if - else) replacement, is a pattern I recognize well. I've been at that point, too: Good enough at programming to feel the need for something like make, but not yet fluent enough in declarative syntax to become comfortable with make itself.

So you sit down and come up with a language of your own. Sooner or later you get to the point where you have to implement the language you drafted, which requires you to define a syntax, and a parser. That's when you realize that syntax definitions are, by nature, declarative... like Makefiles. So you learn to cope with declarative syntaxes... and then you'll realize how clunky your procedural make-replacement suddenly looks. Especially when, by the time you get there, every computer around is massively multithreading, which a declarative system can make good use of, while a procedural one gets into trouble. ;-)

Re: A new build system?

Posted: Thu Jun 18, 2009 11:47 am
by earlz
Solar wrote:Declaring a dislike with the make's (declarative) syntax, and proposing a procedural (if - else) replacement, is a pattern I recognize well. I've been at that point, too: Good enough at programming to feel the need for something like make, but not yet fluent enough in declarative syntax to become comfortable with make itself.

So you sit down and come up with a language of your own. Sooner or later you get to the point where you have to implement the language you drafted, which requires you to define a syntax, and a parser. That's when you realize that syntax definitions are, by nature, declarative... like Makefiles. So you learn to cope with declarative syntaxes... and then you'll realize how clunky your procedural make-replacement suddenly looks. Especially when, by the time you get there, every computer around is massively multithreading, which a declarative system can make good use of, while a procedural one gets into trouble. ;-)
How exactly can make make use of multiple threads without very explicit support(involving several nasty hacks)? Like there is the -P option that lets you have multiple processes running jobs, but really trying to control those so that it won't try to link your object files before they are done compiling is a full time job.I would see it much easier to have a simple fork() function in my system or something similar than to try to parallelize makefiles

Re: A new build system?

Posted: Thu Jun 18, 2009 11:57 am
by Solar
earlz wrote:How exactly can make make use of multiple threads without very explicit support(involving several nasty hacks)? Like there is the -P option that lets you have multiple processes running jobs, but really trying to control those so that it won't try to link your object files before they are done compiling is a full time job.
No it isn't. Make dependencies already define a complete acyclic dependency graph that can trivially be split among threads, provided that the Makefile author didn't follow that age-old, brain-dead practice of writing recursive makefiles. ;-)

Re: A new build system?

Posted: Thu Jun 18, 2009 2:27 pm
by bewing
Except that the practical considerations of 1) compilation output to the screen, 2) handling stops on build errors, and 3) detecting completion of the build of dependencies -- force make to be a singlethreaded system ... which completely negates any theoretical multithreadedness.

I don't really mind declarative syntax -- but I think big strides could be made in making it <b>readable</b>.

Re: A new build system?

Posted: Thu Jun 18, 2009 2:29 pm
by earlz
Solar wrote:
earlz wrote:How exactly can make make use of multiple threads without very explicit support(involving several nasty hacks)? Like there is the -P option that lets you have multiple processes running jobs, but really trying to control those so that it won't try to link your object files before they are done compiling is a full time job.
No it isn't. Make dependencies already define a complete acyclic dependency graph that can trivially be split among threads, provided that the Makefile author didn't follow that age-old, brain-dead practice of writing recursive makefiles. ;-)
Yes, maybe this is true but I believe my syntax could be multithreaded easy enough

Ok, what about somethign like this:

Code: Select all

Everything for this is quite easy.. Ok. More complicated example(this might take me a while to write lol)
We have a kernel. It can be compiled for i386, ppc, or amd64. It all depends on an environmental variable ARCH
Source files are in src/
src has all system independent files in it. Also, it has i386, ppc, and amd64 directories which have the
arch. dependent code. For each target, they all need the system independent and they need only their
arch. dependent files. include/ has some common header files to all of them. src/ has some header files as
does all of the arch. directories. 
When compiled, the files need to not interfere with each other, so they must go in obj/ obj/i386 etc
When linked to output file, they need to go to bin/kernel_ARCH

Let's get hacking.

#include <common_funcs.jfi> //source of compile and link from above

import $ARCH; //import from environmental variable.
string $objs;
string $srcs;
string $headers; //declare all of these global variables.

get_arch_sources(){ //get arch srcs and headers
	cd("src");
	$srcs=concat($srcs,file_glob([$ARCH/*.c]));
	$headers=concat($headers,file_glob([$ARCH/*.h]));
	cd("..");
}

get_common_sources(){ //get common srcs and headers
	cd("src");
	$srcs=concat($srcs,file_glob("*.c"));
	$headers=concat($headers,file_glob("*.h"));
	cd("..");
}

compile_async($src,$trg){
   exec_async([$CC $CFLAGS -c $src -o $trg]); //executes a file asynchronously. We don't wait on the process to finish.
};

default(){
	bool $need_compile;
	$need_compile=false;
	if(notequal($ARCH,"i386") | notequal($ARCH,"amd64") | notequal($ARCH,"ppc")){
		exec([echo "$ARCH is not a supported target"]);
	}
	get_common_sources();
	get_arch_sources();
	for_each($srcs," ") = $this_file{ //for each element in cfiles separated by " ". Put current element in this_file
		string $this_obj;
		$this_obj=glob_replace($this_file,"*.c","*.o"); //replace main.c with main.o, *.c with *.o
		$this_obj=concat("objs/",$this_obj); //make it so it's objs/main.o
		$this_file=concat("src/",$this_file); //make it so it's src/main.c
		for_each($headers," ") = $this_header){
			if(older($this_obj,$this_file) OR older($this_obj,$this_header)){
				$need_compile=true;
			}
		}
		if($need_compile){
			compile_async($this_file,$this_obj);
		}
		$objs=concat($objs,$this_obj);
		
		
	}
	wait_async(); //This will wait on all async processes to finish.
	//This is so everything is compiled before we link it.
	string $outname;
	$outname=concat("bin/kernel_",$ARCH);
	link($objs,$outname);
}
How much harder do you think it would be to write a parallel makefile doing such a thing?

Re: A new build system?

Posted: Thu Jun 18, 2009 5:03 pm
by JackScott
And how is that syntax any easier to read than the Makefile syntax? As long as you know what $^, $<, $% etc mean (or have a reference book handy like I do), it's pretty damn simple.

It seems to be a trap a lot of C programmers fall into. I know C, so I'll make everything else have C-like syntax. That'll make everything simpler to understand for everybody! But remember how much trouble you had learning C's syntax. I had a lot, certainly. Imagine if websites were written in a C dialect. Yuck. Sometimes a declarative language (like XHTML) is better for the job.

As for multithreading your syntax, it wouldn't be unfeasable to have each functional unit run simultaenously on a seperate core, as long as each unit didn't depend on any others. However, this means the programmer then has to think about how these blocks interact, instead of just letting the tool do all the thinking. A properly formed POSIX Makefile will run on anything from a 486 to a Core i7, and get the most performance out of all of them.

As for the comment about make being singlethreaded, you obviously haven't run make with the -j option on a multicore machine. :P

Re: A new build system?

Posted: Thu Jun 18, 2009 5:10 pm
by JamesM
Imagine if websites were written in a C dialect.
They are. PHP is the most C-like scripting language I have ever seen. (And this is not a good thing, my contempt for PHP is ill-hidden).

Re: A new build system?

Posted: Thu Jun 18, 2009 5:12 pm
by JackScott
I meant the HTML part. Or the CSS part.

Javascript is fairly C-like as well, from my limited experience with it.

Re: A new build system?

Posted: Thu Jun 18, 2009 5:28 pm
by earlz
JackScott wrote:And how is that syntax any easier to read than the Makefile syntax? As long as you know what $^, $<, $% etc mean (or have a reference book handy like I do), it's pretty damn simple.

It seems to be a trap a lot of C programmers fall into. I know C, so I'll make everything else have C-like syntax. That'll make everything simpler to understand for everybody! But remember how much trouble you had learning C's syntax. I had a lot, certainly. Imagine if websites were written in a C dialect. Yuck. Sometimes a declarative language (like XHTML) is better for the job.

As for multithreading your syntax, it wouldn't be unfeasable to have each functional unit run simultaenously on a seperate core, as long as each unit didn't depend on any others. However, this means the programmer then has to think about how these blocks interact, instead of just letting the tool do all the thinking. A properly formed POSIX Makefile will run on anything from a 486 to a Core i7, and get the most performance out of all of them.

As for the comment about make being singlethreaded, you obviously haven't run make with the -j option on a multicore machine. :P
I would guess it would have lots of fun output on stdout. lol

on the rest.... sigh... I'll give reworking my makefiles another try. But I really don't think it's possible to use any rules so that you can separate the source and obj directory..

Re: A new build system?

Posted: Thu Jun 18, 2009 5:35 pm
by JackScott
Why is it necessary to seperate the source and object files?

Edit: Makefile has the solution to your original problem.

Re: A new build system?

Posted: Thu Jun 18, 2009 5:48 pm
by earlz
JackScott wrote:Why is it necessary to seperate the source and object files?

Edit: Makefile has the solution to your original problem.
I prefer to have a src/ and objs/ directory so that I could optionally have a read-only source tree and also be able to specify where to put objs, say in the future it could be a separate partition I made just for building(so that make clean only consists of reformatting the partition)


And for the wiki, that's GNU make. $(shell..) and $(patsubt..) isn't in POSIX and doesn't work on BSD make
wait, or are you talking about dependency file generation? I actually have not looked into those..

Re: A new build system?

Posted: Fri Jun 19, 2009 12:28 am
by Solar
earlz wrote:
As for the comment about make being singlethreaded, you obviously haven't run make with the -j option on a multicore machine. :P
I would guess it would have lots of fun output on stdout. lol
No more or less than any multithreaded building system. Most build rules generate only one line of output anyway.

I think the solution JackScott referred to is having make find your source files automatically so you don't have to edit your Makefile for every source file added.

I believe it would be trivial to adapt the Makefile tutorial in the Wiki to support both BSD make and a seperate build directory.

Re: A new build system?

Posted: Fri Jun 19, 2009 8:33 am
by earlz
Solar wrote:
earlz wrote:
As for the comment about make being singlethreaded, you obviously haven't run make with the -j option on a multicore machine. :P
I would guess it would have lots of fun output on stdout. lol
No more or less than any multithreaded building system. Most build rules generate only one line of output anyway.

I think the solution JackScott referred to is having make find your source files automatically so you don't have to edit your Makefile for every source file added.

I believe it would be trivial to adapt the Makefile tutorial in the Wiki to support both BSD make and a seperate build directory.
Believe me, I've tried. The only real solution is to have two makefiles(neither of them following POSIX) one for BSD make and one for GNU. It's been deemed impossible by more than just me.

Re: A new build system?

Posted: Sun Jun 21, 2009 6:16 am
by Solar
So it's not really GNU make's fault for not being POSIX-compliant, since BSD make is neither?

In that case, I'd go with GNU make, as it's the more ubiquitous of the two. Most Unix-oid machines have it installed already, perhaps as "gmake", anyway.