What scripting should I use for building?

[KemyLand]

Should I use GNU Makefiles, CMakeLists, or even a dumb build.sh?

What is best for application building? (Ignore the build.sh example BTW).

I want a model like GNU's apps, where the root dir has a "root script". Every directory (and subdirectory) has its own script for building its specific parts, and call its subdirectories scripts. The root script starts the chain.

[Wajideu]

I typically use either autotools or a makefiles. For makefiles, I made my own template for projects and solutions.

If you like my templates and want to use them, I should point out that the directory structure of my projects is a bit different from most C/C++ projects.

Code: Select all

/Solution
 +- /Project
 |   +- /Assets
 |   |   +- /Internal	; Resources that get built into the program
 |   |   '- /Locale
 |   +- /Output
 |   |   +- /Debug		; Can add more, I just use these by default
 |   |   '- /Release
 |   +- /Shared			; Contains distributed libraries
 |   |   '- /Headers	 ; Contains distributed headers
 |   +- /Source			; Contains source code
 |   '- Makefile		  ; Project makefile (make from Project/Source)
 '- Makefile				; Solution makefile

[Muazzam]

You should use build.sh. It is easy, simple, stable and best.

[mallard]

You should use build.sh. It is easy, simple, stable and best.

And far too slow for any non-trivial project...

[Combuster]

"simple" shell scripts don't reduce compiling 400kloc to the set that actually changed.

[max]

You could use my buildscript, Capri (the article is very outdated, and some things have changed).

It allows you writing a simple buildscript with a C-like syntax and some cool features. Could look like this:

Code: Select all

project MyProject default build {

    task build depends clean {
        foreach(file : File.listTree("src", ".cpp")) {
            System.execute("gcc -c {file} -o {file}.o");
        }
    }

    task clean {
        File.cleanFolder("obj");
    }
}

In the commandline you call it by typing: capri
If you are interested, please write me a PM because its not publicly released by now

[Wajideu]

So I should write my own microcode? In fact, I guess, I'd better build my own processor, which is going to involve mining my own silicon.

You have to draw a sensible limit somewhere. Me, I'm interested in operating systems. I'm not really interested in reinventing vim, make, etc. (perhaps for the final OS, but not for the development host).

I don't think writing your own build system is such a bad thing if you have a good reason for doing so, and if the system you are building won't just be a knockoff of every other tool out there.

Makefiles and autotools are rather slow (especially the later), both are tailored for working with the GNU toolchain (although with a bit of effort can work with other tools), autotools won't work on Fat12 or Fat16 filesystems because of the 8.3 file naming scheme, autotools depends on other programs like m4 and aslocal that are a headache to port, autotools implicitly decides the directory structure for projects based on the standard 'nix filesystem which some people like myself find absolutely horrendous.
Cmake pollutes your project directory with a bunch of crap, the scripts for it are messy, and building resources for projects is tedious and/or impossible in some cases. Scons is slow and requires Python (I can't say much more about it that that because I have little experience with it).

I would actually like to make something more along the lines of a general project management software with support for software development that would allow setting goals, organizing tasks by priority, tracking progress, communicating with team members, managing budgets, etc. The whole 9 yards.

[iansjack]

Make predates GNU software and is not specifically aimed at their projects. Makefiles are extremely versatile and can be as simple or as complicated as you wish. I see no point in reinventing the wheel when I have better uses for my time.

[Wajideu]

Make predates GNU software and is not specifically aimed at their projects. Makefiles are extremely versatile and can be as simple or as complicated as you wish. I see no point in reinventing the wheel when I have better uses for my time.

The main problems with makefiles are that they are slow (as I said before) and take a large amount of time to set up. If I'm not building off of a template, I could spend upwards of an hour just setting up makefiles before I even begin. Then I constantly have to go back to it and modify as the project grows. There are also certain commands that are inconsistent between versions of make, or are system specific. Eg. `pwd` on Windows (without an environment like msys or cygwin) would have to be replaced with `echo %cd%`.

It's more like a scripting language than a project management tool. Each person has their own idea of what a makefile should look like, so there's no consistency whatsoever between projects made by different developers. This leads to confusion when developers want to contribute to open source projects.

As I said before, I would prefer a tool dedicated to project management in general, with software development being a subset of that. Being able to set goals and deadlines, track progress and changes, manage resources and budgets, etc. all with the same tool I build the project itself would be extremely useful. I've yet to see any tools like that though.

[Combuster]

The main problems with makefiles are that they are slow.

I typically write a 5-line Makefile from scratch whenever I get to a project with more than one source file. No point in fiddling with autotools (which gets things wrong anyway), and it very quickly becomes shorter and faster than the corresponding shellscript that just always does everything.

For the sake of it, I put your hypothesis to the test. My OS' build times are typically under a second, because it skips everything that doesn't need doing (I got 0.2s when I tried running make on a built version). Doing a clean build using make takes 5 seconds on my high-end machine. Doing a clean build with the "simple shell script" performing the same commands takes 26 seconds. That's actually factor 5 the wrong way, which makes me wonder what proof you have of the opposite?

A quick and dirty statistic says 1410 commands were executed in the process, so it's not a trivial process either.

My Advise: Do not depend upon tools made by others,

In contrast, NIH is considered a disease by some, and you might make it look contagious as well.

[Wajideu]

@Combuster Building a few 5 line makefiles is not a good benchmark test. Add some custom rules and make the makefiles configurable and recursive; then test it out on a large project and tell me how good the performance is.

That aside, I think I've made a good point that there are plenty of reasons a person might want to write their own project building utility.

[Combuster]

make the makefiles (...) recursive; then test it out on a large project and tell me how good the performance is.

If you explicitly demand to go against the wisdom of Recursive make considered harmful, then basically you're trying to make me rig the experiment in favour of your argument. Of course that doesn't help when I already posted another set of results that would automatically question the validity of either., so of course, I'm not going to do that.

I think the main question here is the following: have you even considered why using make is over 5 times faster than a shell script, even though it actually has more work to do.

Building a few 5 line makefiles is not a good benchmark test

Sorry to pop your bubble, but for the test given...

Code: Select all

# number of LOC in makefiles
find . -iname "Makefile" | xargs cat | wc -l
4767
# number of LOC in the project
find . -type f | grep -v ".svn" | xargs cat | wc -l
405326

. Of course, if you still don't believe, there's a link in my signature and you can check every file involved to see if I cheated anywhere. You're free to publish your findings, but it is much more likely to be a good example for you to show what you can actually do with hand-written makefiles.

[Wajideu]

make the makefiles (...) recursive; then test it out on a large project and tell me how good the performance is.

If you explicitly demand to go against the wisdom of Recursive make considered harmful, then basically you're trying to make me rig the experiment in favour of your argument.

I go against that advice because it is the dumbest advice anyone has ever given. Recursive building is absolutely necessary for enormous projects. Placing everything in a single directory is just begging to make your project un-maintainable.

have you even considered why using make is over 5 times faster than a shell script, even though it actually has more work to do.

I don't even use shell scripts for building. As I said before, I always use either autotools or makefiles (more often than not, the former).

Building a few 5 line makefiles is not a good benchmark test

Sorry to pop your bubble, but for the test given...

Code: Select all

# number of LOC in makefiles
find . -iname "Makefile" | xargs cat | wc -l
4767
# number of LOC in the project
find . -type f | grep -v ".svn" | xargs cat | wc -l
405326

. Of course, if you still don't believe, there's a link in my signature and you can check every file involved to see if I cheated anywhere. You're free to publish your findings, but it is much more likely to be a good example for you to show what you can actually do with hand-written makefiles.

Wtf does this have to do with anything? All I said is that timing a 5-line makefile does nothing to show how fast a makefile that's several hundred lines long is. Your example is meaningless. Moreover, your 5-line makefiles are most likely completely unmaintainable. How do you go about adding target specific options? Including headers from other directories? Linking libraries from other projects? Custom target rules to convert *.foo files into *.bar files? I highly doubt you have a clean way of doing this in a 5-line makefile. Your test pretty much dumbs down to, "makefile is the best because I mindlessly threw together a shoddy one and it works! Everyone should follow my example!"

[iansjack]

Recursive building is absolutely necessary for enormous projects. Placing everything in a single directory is just begging to make your project un-maintainable.

It's a shame that you didn't take the time out to look at the paper that Combuster referenced. Placing everything in a single directory is not a necessary pre-requiste to avoiding recursive Makefiles.

The moral seems to be that there is a perceived need to write your own project building system only because of a lack of understanding of how the established tools work. To quote from (a quote in) the paper:

If make doesn't do what you expect it to there''s a good chance that the makefile is wrong.

As ever, the BSD guys have nailed it.

[bluemoon]

"simple" shell scripts don't reduce compiling 400kloc to the set that actually changed.

But real programmers should use them. If they want to automate this process they should write their own utilities.

I think it's clear that we have different idea on what a real (or good) programmer is.
Anyway, I don't even consider writing own tool is a need, but it should be a decision if that align to your goal. For me, I don't write that tool since I want to be focus to what matter to me, especially if I don't even have the time to afford one hobby OS; but for other if they do it for fun, and found writing a tool interesting, go ahead - but unless he was doing it for no reason, it has nothing to do with the judging of programmer.

by the way, I use some shell script to do sanity checks and generate Makefile, but the shell script itself is minimal code (600 lines including the Makefile template).

Code: Select all

SLOC	Directory	SLOC-by-Language (Sorted)
6904    usr             ansic=2499,asm=2342,cpp=1537,sh=526
74      test            sh=74
0       project         (none)


Totals grouped by language (dominant language first):
ansic:         2499 (35.81%)
asm:           2342 (33.56%)
cpp:           1537 (22.03%)
sh:             600 (8.60%)