Advanced makefile wizardry

[Candy]

Grab cmake or waf and never worry again.

Cmake is one of the most horrible things I've ever seen. It's basically a program written around makefiles so you can write Cmakefiles instead of makefiles.

And then it creates makefiles, so you have to run make afterwards.

My current makefile is some 100 lines and figures out what to do all by itself, no cmaking or whatever needed. If you want to port it, GNU make is available for porting.

As if any Make would work without checking the timestamps. That's what Make systems are about.

Well.... no, not really. Make systems are about:

1. Finding out what's changed
2. Finding out the least set of commands to execute to get the up-to-date output given the knowledge from 1.
3. Running those commands with the available parallellism to get the shortest time before completion.

There's no requirement that 1. uses timestamps, it's just the way it's been done so far. Checking 50000 files for timestamps when my editor has just saved 3 and knows which ones they are - and the filesystem has just received those writes so it knows as well - seems a bit of a waste of time.

Given 1., 2. should be a pretty quick run over the makefile (as the makefile is short). 3. is something make does well already (imo).

The only thing whose duration is in terms of the amount of files you have is 1. If you manage to change that to O(1) your build is *always* quick, even if you put the entire OS (and I mean a full-blown OS) in a single makefile.

[Owen]

Grab cmake or waf and never worry again.

Cmake is one of the most horrible things I've ever seen. It's basically a program written around makefiles so you can write Cmakefiles instead of makefiles.

And then it creates makefiles, so you have to run make afterwards.

CMake is neither written around makefiles, nor does it have to create them. It has a variety of other targets; this is what makes it so valuable.

Well, besides the fact that it can lookup various libraries for you in a way that works very portably, is quick, and doesn't make you want to gnaw your face off (Yes I'm looking at you Autotools. Particularly you, Autoconf)

[Love4Boobies]

As if any Make would work without checking the timestamps. That's what Make systems are about.

Well.... no, not really. Make systems are about:

1. Finding out what's changed
2. Finding out the least set of commands to execute to get the up-to-date output given the knowledge from 1.
3. Running those commands with the available parallellism to get the shortest time before completion.

There's no requirement that 1. uses timestamps, it's just the way it's been done so far. Checking 50000 files for timestamps when my editor has just saved 3 and knows which ones they are - and the filesystem has just received those writes so it knows as well - seems a bit of a waste of time.

Make was standardized to work with timestamps, actually. There's no other way to do this portably. You either need to make your file system aware of make so it can somehow tell it what has changed since the last invocation of the make executable (that would also mean that the file system would need to track dependencies between make and makefiles since make does not follow any directory structure; your files can be anywhere on the volume - thus making the whole file system slower because it needs to check all sorts of lists everytime you write to a file) or you need to distribute your IDE together with the application and make sure that people use only that IDE (however this has the disadvantage that you cannot work with anyone else on a project unless you somehow synchronize your IDEs - anyway, it's complicated).

[gerryg400]

You either need to make your file system aware of make so it can somehow tell it what has changed since the last invocation of the make executable (that would also mean that the file system would need to track dependencies between make and makefiles since make does not follow any directory structure; your files can be anywhere on the volume

clearmake works this way. While a target file is being built the filesystem keeps track of all files that are read or executed. If any of those files changes, even for example a script, or a dll that is part of the tool chain, the targets that depend on it can be rebuilt. The filesystem contains a database. Performance is good. clearmake is also able to provide a build audit of any target file.

[Love4Boobies]

Sounds to me like a crappy design because:

• It slows down the entire system (even if not by much) for a simple (and very specific) application to do its job.
• It's unfair to other applications which don't have this opportunity. And if they do, they could abuse it which is even worse, making the system even slower (besides taking up memory).
• It becomes a bottleneck if you are managing a couple of HUGE projects.

[gerryg400]

It slows down the entire system (even if not by much) for a simple (and very specific) application to do its job.

The database, clearmake and auditing features are built into the filesystem on a particular volume so it only affects the speed of the volume that has that filesystem on it. I'm fairly sure that read, wirte etc. are slower on that volume but the rest of the system is unaffected.

The system is brilliant. It's only problem, in my opinion was the cost of buying the software.

There are features of clearmake that no other make system has. For example, because clearmake absolutely and positively knows (at the hardware level!!) what the dependencies of a file are, you never need to build a configuration that has been built before. What this means is that if I set a particular configuration and do a build, and later you were to come along at your terminal and set the identical configuration, it would provide you with my binaries without needing to execute CC or LD or anything. On a big project the time saving is huge.

If 2 people compile the same object file with identical configuration, that object is added to an internal database in the filesystem for when the next person comes along and builds that object. When that person types clearmake, the binaries just appear in his view magically. The compiler doesn't even get executed. So a file which doesn't change, and whose dependencies don't change only needs to be build once during the entire life of the project.

[Love4Boobies]

Yes, volume was what I meant. I don't really understand how it would know at the volume level (not hardware!) what the dependencies between the different files are. Some sort of instructions still need to be given one way or another because there's no general rule that you can come up with.

[gerryg400]

I don't really understand how it would know at the volume level (not hardware!) what the dependencies between the different files are. Some sort of instructions still need to be given one way or another because there's no general rule that you can come up with.

Remember I'm talking about a filesystem here. An installable filesystem driver that replaces ext3 or whatever. During clearmake the filesystem can see every file that is opened for read, execute or write. This provides canonical information about what happened during the build. Time stamps are ignored. But there is a perfect build audit trail for every build. You can query an object file on the filesystem and it can tell you who built it and the version/contents of every file that was executed, read or written during the creation of that file. For build managers and quality auditors this type of tool provides information that just cannot be got anywhere else.

In clearmake, a makefile looks just like any other makefile except that there is no dependency information of any type needed.

[Love4Boobies]

The lack-of-timestamps thing I got. I was talking about the dependencies which need to be specified by the programmer (i.e., the build system cannot know what somefile.ext depends on or how it is produced; all this information still needs to be provided one way or another).

[gerryg400]

I'm guess I'm using the word dependency a bit differently (wrongly) from you. By dependency I mean a file that not part of the rule but upon which the object still depends. For example header file. Sorry about that.

It's been a few years since I used clearmake and the detail is sketchy. So going from memory...

There is a makefile, actually your normal makefile will work. IIRC it's gnu make compatible. There are rules. But you don't need this

Code: Select all

foo.1 : foo.man sedscript 
        sed -e sedscript foo.man > foo.1

Just this

Code: Select all

foo.1 :
        sed -e sedscript foo.man > foo.1

because clearmake will build foo.1 the first time and after that will know what is included/used when foo.1 is built.

Believe me, clearmake works. We had more than 10000 files in our build, and 100's of engineers working in several countries. Never an issue.

There's some info here http://en.wikipedia.org/wiki/IBM_Ration ... e#Features

[coreybrenner]

One thing I don't get is, what happens when you do this:

Code: Select all

$ clearmake
$ FOO=yes clearmake

[gerryg400]

Sorry I don't understand the question. I'm not a clearmake expert, just a satisfied user. What do you mean?

[gerryg400]

One thing I don't get is, what happens when you do this:

Code: Select all

$ clearmake
$ FOO=yes clearmake

Oh I guess you mean that FOO might affect the build and because it's set as a variable outside clearmake, clearmake may be unaware. Well it all depends on what FOO is used for within the build process. When Clearmake executes a file, script or binary, also records the args passed to it. Perhaps it also records environment vars. I'm not sure but it certainly copes with this very common situation.

[fronty]

It is standard feature of make that you can access environment variables as macros. I don't know if clearmake implements this, just heard about it in this thread.