Makefile tutorial, reloaded

[Solar]

I am working on a new Makefile tutorial, with its scope much expanded into the realm of having multiple submodules that might be building libraries, executables or whatnot - and still handling them with a single global Makefile.

To make sure I don't miss my targetted audience's needs, what would be your demands for such a Makefile?

• Automatic dependency handling;
• handling various types of input (.c, .cpp, .s, .S, .ll, .yy, ...);
• keeping source directories clean, i.e. building in seperate directories;
• allowing each module to specify what it builds and which flags it uses in addition to the global ones (through Makefile includes akin in syntax to automake "Makefile.am");
• distinguishing between "local" headers (for in-module use only) and "global" headers (external interface of modules);
• capability of building per-translation-unit test drivers, as the current tutorial does;
• no additional dependencies other than a recent version of GNU make (i.e., no $(shell ...) calls, one of the central improvements over the current tutorial);
• ...

What did I forget?

[Combuster]

- All features working on both cygwin and linux (And I might just as well add msys and MacOS to preempt some other converts. Dependency file generation is a ***** to do right across platforms, same goes for unit tests. Stupid path conventions...).

And something that's rather specific to my needs: be able to build components for multiple architectures at once. Like one would be building ppc/x86/x86-64 universal binaries for mac or armv6/armv7 universals for handheld devices. In my case it's even trickier as I have to deal with some AMP architectures where you need some built for on one architecture, some on the other, and several core libraries built for either. (Amongst my collection are 68k+sh2+z80, arm+z80, ppc+arm, sh4+arm and x86(-64)+gpu platforms). Not that I'm expecting you to do all that.

[Solar]

Not that I'm expecting you to do all that.

Phew.

You did have me scared for a moment there.

Edit: I don't think I will cover the subject in the tutorial itself. But if you have a look at the draft I submitted to the Wiki in my user namespace, you might see a way to adapt the underlying technique to your needs.

(I got the idea for this tutorial when I realized that $(foreach), $(eval) and $(call) could be used for automatic self-generating of the Makefile. $(foreach architecture,$(ARCHITECTURES),...) could perhaps be the ticket for you.)

[Solar]

- All features working on both cygwin and linux (And I might just as well add msys and MacOS to preempt some other converts.) Dependency file generation is a ***** to do right across platforms, same goes for unit tests. Stupid path conventions...).

I don't quite see the problem here...? I would say whether I'm in a Cygwin bash or a Linux bash doesn't really matter... what have I missed?

(Actually I did the current draft on Cygwin, waiting for a lengthy compile to finish. Where's the catch you obviously encountered?)

[Combuster]

It's been quite a while since I did the last revamp of my makefile system to the design I currently have (which sadly, does not include automatic dependencies). Dependencies had a habit of giving syntax errors in the next run of make whenever files outside the source file's path were included. I also needed to implement a hack for a mingw build of make once where it mattered if you used / instead of \. I also know that depending on a directories is broken (windows changes the modification time of the folder when files in it change, so a dependency to that will cause everything to be rebuilt over and over again.)

Anyway, that configuration has vanished with this machine's last reinstall and I haven't yet bothered to reconfigure it as I don't do much development on windows machines anymore. (that goes for both office and home) I should do that again to check for myself if my project is still customer-friendly enough and to create the needed test cases for anything broken.

[Rusky]

A slight extension of this:

• allowing each module to specify what it builds and which flags it uses in addition to the global ones (through Makefile includes akin in syntax to automake "Makefile.am");

would be to allow specific build instructions for specific files.

[Solar]

...allow specific build instructions for specific files.

The current draft is functionally equivalent to:

Code: Select all

foo/obj/%.o: foo/src/%.c
	$(CC) $(CFLAGS) $(CFLAGS_foo) -c $< -o $@

bar/obj/%.o: bar/src/%.c
	$(CC) $(CFLAGS) $(CFLAGS_bar) -c $< -o $@

Each binary / library also has its own (optional) flags:

Code: Select all

bin/someexe: $(someexe_OBJECTS) $(someexe_LIBRARIES)
	$(LD) $(LDFLAGS) $(LDFLAGS_someexe) $^ -o $@

Similar rules for other conversions (.S -> .o etc.) or output types (shared libs, anyone?) can easily be added.

You seem to say that this (having per-module / per-exe compiler options) would not be enough for you, and that you would like to define seperate ways to turn e.g. a .c file into a .o file, or .o files into an .exe?

Would you care to give an example? I am not quite sure what you're trying to achieve (or, actually, how different turning a .c into an .o could be made).

[Rusky]

It would be more for other types of sources or for simple third-party sources like glew.c that don't need their own library. The way I've done it in the past is to add a new rule to the module's included Makefile- I'm just hoping to bring that kind of thing to your attention in case it interacts weirdly with how your tutorial works.

edit: looking at your draft so far, another nice thing to have would be command-line overrides of build flags- Makefiles generally allow stuff like "make CFLAGS='-g -O2'" to work. Currently, doing that would override the -MMD -I include flags.

[Love4Boobies]

For the record, POSIX:2008 TC1 will enhance standard make a great deal (enough to finally make it useful) so this time it might actually be worth waiting until it comes out or at least refactoring the tutorial so that it covers portable makefiles instead.

Here are a couple of enhancements that come to mind:

• Macro functions and, possibly, pattern rules. Exactly which macro functions will make it in (pun intended) is still an open discussion. To give an example, SunPro handles GNU make's addprefix and addsuffix functionality with pattern macro expansions.
• The addition of several assignment operators: !=, +=, ?=, and (probably) ::=. The latter would be equivalent to GNU's :=, which would not be suitable for standardization due to the fact that it breaks common practice, particularly on BSD and Solaris. As for !=, it's been around in *BSD for ages and is now also available in GNU make's trunk.
• Silent includes (via -include). It wasn't discussed yet, but the ability to include multiple pathnames on one line is also very important. The following two bullet point goes hand in hand with this one.

Dependency file generation is a ***** to do right across platforms

• Automatic dependency generation. This isn't really an enhancement to the make utility itself but it's something the tutorial would need to cover anyway. I was the one to make this particular proposal; right now, we're discussing which solution would make most sense: GCC's -M? family of switches (it breaks compatibility with some environments, such as MKS), GCC's DEPENDENCIES_OUTPUT environment variable---SunPro provides SUNPRO_DEPENDENCIES which is similar except it doesn't exclude system headers, or the separate makedepend utility (except there are a couple of issues with its current incarnation---i.e., the fact that it has no flag to ignore system headers, the non-standard -include flag, and the -- thingy which only makes sense as an extension).
• The ability to use dashes in macro names.

This is not an exhaustive list.

One thing that will not make it in, unfortunately, is the automatic remaking of makefiles, as neccessary (this is the default behavior of GNU make and can also be enabled in FreeBSD make by specifying .MAKEFILEDEPS).

[Solar]

It would be more for other types of sources...

Those can easily be added to the RULES_template. The finished tutorial will give some more examples.

...or for simple third-party sources like glew.c that don't need their own library.

You'd put it in a linker lib/archive, then, and link that as needed...? (Sorry, I don't know about glew.c or how it is commonly used.)

...another nice thing to have would be command-line overrides of build flags...

Of course. I have this nasty habit of defining CFLAGS myself; that would have to be edited for the tutorial.

For the record, POSIX:2008 TC1 will enhance standard make a great deal (enough to finally make it useful) so this time it might actually be worth waiting until it comes out or at least refactoring the tutorial so that it covers portable makefiles instead.

We had this discussion before. I don't much care for making Makefiles any more "portable" than using the ubiquitous GNU make. Building GNU make on any machine that doesn't have it is dead easy, and certainly an improvement over other incarnations of make that I have seen so far. It is easier, for example, than building a cross-compiler. Do binutils / gcc even build without an available GNU make?

Moreover, having a standard doesn't help anyone unless you also have the tools that implement the standard... until that day, there are gazillions of machines with GNU make installed.

If you feel like it, you can take my tutorial and make it "portable" when and if a workable "standard" make emerges...

[Love4Boobies]

I hope it didn't sound like I was imposing anything. Like I said last time, it's your tutorial and thus your decision. The only reason for which I mentioned this is that I assumed that you chose GNU Make over POSIX make because the latter lacked so many features, which is something that is going to change.

Although I'm sure there people who have other make implementations installed (e.g., I know Brynet-Inc uses BSD), (a) it's a tutorial, not a framework and thus people should understand what's going on rather than copypasta, and (b) as you say, GNU Make is quite easy to set up (it actually happens to be the very first UNIX tool I ever built on Windows).

[Brynet-Inc]

The entire OpenBSD userland including GNU gcc + binutils are built using OpenBSD's make, with some wrappers. GNU make isn't in base, it's a port.

GNU autotools can generate Makefiles that work with more than just GNU make, most of the time though it breaks because people let little GNUisms seep through.

For example, all of modular X11R7 uses autotools now, and they manage to build just find using BSD's make.

I don't believe I've commented on your article yet Solar, but, assuming that everyone uses GNU make is wrong, even if it is popular.

[Love4Boobies]

That's ok, that's why it's a community project. It's worth appreciating that he's writing the tutorial in the first place; after that, others can help improve it by explaining what's portable and what isn't, what's recommended for each make implementation, etc.