xbstrap: A tool to build OS distributions
Posted: Fri Jan 04, 2019 9:04 am
Hi everyone,
some time ago, I found that building "large" OSes that depend on multiple interdependent packages¹ with traditional build system like make is a pain: Cross compilers need to be built, external packages need to be downloaded, patched, built and installed into the sysroot in the correct order (that satisfied their dependencies). Often, those packages need to be modified (e.g. for debugging reasons) and recompiled. Rebuilding the whole system from scratch can take hours -- when building managarm, about 5 GiB of object files are generated. Furthermore, modeling all inter-package dependencies in systems like make is complicated: Take for example a cross-GCC. To regenerate the build system of GCC specific versions of autoconf and automake are necessary. The target libraries like libgcc depend on libc and building libc in turn first requires building the compiler itself.
To automate this process and simplify rebuilding single packages, as well as the whole system, I wrote a new build system called xbstrap. This is an announcement as well as a RFC about this build system. xbstrap is specifically adapted to building (hobby) OS distributions. It does not replace build systems for individual packages (such a make, cmake or meson); rather, it augments them. Regarding usability and maturity, xbstrap is powerful is enough to build the entire kernel and userspace of managarm, including it's Wayland stack and Weston; however, it's command-line interface might still have some rough edges.
How does xbstrap work? It takes as input a YAML file to describe the tools (things running on the build machine) and packages (things running on the target machine) that need to be built. For example, to build libpng (which depends on zlib), the following YAML fragment is used:
For a full example, see managarm's YAML file. I am looking forward to hear your comments on this tool -- and if turns out that you want to try it to build your own OS, I'd be happy to help you and/or hear about your experience.
Best,
Alexander
¹ Such as my own OS, managarm.
some time ago, I found that building "large" OSes that depend on multiple interdependent packages¹ with traditional build system like make is a pain: Cross compilers need to be built, external packages need to be downloaded, patched, built and installed into the sysroot in the correct order (that satisfied their dependencies). Often, those packages need to be modified (e.g. for debugging reasons) and recompiled. Rebuilding the whole system from scratch can take hours -- when building managarm, about 5 GiB of object files are generated. Furthermore, modeling all inter-package dependencies in systems like make is complicated: Take for example a cross-GCC. To regenerate the build system of GCC specific versions of autoconf and automake are necessary. The target libraries like libgcc depend on libc and building libc in turn first requires building the compiler itself.
To automate this process and simplify rebuilding single packages, as well as the whole system, I wrote a new build system called xbstrap. This is an announcement as well as a RFC about this build system. xbstrap is specifically adapted to building (hobby) OS distributions. It does not replace build systems for individual packages (such a make, cmake or meson); rather, it augments them. Regarding usability and maturity, xbstrap is powerful is enough to build the entire kernel and userspace of managarm, including it's Wayland stack and Weston; however, it's command-line interface might still have some rough edges.
How does xbstrap work? It takes as input a YAML file to describe the tools (things running on the build machine) and packages (things running on the target machine) that need to be built. For example, to build libpng (which depends on zlib), the following YAML fragment is used:
Code: Select all
packages:
- name: libpng
source:
subdir: 'ports'
git: 'http://git.code.sf.net/p/libpng/code'
tag: 'v1.6.34'
tools_required:
- host-autoconf-v2.69
- host-automake-v1.15
- host-libtool
regenerate:
- args: ['./autogen.sh']
tools_required:
- system-gcc
pkgs_required:
- zlib
configure:
- args:
- '@THIS_SOURCE_DIR@/configure'
- '--host=x86_64-managarm'
- '--prefix=/usr'
build:
- args: ['make', '-j@PARALLELISM@']
- args: ['make', 'install']
environ:
DESTDIR: '@THIS_COLLECT_DIR@'Best,
Alexander
¹ Such as my own OS, managarm.
