OSDev.org

Posted: **Sun Feb 28, 2021 2:04 pm**

Dear All,

I'm looking for testers for my new project. It is a system package management library, that can be compiled for both POSIX and WIN32 API (and ported to others easily). Testing it on Linux, MacOS and Windows is just as much appreciated as testing it on your own hobby OS.

https://gitlab.com/bztsrc/syspkg

Why is it different to the existing packaging solutions?
It is a dependency-free static library written in ANSI C. You can integrate it into your OS just by writing a CLI or GUI wrapper around the library, which obviously uses your OS' native user interface. No other existing solution can provide that.
It is also a COMPLETE solution. No armada of shell and python scripts and additional commands like with all the other existing solutions, just a single small, dependency-free library that solves everything. This simplifies porting to new OSes greatly.

I've spent quite some time with summarizing all my experience using apt, emerge, pacman, portage, port, macports and I've created a solution that gets the very best of them, hopefully without their drawbacks. I've also added features unknown to these: translatable descriptions and package names, canonical package versioning, screenshots, etc. Oh, and unlike these, syspkg is lightning fast. I've generated 32768 packages with over a million files, and the local database was created in less than a minute (including downloading package lists, meta JSONs, verifying certificates, checksums, digital signatures etc.) Finding which package provides a particular file took about a sec

What does dependency-free mean?
If your OS is a POSIX compatible one, then you'll need a libc (a fairly complete one with Berkley sockets, but still that's all). Nothing else is needed. If your OS uses its own API (or WIN32 API for some reason), then all OS-dependent functions are gathered in one file (and the network related ones in another), so that you can tailor it to your needs. The library uses this abstraction everywhere. You can also turn the networking support entirely off with a single define if your OS uses CDROM install media for example. All OS file system layout related configuration gathered into a single config.h file for simplicity too.

Despite of being dependency-free, it's small, and provides API for everything you'll ever need: obviously storing and looking up package info, but also creating and extracting payloads, compression and decompression of streams, RSA key generation, PEM certificate management, generating and checking checksums and digital signatures, SSL/TLS support for downloading, decoding PNGs and resampling screenshots into thumbnails, etc.

How to use?
As it's a library, I've provided a demo CLI application (576K compiled on my Linux). There's a tutorial to get you acquaintanced with syspkg's concepts, as well as a detailed library API documentation.

In a nutshell:

syspkg assumes that application development has moved to code hosting servers (like github or gitlab), but supports old-style tarballs on a static webserver method too.
You create a JSON describing the package and where to download it from (this can be independent to the application's repository, it can be even on a different server)
You create lists of URLs to those JSONs, these are called package repositories or package lists
End users and OS creators can add repo URLs to their local syspkg configuration file, and with a single command they can update their local package database
End users then can search this local database and use it to install or upgrade packages

If someone would like to test it out, please use the gitlab's issue tracker to report feedbacks.

Cheers,
bzt

Posted: **Sun Feb 28, 2021 3:30 pm**

1. Sorry, I am quite busy with my current projects, so I'm not willing to help. Except maybe trying to use it on my Linux system-

2. Cool idea, I hope for you and for me that this project will be a success and will be used quite a lot. Because this would eliminate a lot of trouble.

3. Does it have meta packages support? Like for example "development" including gcc, g++, make, python, headers etc.

4. How can I create a package for syspkg? Or is it always simply direct from repo/tar-ball? EDIT: I seem to have found an answer to this here: https://gitlab.com/bztsrc/syspkg/-/blob ... utorial.md I will read that.

Greetings
Peter

Posted: **Sun Feb 28, 2021 6:36 pm**

PeterX wrote:1. Sorry, I am quite busy with my current projects, so I'm not willing to help. Except maybe trying to use it on my Linux system-

That's more than enough

I appreciate Linux tests as well!

PeterX wrote:2. Cool idea, I hope for you and for me that this project will be a success and will be used quite a lot. Because this would eliminate a lot of trouble.

Thanks! Well, I'm going to use it for my OS that's for sure so it is already a success

If others can use it too, then even better! I wrote it to be easily ported because I don't want to suck with porting it to my OS

And I also needed to run it under Linux, because I'm generating the packages on Linux for now (until I port gcc and friends too). And if I was at it, I also implemented it for Windows, so that OSdevers using that can create packages on their devmachine too.

PeterX wrote:3. Does it have meta packages support? Like for example "development" including gcc, g++, make, python, headers etc.

Yes. If a package has no payloads only dependencies, that's a meta package. It also has alternate packages support (for example you could either install "xserver.xfree86" or "xserver.xorg", both would satisfy the dependency for "xserver"). It also supports post install hooks for source packages (containing only source, no binary), like calling "configure"/"cmake" and "make". Read maintainers.md for the details.

PeterX wrote:4. How can I create a package for syspkg? Or is it always simply direct from repo/tar-ball? EDIT: I seem to have found an answer to this here: https://gitlab.com/bztsrc/syspkg/-/blob ... utorial.md I will read that.

In short, with syspkg build. It will fill up the metajson with some fields (like list of files, archive's sha, uncompressed size etc.) and will sign it with your certificate. It can also create ZIP64 archives for you with ZStandard compression (otherwise you can use github's "Download ZIP" url if you wish).

NOTE: the phrase "package" is quite overloaded in this context, so I've established the following naming:
- config file: list of repo URLs, repo list
- repo: list of metajson URLs, also called package list
- metajson: describes a package, here "id" refers to the entity that I call a package
- payloads: compressed binary bitchunks containing files for the package. (I've avoided the name tar-ball since they are not in ustar format)

Cheers,
bzt

Posted: **Mon Mar 01, 2021 12:57 pm**

I've created an overview to help understand the big picture. It explains how components are fit together, and also how syspkg helps rapid development cycles and beta testing without touching the package repositories.

Of course Server #1, Server #2 and Server #3 could be the same server; also each repo.txt, package.json and payload.zip could be on separate servers, etc. It is totally up to the OS developer where these files are downloaded from.

Hope this figure is useful.

Cheers,
bzt

Posted: **Tue Mar 02, 2021 2:08 am**

I would potentionally be interested in using this as I've reached a point where i need a package manager. What I'm wondering is though, does this support multiple versions of a package (dependencies relying on specific versions) and how is installation of a package working/configured?

Posted: **Tue Mar 02, 2021 3:10 am**

Looks interesting. We are currently using xbps as a package manager on Managarm (and xbstrap as our distro build system). This tool seems to be both a package manager and a distro build system? Why was this design choice made, doesn't that mean that existing distro building tools are incompatible with syspkg? Is it possible to pack an existing DESTDIR into a package w/o using syspkg as a build system (like with dpkg, rpm, pacman or xbps)? If yes, xbstrap could potentially integrate this as a backend.

Posted: **Tue Mar 02, 2021 3:52 am**

Ah, wait, this is a source package manager (like emerge), not a binary package manager (like the others that you have listed, like dpkg, rpm, pacman, xbps, ...) right? Then I guess my question has been answered.

EDIT: or am I maybe confused by the example in the maintainer's overview document with the ./configure call and the GitHub URLs? bzt, can you explain whether this supports source packages, binary packages or both? Where does building happen, on the maintainer's machine or on the user's machine? Does the maintainer upload a binary package to a web server?

Posted: **Tue Mar 02, 2021 6:55 am**

MollenOS wrote:What I'm wondering is though, does this support multiple versions of a package (dependencies relying on specific versions)

This provides rolling-release package management, meaning it's expecting the latest of every package. You can specify minimum versions for dependencies, for example "depends": [ "libx11" ] means the package depends on libx11 and "depends": [ "libx11 1.0.0" ] means the package is depending on libx11 version 1.0.0 or later. Since it provides rolling-release, there's no "depends on package version not after X", only "depends on version X and not before".

MollenOS wrote:and how is installation of a package working/configured?

There's a separate config.h that describes where a particular OS expects the installed files, what architectures and categories are allowed etc. Files in the payloads are categorized into directories (like "bin", "lib", "src"), and each directory will be extracted under a directory specified in that config.h (so "bin" extracted to BINDIR, "src" extracted under SRCDIR etc.).

Korona wrote:This tool seems to be both a package manager and a distro build system?

I'm not entirely sure what you mean by distro build system, but this is a package manager. Packages are installed into preconfigured directories, with one exception: the package "base" (and only that) extracted to the root directory.

Korona wrote:Ah, wait, this is a source package manager (like emerge), not a binary package manager (like the others that you have listed, like dpkg, rpm, pacman, xbps, ...) right?

Depends how an OS wants to use it. It is primarily a binary package manager, but also capable of installing source packages and calling postinst hooks to patch and compile. Pretty much like deb. Although this is just for my OS, but I'm planning to have two packages for each application: something (that's the binary) and something-src (that's the source version). Of course you can create whatever scheme you'd like, for example apt solves this by duplicating the packages with the names typically being: something (binary version) and something-dev (source version, or version with header files needed for compilation).

Korona wrote:Where does building happen, on the maintainer's machine or on the user's machine? Does the maintainer upload a binary package to a web server?

The idea is that building happens on the maintainer's machine, but since GPL and many other Open Source license mandates that users must be able to install the source and compile it on their machine, that's a possibility too (however with a different package, because the binary must contain the architecture, while the second must use architecture "any"). It depends on how you set up your metajsons, syspkg tries not to tie your hand on this matter. It just installs files and calls a postinst hook if/when required.

Cheers,
bzt

Posted: **Tue Mar 02, 2021 7:17 am**

Okay, but why does the example contain a configure call and a GitHub URL (which presumably contains the sources, not the binaries)? How are these build instructions connected to the binary package?

From a binary package manager, I'd expect a tool that takes a DESTDIR and converts it into a package (like dpkg-deb or xbps-create). But it seems that the mechanism is different here, hence I am confused.

EDIT: also, why does configure run in a postinst step? Doesn't postinst run on the user's machine, not the maintainer's one?

Posted: **Tue Mar 02, 2021 8:02 am**

Korona wrote:Okay, but why does the example contain a configure call and a GitHub URL (which presumably contains the sources, not the binaries)? How are these build instructions connected to the binary package?

Those are just examples to one possible configuration.

Korona wrote:From a binary package manager, I'd expect a tool that takes a DESTDIR and converts it into a package (like dpkg-deb or xbps-create). But it seems that the mechanism is different here, hence I am confused.

Yes, one thing how syspkg is different to all the other package managers is that it doesn't have DESTDIR, instead it has 8 variables, one for each kind. BINDIR for binaries, LIBDIR for libraries, INCDIR for include files, SRCDIR for source files, etc. It's a bit more complicated to set up at first, granted, but has huge advantage on the long run.

As for creation, it doesn't necessarily take a directory as an argument, it could be a github "Download ZIP" url too. But if it's a directory, then you must specify multiple "DESTDIR"s, one for each architecture, for example:

Code: Select all

syspkg build mypackage.json x86_64=./buildIntel/ aarch64=./buildARM/

That is, if you choose binary distribution. With source packages you specify a directory with the source as "DESTDIR", and add postinst hooks in the json instead:

Code: Select all

syspkg build mypackage.json any=./srcdir/
cat mypackage.json | grep "postinst"
   "postinst":

When the user installs such a package, SRCDIR will be used as the one you call DESTDIR.

Korona wrote:EDIT: also, why does configure run in a postinst step? Doesn't postinst run on the user's machine, not the maintainer's one?

It totally up to you. There are two approaches:

1. you compile the application
in this case you run configure and make on your machine, the payload contains pre-compiled binaries and there's no need for postinst in the metajson

2. the user compiles the application
in this case the payload contains source files, and the postinst tells how to compile it into binary on the user's computer

Which one you choose is up to you. You can even mix those, for example having a binary payload and postinst not compiling but generating configuration files for it. It's totally your call how you write the metajsons. Syspkg tries to give you maximum freedom on that matter.

Cheers,
bzt

Posted: **Tue Mar 02, 2021 8:24 am**

Okay, I get it now. Maybe adding the json file for a binary package as another example to the docs would be useful.

Another suggestion: the directories don't seem to match usual DESTDIR (= usr/) directory names (e.g., inc vs. include). If standard names were used, existing build systems (such as cmake) could be used as-is.

Posted: **Tue Mar 02, 2021 8:28 am**

I try to collect some combinations for possible use cases, but this is list not complete by no means.

1. you support one architecture and you want binary packages

Code: Select all

syspkg build mypackage.json x86_64=./build

2. you support multiple architecture and you cross-compile

Code: Select all

syspkg build mypackage.json x86_64=./buildIntel aarch64=./buildARM

3. you support multiple architecture and you set up VMs with shared folders
(here you compile the package in VMs, then you can use both zip or syspkg to create the payloads)

Code: Select all

syspkg build mypackage.json x86_64=localdir/x86_64.zip x86_64=localdir/aarch64.zip

3. you support multiple architecture and you set up VMs to access github
(here you compile the package in VMs, then you can use both zip or syspkg to create the payload and upload, or git to upload the files and rely on "Download ZIP" url)

Code: Select all

syspkg build mypackage.json x86_64=https://github.com/me/my/archive/x86_64.zip x86_64=https://github.com/me/my/archive/aarch64.zip

4. you want a source package
(here you have to add postinst to mypackage.json)

Code: Select all

syspkg build mypackage any=./srcdir

5. you have a non-binary package (like Python scripts for example)

Code: Select all

syspkg build mypackage any=./scriptsdir

Posted: **Tue Mar 02, 2021 8:37 am**

Korona wrote:Okay, I get it now. Maybe adding the json file for a binary package as another example to the docs would be useful.

Yes, that's a good idea.

Korona wrote:Another suggestion: the directories don't seem to match usual DESTDIR (= usr/) directory names (e.g., inc vs. include).

Of course not, you're a bit confused here. There are three independent directory specifications:

1. directory on the file system
these are specified in config.h, like BINDIR = /usr/local/bin, INCDIR = /usr/include.

2. directory category
These are the top level directories in the metajson file list (and probably in the payload), but ain't real directories, just categories. For example "inc/mylib.h". It's using "inc" instead of "include" because all top level categories are 4 bytes long. This has nothing to do with the actual directory where the file will be extracted to.

3. payload directories
If you can't use 2. in payloads, because you're importing someone else's repo, then you can specify overrides per package. For example "inc/" = "code/headers" or "inc/" = "include".

I'll try to explain why is this. There are three separate levels where directories are stored:

payload: 3. is the directory in the payload (could be different or the same as 2.), which is translated into 2.
metajson file list: here 2. is just an abstraction, has nothing to do with the actual directories. Here all top level directories have 3 letters because they are actually categories
file system: finally files will be extracted to 1., each abstraction into its own directory, "src/" => SRCDIR, "inc/" => INCDIR etc.

Korona wrote:If standard names were used, existing build systems (such as cmake) could be used as-is.

What do you mean by "standard names"? Environment variable names? Those can be set up in postinst as you wish, eg. DESTDIR=$SRCDIR. Directory names? Those are specified by you in config.h, like SRCDIR = "/usr/src/%s/" or BINDIR = "/Applications/%s.app/Content/MacOS".

Cheers,
bzt

Posted: **Tue Mar 02, 2021 8:56 am**

Cool, thanks for your answers. I will probably give this a go - it's a lot easier to implement than Conan.

Posted: **Tue Mar 02, 2021 8:57 am**

MollenOS wrote:Cool, thanks for your answers. I will probably give this a go - it's a lot easier to implement than Conan.

If you run into any trouble, just open a gitlab issue.

Cheers,
bzt

OSDev.org

Syspkg

Syspkg

Re: Syspkg

Re: Syspkg

Re: Syspkg

Re: Syspkg

Re: Syspkg

Re: Syspkg

Re: Syspkg

Re: Syspkg

Re: Syspkg

Re: Syspkg

Re: Syspkg

Re: Syspkg

Re: Syspkg

Re: Syspkg