OSDev.org

Hello everyone,
after a long time, I'm pleased to announce the first release of Tilck (version 0.1).

What is Tilck?
Tilck is a x86 monolithic kernel designed to be partially compatible with linux-i686 at binary level,
but it does not aim to re-implement Linux nor to become a desktop operating system. It is, for the
moment, an educational project that I see as the perfect playground for playing in kernel mode while
retaining the ability to compare how the very same usermode bits run on the Linux kernel as well.

Why trying to be compatible with Linux?
To me, this feature is extremely valuable as it allows the kernel to run third-party software like BusyBox,
VIM, fbDOOM, Lua, TinyCC etc. never written for Tilck: this way, I'm able to test the correctness and the
robustness of my implementation against well-known software and, in case there's a difference, I can
trace the same application/test on both Tilck and Linux and observe where differences start to appear.
To put it in a different way, I believe that writing user code for my own kernel it's like playing a video game
I wrote myself: I'm biased because I know too well how it works under-the-hood and it's less likely I'll
exploit issues. In addition, I value this feature also because it allows a whole class of Linux applications
to work as-it-is, without any porting. And, even when some effort is required to port something to Tilck (e.g.
the Linux feature XY needs to be implemented in the kernel), the payoff for that effort is high, as other
programs requiring the same feature will work, from now on. Still, that doesn't mean it make sense to
re-implement the whole Linux interface; it's a case-by-case thing.

Booting Tilck
Tilck comes with a custom interactive bootloader working both on legacy BIOS and on UEFI systems
as well. The bootloader allows the user to choose the desired video mode, the kernel file itself and to edit
kernel's cmdline. Also, Tilck can be loaded by multiboot-compatible bootloaders like GRUB.

Testing Tilck
Tilck has unit tests, kernel self-tests, system tests (using the syscall interface), and automated
interactive system tests (simulating real user input through QEMU's monitor) all in the same repository,
completely integrated with its build system. In addition to that, there's full code coverage support and
useful scripts for generating HTML reports. Finally, Tilck is fully integrated with the Azure Pipelines CI, which
validates each pushed branch with builds and test runs in a variety of configurations. Kernel's coverage
data is also uploaded to CodeCov.

Hardware support
While the kernel uses a fair amount of legacy hardware like:

• the 8259 PICs for IRQs
• the 8254 PIT for the system timer
• the 16550 UART for serial communication
• the 8042 kb controller
• the 8237 ISA DMA
• the Sound Blaster 16 sound card (QEMU only)

It has also support for some recent hardware features like:

• SSE
• AVX
• AVX 2
• PAT
• sysenter
• enumeration of PCI and PCI Express devices (via ECAM)
• ACPI support via ACPICA

ACPI is currently used to receive power-button events, to reboot or power-off the machine,
and to read the current parameters of machine's batteries (when implemented via ACPI control
methods).

File systems
Tilck has a simple but full-featured (both soft and hard links, file holes, memory mapping, etc.) ramfs implementation,
a minimalistic devfs implementation, read-only support for FAT16 and FAT32 (used for initrd) allowing memory-mapping
of files, and a sysfs implementation used to provide a full view of ACPI's namespace, the list of all PCI(e) devices and
Tilck's compile-time configuration. Clearly, in order to work with multiple file systems at once, Tilck has a simple VFS
implementation as well.


Future plans
In the long term, depending on how successful the project will be, Tilck might become suitable for
embedded systems. In particular, the plan to port this kernel to ARM existed from the very beginning
of the project. x86 was simply the most convenient architecture for me to start with. Also, it had the
advantage that most laptops and workstations are, indeed, x86 (x86_64) and that's great for testing
purposes and to show it to friends and colleagues. But, given the non-existent presence of x86 in the
SoC world, Tilck's real target architectures are ARM (e.g. Cortex-A) and maybe RISC-V. Because of that,
Tilck can already run today on a QEMU vm with just 4 MB of RAM, even if it's pointless on x86. Finally,
I'd like to make it work on MMU-less CPUs (e.g. ARM Cortex-R series), and enter in the realm of
microcontrollers and hard real-time operating systems. Clearly, the project is far from there, but we're
talking about the long term. Also, the kernel is already modular (has compile-time modules) and it has
been designed from to beginning to depend on a very few HW features. The biggest of them is the MMU.
But, given that Embedded Linux does support MMU-less systems, Tilck might evolve in that direction as well.
For example, it already has vfork(). Therefore, Tilck might become yet-another "linux-tinification" attempt,
with the plus of the BSD license, which is much more permissive than Linux's GPL v2.

Said that, the future remains uncertain and the project might not get enough engagement, as it happens to
most software projects. In that case, I'll continue to maintain it as an educational project with bug-fixes,
polishing and minor features here and there, depending on the amount of time I can spend on it. The
good news is that I've already received good feedback from students attending to operating systems
courses. Therefore, at least in the educational world, I believe this project has some serious hope.

Contributing to the project
While I obviously don't expect hordes of contributors, as anyone who created an open source project,
I hope that someone will take interest in mine, even if by making small contributions. From my
side, I took care to make the whole thing as painless as possible with scripts, tests, documentation etc.
and I'm prepared to spend time helping contributors as well.

Tilck on Github
https://github.com/vvaltchev/tilck

A short presentation video
https://www.youtube.com/watch?v=Ce1pMlZO_mI

Screenshots

vvaltchev wrote: Tilck on Github
https://github.com/vvaltchev/tilck

A short presentation video
https://www.youtube.com/watch?v=Ce1pMlZO_mI

Can I just way, awesome work. I was looking at this last week, and this is right up my street.

thewrongchristian wrote: Can I just way, awesome work. I was looking at this last week, and this is right up my street.

Thank you, man!

Hi everyone!
I'd like to share a small update about things that changed in Tilck since the last time:

• Several bugfixes
• Added support for UBSAN
• Reduced potential UB in several places
• Improved considerably the built-in syscall tracer: better logs for some syscalls + tracing of signal delivery
• Made the tracer runnable outside of the debug panel, with dp -t
• Introduced trace_printk()
• Added partial support for POSIX reliable signals
• Implemented a graceful shutdown in `init`, thanks to newest signal support

I believe that people interested in Linux and/or classic monolithic UNIX systems, might enjoy taking a look.
Contributors and any kind of feedback are always appreciated

Super cool! I'm excited for whenever I implement tracing in my OS.

nexos wrote:Super cool!

Thanks!

nexos wrote:I'm excited for whenever I implement tracing in my OS.

It's good that you're excited about that!

About tracing I was excited mostly about the final result, as I needed it for debugging more complex applications like vim. I'm happy with my implementation but it was not fun writing it the whole time: there was plenty of non-avoidable per-syscall work. For example, in order to tracing to be useful, it's not enough to save the (at most) 6 params in some buffer. You'd need to know what those params are and what they might contain and in some cases even de-reference the pointers and read some or all of the data: because typically the tracing data is stored in a ring buffer and then read by a tool in an async way, if you don't de-reference pointers when the tracing happens, later you cannot do that. So, you need to define several param types and assign those types for each syscall for which you want detailed info. Also, for each trace event you need some buffers where to store the "partially rendered data". Finally, at the other side, you need additional code that knows for each param type, how to render it in a human-readable way. It's plenty of work to get a result as:
As you can see, while many parameters are "rendered", there are still other which aren't. I have all the infrastructure to make that possible, but I don't have enough any time and patience at the moment for that. I'm doing it gradually as I need more info about the params of a specific syscall. Take a look at the metadata:

https://github.com/vvaltchev/tilck/blob ... metadata.c

You might find interesting to take ideas from, when implementing your own tracing mechanism. When that happens, let me know how did you designed your tracing subsystem

vvaltchev wrote: As you can see, while many parameters are "rendered", there are still other which aren't. I have all the infrastructure to make that possible, but I don't have enough any time and patience at the moment for that. I'm doing it gradually as I need more info about the params of a specific syscall. Take a look at the metadata:

https://github.com/vvaltchev/tilck/blob ... metadata.c

You might find interesting to take ideas from, when implementing your own tracing mechanism. When that happens, let me know how did you designed your tracing subsystem

If you describe your syscalls in some more abstract description, you could get tracing for free by extending the syscall code generation. I already describe my syscalls in an abstract simple text format, from which I parse and generate the kernel and user side stubs, so adding tracing should be a relatively easy. My final solution will likely be XML with XSLT to generate the stubs, tracing should be some XSLT away.

Food for thought.

thewrongchristian wrote:If you describe your syscalls in some more abstract description, you could get tracing for free by extending the syscall code generation. I already describe my syscalls in an abstract simple text format, from which I parse and generate the kernel and user side stubs, so adding tracing should be a relatively easy. My final solution will likely be XML with XSLT to generate the stubs, tracing should be some XSLT away.

Food for thought.

You're making a good point here. I believe I seriously considered that approach at the time. With a two-step build, I could save myself the effort for writing the metadata for each syscall. But here's the problem: most of code is in the param types, not in the metadata itself. For example, if I wanted to dump ("render") a struct stat, I still will have to create a special type for it. Also, if I wanted to render a 32-bit flags parameter, same story: I have to know all the flags it supports and display them in a human "A | B | C" format.

You could argue that I could even parse with a clang plug-in (for example) the AST and generate metadata for structs as well. True. I even have the necessary experience to do that. But I still will miss the flags case (and maybe others too), because there's no formal metadata other than human documentation linking the parameter with those flag #defines.

Overall, considering the complexity in supporting a two-step build plus eventually some pre-generated metadata for the structs, I came up with the conclusion that's not worth the effort for my specific case. I don't have so much metadata to justify that level of abstraction, IMO. Also, I could never always generate the metadata for the structs on-the-fly, because I'd like to support multiple compilers etc. So, at least the struct metadata should be pre-generated in order the build to be easier to configure etc. I'm so happy with the approach of using pre-compiled toolchains. It would be a nightmare for me if a person would have to download the whole LLVM + clang (or GCC) and build it, just to try Tilck.

I honestly believe that downloading and building a huge toolchain stops some people from building and trying many open source projects. In particularly, when something goes wrong because developer's machine had some packages and mine doesn't and that's not written anywhere. Also, even if it's documented, something might still miss and still not work the first time. If I have to sit down and spend 2-3 hours working hard trying to build something just out of curiosity, I'll probably give up (unless, I really need to work on that project, of course).

Still, thanks for making that point. Maybe it's not the right solution for Tilck but.. for other projects, maybe written in languages with built-in reflection like D lang, that approach would certainly be very attractive. Unfortunately, my arguments still apply to C++, which despite a huge progress in the recent years, it's still far from having compile-time reflection. Having that would probably take another 10 years or so.

What you could do is have a Scheme or Perl script which writes out a syscall source file from a test file that describes each call. That wouldn't require a two step compile. I believe Linux has a Perl script for this.

Nicely done! You definitely should add this to the wiki's projects page if you haven't already.

BTW an idea for a mascot:

(just kidding, SG was the first that popped into my mind when I read the name, which is not bad at all.)

Cheers,
bzt

bzt wrote:Nicely done! You definitely should add this to the wiki's projects page if you haven't already.

Thanks, man! I actually mentioned the tracing feature in the debugging document, but maybe it deserves a dedicated document, I don't know. What do you think about?

bzt wrote:BTW an idea for a mascot:

(just kidding, SG was the first that popped into my mind when I read the name, which is not bad at all.)

Ahaahhaa Teal'c of Chulak
I hope that most people are not fans of SG because Teal'c with that creature inside him is kind of creepy for me

nexos wrote:What you could do is have a Scheme or Perl script which writes out a syscall source file from a test file that describes each call. That wouldn't require a two step compile. I believe Linux has a Perl script for this.

I could, but then I'd have to store in the source tree an "automatically generated file" and I'll have to re-commit it every time I change it. Mmmh.. maybe I would prefer the two-step build in this case.

Overall, I totally agree with you and @thewrongchristian that describing the syscalls in a higher level language and then use that to generate C code will have several advantages; it will reduce the current "redundancy" around syscalls, not only in the tracing module, but in other places as well. It would be nice to have all the signatures, the big "syscalls" array and the tracing metadata generated somehow for me; I like the DRY principle. Just, I hate over-engineering things and ending up trying to "kill a fly with a bazooka". So, I will have to dedicate some time thinking on that and figure out a lightweight solution perfectly in style with the rest of the project. Unfortunately, in no case it won't generate code for dumping structs or flags params (for the reasons I've mentioned before) but... it might still be a nice perk, I guess.

Also, I'll have to write the "script" in C or C++, as I don't wanna add more dependencies (like python or perl) to the build itself, but that won't be a big problem. I'll have to think carefully about this whole idea.

Thanks to both of you for making such suggestions

vvaltchev wrote:Thanks, man! I actually mentioned the tracing feature in the debugging document, but maybe it deserves a dedicated document, I don't know. What do you think about?

I meant add to OS Projects, but now I see it's already there. Good. About debugging well done, you've added all the missing commands to the protocol. Even though it's not GDB standard anymore, worth it, well done!

vvaltchev wrote:Ahaahhaa Teal'c of Chulak
I hope that most people are not fans of SG because Teal'c with that creature inside him is kind of creepy for me

Don't you worry, he got rid of that baby goauld. BTW it popped into my mind only because Tilck and Teal'c are pronounced the same in my language (we have a bad habit of "localizing" the pronounciation of foreign names).

Cheers,
bzt

bzt wrote:I meant add to OS Projects, but now I see it's already there. Good.

Ah, OK. Sorry, I misinterpreted.

bzt wrote:About debugging well done, you've added all the missing commands to the protocol. Even though it's not GDB standard anymore, worth it, well done!

Thanks again!

IMO, I feel that a lot is missing compared to GDB, but it offers a bunch of stuff not in GDB as well: in other words, neither one of the two can replace the other. When I need GDB, I use it through QEMU's integrated gdbserver. Clearly, that solution doesn't work on real hardware, but at the moment I was able to always somehow "get away" with that. A ton of asserts and fully-resolved backtraces helped a lot there. But yeah, sooner or later I might need something more powerful than that.

bzt wrote:Don't you worry, he got rid of that baby goauld.

Ahahah, good to know! I just watched a little the series and then gave up.

bzt wrote:BTW it popped into my mind only because Tilck and Teal'c are pronounced the same in my language (we have a bad habit of "localizing" the pronounciation of foreign names).

I don't remember how the name was pronounced in the series, but I pronounce it the same way. Google translate pronounces it almost the same way. But it doesn't matter, at the end. I think even like the whole joke around its name

If anybody is interested to learn more about this project, recently I gave a talk about it at Kernel Recipes 2022: https://youtu.be/tI0nnfuHsGs