ToaruOS-NIH: ToaruOS with no third-party components

This forums is for OS project announcements including project openings, new releases, update notices, test requests, and job openings (both paying and volunteer).
Post Reply
klange
Member
Member
Posts: 679
Joined: Wed Mar 30, 2011 12:31 am
Libera.chat IRC: klange
Discord: klange

ToaruOS-NIH: ToaruOS with no third-party components

Post by klange »

A few years ago, after giving a talk about ToaruOS, someone came up to me and expressed some disappointment that ToaruOS used Newlib as a C library and employed Cairo and FreeType in its userspace. If it wasn't all entirely from scratch, they lamented, was it really my own OS? While I think their conclusion was a stretch, those comments have eaten at me in the back of my head for years, and with my work on mainline ToaruOS winding down after the release of 1.0, I thought maybe it was finally time to address them.

Image

ToaruOS-NIH is a ToaruOS distribution with no third-party components. While the kernel remains mostly untouched (some PCI Database headers have been removed, and there was a general restructuring and deduplication of headers and datastructure implementations between the kernel and userspace), a lot has changed in the userspace. Gone is Newlib, replaced by a (very incomplete) in-house C library. Gone is Cairo, with the compositor now using its own SSE-accelerated alpha blitting as an extension and improvement on the existing ToaruOS graphics library (other consumers of Cairo were mostly Python applications that are still in the process of being rewritten, or were trivially replaced with the native graphics library). Gone, too, is FreeType, replaced with a very special text rasterization method. At the moment, Python is also missing - currently due to incompleteness of the C library, but also as I wanted to take this opportunity to port some of the more important functionality that was originally implemented in Python back to C for performance reasons, and to make it more widely accessible to C applications throughout the OS. Finally, GRUB has been replaced by a purpose-built in-house bootloader, though not much focus is placed in this area and it remains limited in its hardware support.

But ToaruOS-NIH is more than just removing third-party libraries. As functionality has been ported from Python to C, many improvements have been made to the operating system as a whole. The window decorator now includes context menus. New window management options have been added. Clipboard support is now present in the compositor, with the terminal supporting text selection. With the complete nested menu library ported from Python, applications like the terminal also have a menu bar available. ToaruOS-NIH also includes a completely new build system, which is much faster and better organized than the original ToaruOS build process.

Ultimately, the goal with ToaruOS-NIH is to replace the core of ToaruOS and eventually support all of the applications that were previously ported to the Newlib mainline distribution.

PS: With this announcement, ToaruOS is migrating to GitLab from Github.
Korona
Member
Member
Posts: 1000
Joined: Thu May 17, 2007 1:27 pm
Contact:

Re: ToaruOS-NIH: ToaruOS with no third-party components

Post by Korona »

I have to admit that I sometimes thought about ToaruOS: The user-space is nice but except for the compositing system, it is all ported and the system-level stuff is nothing special. However, your NIH port proves that wrong. It is a very impressive amount of code that you produced there; I have much respect for the NIH port. I'm especially impressed that you could replace Cairo and FreeType without affecting visual quality. How does the performance of your own graphics routines compare to Cairo's and pixman's optimized routines?

Also: Is there any image one can download to test the NIH port? Gitlab is very slow right now, probably because of increased traffic caused by recent Github announcements...
managarm: Microkernel-based OS capable of running a Wayland desktop (Discord: https://discord.gg/7WB6Ur3). My OS-dev projects: [mlibc: Portable C library for managarm, qword, Linux, Sigma, ...] [LAI: AML interpreter] [xbstrap: Build system for OS distributions].
klange
Member
Member
Posts: 679
Joined: Wed Mar 30, 2011 12:31 am
Libera.chat IRC: klange
Discord: klange

Re: ToaruOS-NIH: ToaruOS with no third-party components

Post by klange »

I'm happy to announce that ToaruOS-NIH's C library has reached the point where we can build Python again, including support for the ctypes module. Even the Yutani (windowing system) bindings are working.
Korona wrote:How does the performance of your own graphics routines compare to Cairo's and pixman's optimized routines?
Poorly, I'm afraid. Cairo does some really amazing tricks to ensure efficient rendering, and I'm not quite there yet.
Korona wrote:Also: Is there any image one can download to test the NIH port? Gitlab is very slow right now, probably because of increased traffic caused by recent Github announcements...
I have a new build with Python available here - it's a bit big and takes some time to boot.
Korona
Member
Member
Posts: 1000
Joined: Thu May 17, 2007 1:27 pm
Contact:

Re: ToaruOS-NIH: ToaruOS with no third-party components

Post by Korona »

I checked it out and the performance seems to be good on my machine. Only when I start >4 of those plasma demos, compositor lag becomes an issue (i.e. it gets noticeably slower to move around windows). That's better than I expected - I suspected that it would be very hard to beat Cairo. Certainly very nicely done.
managarm: Microkernel-based OS capable of running a Wayland desktop (Discord: https://discord.gg/7WB6Ur3). My OS-dev projects: [mlibc: Portable C library for managarm, qword, Linux, Sigma, ...] [LAI: AML interpreter] [xbstrap: Build system for OS distributions].
klange
Member
Member
Posts: 679
Joined: Wed Mar 30, 2011 12:31 am
Libera.chat IRC: klange
Discord: klange

Re: ToaruOS-NIH: ToaruOS with no third-party components

Post by klange »

It's been a little over a month since I updated this thread, so here's what's new since then:

What's New
- The netboot loader (now called "netinit") is back, though missing gzip and TLS support.
- An EFI bootloader is now available, in both IA32 and X64 forms, so even if you have an old 32-bit EFI system it should work. The EFI bootloader is much more robust than the BIOS loader, and has been tested on real hardware.
- Significant improvements have been made to the C standard library implementation for compliance with the standards.
- CDs are now built as ISO/FAT hybrids and contain both the BIOS loader as an El Torito payload as well as the EFI loaders. The ISO filesystem describes the files contained with the FAT image.
- Several more applications from mainline have been ported over.
- A number of bug fixes have been implemented in the kernel and driver modules.
- A headless boot mode is now available, along with tools to run login shells on serial ports.
- Support for VirtualBox's Seamless mode and hardware cursor rendering have been added.
- With the addition of PAT support, the framebuffer drivers now function properly on real hardware.
- A number of changes have been made to the init system for better session management.
- New commands have been added: killall, pidof, true, false, pwd; sudo -s, uname -o
- New shell features have been added: redirecting stdout to a file, tab completion for variable names, argument variables, last command return code variable
- Menus now support keyboard interaction (as they did in mainline's Python implementation)
- The old C graphical login screen has been revived and ported to work with the new glogin system.

Roadmap
Recently, I've been focused on "quality of life" improvements for the OS - adding more commands, making the system feel more like a "real" OS. Moving forward, I'd like to revive support for hard disk installations and write an installer application.

Downloads
Volatile releases are available from my website.
User avatar
Octacone
Member
Member
Posts: 1138
Joined: Fri Aug 07, 2015 6:13 am

Re: ToaruOS-NIH: ToaruOS with no third-party components

Post by Octacone »

Looks very promising.
I prefer this version over the other one just because you ditched all the third-party components.
I tested it in both the Virtual Box and Qemu and I'm amazed by how fast it is. Maybe it's just my beefy PC or you really know how to optimize a compositor.
How do I test it on real hardware btw? I would like to put it on a stick, what should I do? (or to just find an empty CD and directly burn it??)
OS: Basic OS
About: 32 Bit Monolithic Kernel Written in C++ and Assembly, Custom FAT 32 Bootloader
klange
Member
Member
Posts: 679
Joined: Wed Mar 30, 2011 12:31 am
Libera.chat IRC: klange
Discord: klange

Re: ToaruOS-NIH: ToaruOS with no third-party components

Post by klange »

Octacone wrote:How do I test it on real hardware btw? I would like to put it on a stick, what should I do? (or to just find an empty CD and directly burn it??)
You could burn the ISO to a CD, that should work. Boot it with EFI for best results - the BIOS loader is only built for QEMU. You'll need to select a video mode option as the default is to let the native drivers do modesetting and those only work in VMs at the moment. Alternatively, you could copy the kernel, ramdisk, and modules directory from the CD into the root of an existing EFI system partition, along with the the appropriate EFI loader (probably bootx64.efi - rename it to something more identifiable) - this is what I do normally for testing (I use rEFInd on my desktop as a boot menu, and it detects my loader and provides a menu entry for it). You may also want to disable all the extra hardware drivers, as shown here, just to ensure none of them interfere (you won't need them to run the desktop, and you likely don't have the hardware for the network or audio drivers anyway).
Octacone wrote:Maybe it's just my beefy PC or you really know how to optimize a compositor
There's a little bit of SSE magic at play, but it's still nothing compared to the performance I was getting with Cairo... *sigh*
User avatar
Octacone
Member
Member
Posts: 1138
Joined: Fri Aug 07, 2015 6:13 am

Re: ToaruOS-NIH: ToaruOS with no third-party components

Post by Octacone »

klange wrote:You could burn the ISO to a CD, that should work. Boot it with EFI for best results - the BIOS loader is only built for QEMU. You'll need to select a video mode option as the default is to let the native drivers do modesetting and those only work in VMs at the moment. Alternatively, you could copy the kernel, ramdisk, and modules directory from the CD into the root of an existing EFI system partition, along with the the appropriate EFI loader (probably bootx64.efi - rename it to something more identifiable) - this is what I do normally for testing (I use rEFInd on my desktop as a boot menu, and it detects my loader and provides a menu entry for it). You may also want to disable all the extra hardware drivers, as shown here, just to ensure none of them interfere (you won't need them to run the desktop, and you likely don't have the hardware for the network or audio drivers anyway).
Framebuffer address is 0x00000000d1000000
Mode information passed to multiboot:
Address: 0xd100000
Width: 1024
Height: 768

Exiting boot services and jumping to 00100055 with mboot_mag=2badboo2 and mboot_
ptr=02ea3000...
&_xmain = 00000000768b2e50

and just stops. I tried switching to legacy, nothing changed.
Tested on two machines (one is from 2017, one is from 2003), neither worked.
OS: Basic OS
About: 32 Bit Monolithic Kernel Written in C++ and Assembly, Custom FAT 32 Bootloader
klange
Member
Member
Posts: 679
Joined: Wed Mar 30, 2011 12:31 am
Libera.chat IRC: klange
Discord: klange

Re: ToaruOS-NIH: ToaruOS with no third-party components

Post by klange »

Octacone wrote:and just stops. I tried switching to legacy, nothing changed.
Tested on two machines (one is from 2017, one is from 2003), neither worked.
You have a UEFI machine from 2003? That seems... unlikely given that UEFI is from 2005.

The 2017 machine may be having issues jumping back to protected mode... I will need to investigate.

e: I may have accidentally told you to turn off the video modules - definitely turn those back on.

e2: I'm investigating issues with booting on my Surface, which seems to be failing to exit EFI boot services, but then also fails later after loading the kernel.
User avatar
Octacone
Member
Member
Posts: 1138
Joined: Fri Aug 07, 2015 6:13 am

Re: ToaruOS-NIH: ToaruOS with no third-party components

Post by Octacone »

klange wrote:
Octacone wrote:and just stops. I tried switching to legacy, nothing changed.
Tested on two machines (one is from 2017, one is from 2003), neither worked.
You have a UEFI machine from 2003? That seems... unlikely given that UEFI is from 2005.

The 2017 machine may be having issues jumping back to protected mode... I will need to investigate.

e: I may have accidentally told you to turn off the video modules - definitely turn those back on.

e2: I'm investigating issues with booting on my Surface, which seems to be failing to exit EFI boot services, but then also fails later after loading the kernel.
lol, that thing can barely boot, let alone have UEFI.

I figured it out, but everything stayed the same.

Is 512 MB of RAM enough to run the thing? I guess I won't be having a pleasant experience, but at least some VGA (text mode) would be nice.
OS: Basic OS
About: 32 Bit Monolithic Kernel Written in C++ and Assembly, Custom FAT 32 Bootloader
klange
Member
Member
Posts: 679
Joined: Wed Mar 30, 2011 12:31 am
Libera.chat IRC: klange
Discord: klange

Re: ToaruOS-NIH: ToaruOS with no third-party components

Post by klange »

Octacone wrote:lol, that thing can barely boot, let alone have UEFI.

I figured it out, but everything stayed the same.

Is 512 MB of RAM enough to run the thing? I guess I won't be having a pleasant experience, but at least some VGA (text mode) would be nice.
The BIOS machine is probably not going to work with my loaders, but you can try to set up GRUB :) 512MB should be enough.

I'll get back to you on the EFI machine if I get my Surface working. I have a debug build you can try that may get me more information; I'll upload that somewhere when I get ~to work~ home.
User avatar
Octacone
Member
Member
Posts: 1138
Joined: Fri Aug 07, 2015 6:13 am

Re: ToaruOS-NIH: ToaruOS with no third-party components

Post by Octacone »

klange wrote:
Octacone wrote:lol, that thing can barely boot, let alone have UEFI.

I figured it out, but everything stayed the same.

Is 512 MB of RAM enough to run the thing? I guess I won't be having a pleasant experience, but at least some VGA (text mode) would be nice.
The BIOS machine is probably not going to work with my loaders, but you can try to set up GRUB :) 512MB should be enough.

I'll get back to you on the EFI machine if I get my Surface working. I have a debug build you can try that may get me more information; I'll upload that somewhere when I get ~to work~ home.
Any luck?
Btw I managed to compile your toolchain, +1 for the simplicity.
Right now I'm messing around with your OS. Dude it took me like 10 minutes to find a way to change the terminal background color.
Still couldn't find the name of that gray/brownish thing that sits below the application title.
OS: Basic OS
About: 32 Bit Monolithic Kernel Written in C++ and Assembly, Custom FAT 32 Bootloader
klange
Member
Member
Posts: 679
Joined: Wed Mar 30, 2011 12:31 am
Libera.chat IRC: klange
Discord: klange

Re: ToaruOS-NIH: ToaruOS with no third-party components

Post by klange »

Octacone wrote:Any luck?
Sorry, haven't had much luck... Surface had a problem with ExitBootServices, but I don't have a solution for it yet (and the kernel was seizing later during boot anyway) - will look into it more this weekend.
Octacone wrote:Btw I managed to compile your toolchain, +1 for the simplicity.
Yeah, a huge benefit of a from-scratch OS is all you really need is the compiler. I'm hoping to strip more of the required utilities away from the build to help get things closer to being self-hosted with just a gcc+binutils port...
Octacone wrote:Right now I'm messing around with your OS. Dude it took me like 10 minutes to find a way to change the terminal background color.
I don't know how to take that... a suggestion that I improve themability? The terminal in particular isn't specifically set up for good color theming.
Octacone wrote:Still couldn't find the name of that gray/brownish thing that sits below the application title.
Menu bar. Color is currently hardcode in lib/menu.c:menu_bar_render, default is #3B3B3B (so while it may appear brownish, it is actually a pure gray), which matches the color of the default decoration themes.
Post Reply