ToaruOS-NIH: ToaruOS with no third-party components
ToaruOS-NIH: ToaruOS with no third-party components
A few years ago, after giving a talk about ToaruOS, someone came up to me and expressed some disappointment that ToaruOS used Newlib as a C library and employed Cairo and FreeType in its userspace. If it wasn't all entirely from scratch, they lamented, was it really my own OS? While I think their conclusion was a stretch, those comments have eaten at me in the back of my head for years, and with my work on mainline ToaruOS winding down after the release of 1.0, I thought maybe it was finally time to address them.
ToaruOS-NIH is a ToaruOS distribution with no third-party components. While the kernel remains mostly untouched (some PCI Database headers have been removed, and there was a general restructuring and deduplication of headers and datastructure implementations between the kernel and userspace), a lot has changed in the userspace. Gone is Newlib, replaced by a (very incomplete) in-house C library. Gone is Cairo, with the compositor now using its own SSE-accelerated alpha blitting as an extension and improvement on the existing ToaruOS graphics library (other consumers of Cairo were mostly Python applications that are still in the process of being rewritten, or were trivially replaced with the native graphics library). Gone, too, is FreeType, replaced with a very special text rasterization method. At the moment, Python is also missing - currently due to incompleteness of the C library, but also as I wanted to take this opportunity to port some of the more important functionality that was originally implemented in Python back to C for performance reasons, and to make it more widely accessible to C applications throughout the OS. Finally, GRUB has been replaced by a purpose-built in-house bootloader, though not much focus is placed in this area and it remains limited in its hardware support.
But ToaruOS-NIH is more than just removing third-party libraries. As functionality has been ported from Python to C, many improvements have been made to the operating system as a whole. The window decorator now includes context menus. New window management options have been added. Clipboard support is now present in the compositor, with the terminal supporting text selection. With the complete nested menu library ported from Python, applications like the terminal also have a menu bar available. ToaruOS-NIH also includes a completely new build system, which is much faster and better organized than the original ToaruOS build process.
Ultimately, the goal with ToaruOS-NIH is to replace the core of ToaruOS and eventually support all of the applications that were previously ported to the Newlib mainline distribution.
PS: With this announcement, ToaruOS is migrating to GitLab from Github.
ToaruOS-NIH is a ToaruOS distribution with no third-party components. While the kernel remains mostly untouched (some PCI Database headers have been removed, and there was a general restructuring and deduplication of headers and datastructure implementations between the kernel and userspace), a lot has changed in the userspace. Gone is Newlib, replaced by a (very incomplete) in-house C library. Gone is Cairo, with the compositor now using its own SSE-accelerated alpha blitting as an extension and improvement on the existing ToaruOS graphics library (other consumers of Cairo were mostly Python applications that are still in the process of being rewritten, or were trivially replaced with the native graphics library). Gone, too, is FreeType, replaced with a very special text rasterization method. At the moment, Python is also missing - currently due to incompleteness of the C library, but also as I wanted to take this opportunity to port some of the more important functionality that was originally implemented in Python back to C for performance reasons, and to make it more widely accessible to C applications throughout the OS. Finally, GRUB has been replaced by a purpose-built in-house bootloader, though not much focus is placed in this area and it remains limited in its hardware support.
But ToaruOS-NIH is more than just removing third-party libraries. As functionality has been ported from Python to C, many improvements have been made to the operating system as a whole. The window decorator now includes context menus. New window management options have been added. Clipboard support is now present in the compositor, with the terminal supporting text selection. With the complete nested menu library ported from Python, applications like the terminal also have a menu bar available. ToaruOS-NIH also includes a completely new build system, which is much faster and better organized than the original ToaruOS build process.
Ultimately, the goal with ToaruOS-NIH is to replace the core of ToaruOS and eventually support all of the applications that were previously ported to the Newlib mainline distribution.
PS: With this announcement, ToaruOS is migrating to GitLab from Github.
Re: ToaruOS-NIH: ToaruOS with no third-party components
I have to admit that I sometimes thought about ToaruOS: The user-space is nice but except for the compositing system, it is all ported and the system-level stuff is nothing special. However, your NIH port proves that wrong. It is a very impressive amount of code that you produced there; I have much respect for the NIH port. I'm especially impressed that you could replace Cairo and FreeType without affecting visual quality. How does the performance of your own graphics routines compare to Cairo's and pixman's optimized routines?
Also: Is there any image one can download to test the NIH port? Gitlab is very slow right now, probably because of increased traffic caused by recent Github announcements...
Also: Is there any image one can download to test the NIH port? Gitlab is very slow right now, probably because of increased traffic caused by recent Github announcements...
managarm: Microkernel-based OS capable of running a Wayland desktop (Discord: https://discord.gg/7WB6Ur3). My OS-dev projects: [mlibc: Portable C library for managarm, qword, Linux, Sigma, ...] [LAI: AML interpreter] [xbstrap: Build system for OS distributions].
Re: ToaruOS-NIH: ToaruOS with no third-party components
I'm happy to announce that ToaruOS-NIH's C library has reached the point where we can build Python again, including support for the ctypes module. Even the Yutani (windowing system) bindings are working.
Poorly, I'm afraid. Cairo does some really amazing tricks to ensure efficient rendering, and I'm not quite there yet.Korona wrote:How does the performance of your own graphics routines compare to Cairo's and pixman's optimized routines?
I have a new build with Python available here - it's a bit big and takes some time to boot.Korona wrote:Also: Is there any image one can download to test the NIH port? Gitlab is very slow right now, probably because of increased traffic caused by recent Github announcements...
Re: ToaruOS-NIH: ToaruOS with no third-party components
I checked it out and the performance seems to be good on my machine. Only when I start >4 of those plasma demos, compositor lag becomes an issue (i.e. it gets noticeably slower to move around windows). That's better than I expected - I suspected that it would be very hard to beat Cairo. Certainly very nicely done.
managarm: Microkernel-based OS capable of running a Wayland desktop (Discord: https://discord.gg/7WB6Ur3). My OS-dev projects: [mlibc: Portable C library for managarm, qword, Linux, Sigma, ...] [LAI: AML interpreter] [xbstrap: Build system for OS distributions].
Re: ToaruOS-NIH: ToaruOS with no third-party components
It's been a little over a month since I updated this thread, so here's what's new since then:
What's New
- The netboot loader (now called "netinit") is back, though missing gzip and TLS support.
- An EFI bootloader is now available, in both IA32 and X64 forms, so even if you have an old 32-bit EFI system it should work. The EFI bootloader is much more robust than the BIOS loader, and has been tested on real hardware.
- Significant improvements have been made to the C standard library implementation for compliance with the standards.
- CDs are now built as ISO/FAT hybrids and contain both the BIOS loader as an El Torito payload as well as the EFI loaders. The ISO filesystem describes the files contained with the FAT image.
- Several more applications from mainline have been ported over.
- A number of bug fixes have been implemented in the kernel and driver modules.
- A headless boot mode is now available, along with tools to run login shells on serial ports.
- Support for VirtualBox's Seamless mode and hardware cursor rendering have been added.
- With the addition of PAT support, the framebuffer drivers now function properly on real hardware.
- A number of changes have been made to the init system for better session management.
- New commands have been added: killall, pidof, true, false, pwd; sudo -s, uname -o
- New shell features have been added: redirecting stdout to a file, tab completion for variable names, argument variables, last command return code variable
- Menus now support keyboard interaction (as they did in mainline's Python implementation)
- The old C graphical login screen has been revived and ported to work with the new glogin system.
Roadmap
Recently, I've been focused on "quality of life" improvements for the OS - adding more commands, making the system feel more like a "real" OS. Moving forward, I'd like to revive support for hard disk installations and write an installer application.
Downloads
Volatile releases are available from my website.
What's New
- The netboot loader (now called "netinit") is back, though missing gzip and TLS support.
- An EFI bootloader is now available, in both IA32 and X64 forms, so even if you have an old 32-bit EFI system it should work. The EFI bootloader is much more robust than the BIOS loader, and has been tested on real hardware.
- Significant improvements have been made to the C standard library implementation for compliance with the standards.
- CDs are now built as ISO/FAT hybrids and contain both the BIOS loader as an El Torito payload as well as the EFI loaders. The ISO filesystem describes the files contained with the FAT image.
- Several more applications from mainline have been ported over.
- A number of bug fixes have been implemented in the kernel and driver modules.
- A headless boot mode is now available, along with tools to run login shells on serial ports.
- Support for VirtualBox's Seamless mode and hardware cursor rendering have been added.
- With the addition of PAT support, the framebuffer drivers now function properly on real hardware.
- A number of changes have been made to the init system for better session management.
- New commands have been added: killall, pidof, true, false, pwd; sudo -s, uname -o
- New shell features have been added: redirecting stdout to a file, tab completion for variable names, argument variables, last command return code variable
- Menus now support keyboard interaction (as they did in mainline's Python implementation)
- The old C graphical login screen has been revived and ported to work with the new glogin system.
Roadmap
Recently, I've been focused on "quality of life" improvements for the OS - adding more commands, making the system feel more like a "real" OS. Moving forward, I'd like to revive support for hard disk installations and write an installer application.
Downloads
Volatile releases are available from my website.
Re: ToaruOS-NIH: ToaruOS with no third-party components
Looks very promising.
I prefer this version over the other one just because you ditched all the third-party components.
I tested it in both the Virtual Box and Qemu and I'm amazed by how fast it is. Maybe it's just my beefy PC or you really know how to optimize a compositor.
How do I test it on real hardware btw? I would like to put it on a stick, what should I do? (or to just find an empty CD and directly burn it??)
I prefer this version over the other one just because you ditched all the third-party components.
I tested it in both the Virtual Box and Qemu and I'm amazed by how fast it is. Maybe it's just my beefy PC or you really know how to optimize a compositor.
How do I test it on real hardware btw? I would like to put it on a stick, what should I do? (or to just find an empty CD and directly burn it??)
OS: Basic OS
About: 32 Bit Monolithic Kernel Written in C++ and Assembly, Custom FAT 32 Bootloader
About: 32 Bit Monolithic Kernel Written in C++ and Assembly, Custom FAT 32 Bootloader
Re: ToaruOS-NIH: ToaruOS with no third-party components
You could burn the ISO to a CD, that should work. Boot it with EFI for best results - the BIOS loader is only built for QEMU. You'll need to select a video mode option as the default is to let the native drivers do modesetting and those only work in VMs at the moment. Alternatively, you could copy the kernel, ramdisk, and modules directory from the CD into the root of an existing EFI system partition, along with the the appropriate EFI loader (probably bootx64.efi - rename it to something more identifiable) - this is what I do normally for testing (I use rEFInd on my desktop as a boot menu, and it detects my loader and provides a menu entry for it). You may also want to disable all the extra hardware drivers, as shown here, just to ensure none of them interfere (you won't need them to run the desktop, and you likely don't have the hardware for the network or audio drivers anyway).Octacone wrote:How do I test it on real hardware btw? I would like to put it on a stick, what should I do? (or to just find an empty CD and directly burn it??)
There's a little bit of SSE magic at play, but it's still nothing compared to the performance I was getting with Cairo... *sigh*Octacone wrote:Maybe it's just my beefy PC or you really know how to optimize a compositor
Re: ToaruOS-NIH: ToaruOS with no third-party components
Framebuffer address is 0x00000000d1000000klange wrote:You could burn the ISO to a CD, that should work. Boot it with EFI for best results - the BIOS loader is only built for QEMU. You'll need to select a video mode option as the default is to let the native drivers do modesetting and those only work in VMs at the moment. Alternatively, you could copy the kernel, ramdisk, and modules directory from the CD into the root of an existing EFI system partition, along with the the appropriate EFI loader (probably bootx64.efi - rename it to something more identifiable) - this is what I do normally for testing (I use rEFInd on my desktop as a boot menu, and it detects my loader and provides a menu entry for it). You may also want to disable all the extra hardware drivers, as shown here, just to ensure none of them interfere (you won't need them to run the desktop, and you likely don't have the hardware for the network or audio drivers anyway).
Mode information passed to multiboot:
Address: 0xd100000
Width: 1024
Height: 768
Exiting boot services and jumping to 00100055 with mboot_mag=2badboo2 and mboot_
ptr=02ea3000...
&_xmain = 00000000768b2e50
and just stops. I tried switching to legacy, nothing changed.
Tested on two machines (one is from 2017, one is from 2003), neither worked.
OS: Basic OS
About: 32 Bit Monolithic Kernel Written in C++ and Assembly, Custom FAT 32 Bootloader
About: 32 Bit Monolithic Kernel Written in C++ and Assembly, Custom FAT 32 Bootloader
Re: ToaruOS-NIH: ToaruOS with no third-party components
You have a UEFI machine from 2003? That seems... unlikely given that UEFI is from 2005.Octacone wrote:and just stops. I tried switching to legacy, nothing changed.
Tested on two machines (one is from 2017, one is from 2003), neither worked.
The 2017 machine may be having issues jumping back to protected mode... I will need to investigate.
e: I may have accidentally told you to turn off the video modules - definitely turn those back on.
e2: I'm investigating issues with booting on my Surface, which seems to be failing to exit EFI boot services, but then also fails later after loading the kernel.
Re: ToaruOS-NIH: ToaruOS with no third-party components
lol, that thing can barely boot, let alone have UEFI.klange wrote:You have a UEFI machine from 2003? That seems... unlikely given that UEFI is from 2005.Octacone wrote:and just stops. I tried switching to legacy, nothing changed.
Tested on two machines (one is from 2017, one is from 2003), neither worked.
The 2017 machine may be having issues jumping back to protected mode... I will need to investigate.
e: I may have accidentally told you to turn off the video modules - definitely turn those back on.
e2: I'm investigating issues with booting on my Surface, which seems to be failing to exit EFI boot services, but then also fails later after loading the kernel.
I figured it out, but everything stayed the same.
Is 512 MB of RAM enough to run the thing? I guess I won't be having a pleasant experience, but at least some VGA (text mode) would be nice.
OS: Basic OS
About: 32 Bit Monolithic Kernel Written in C++ and Assembly, Custom FAT 32 Bootloader
About: 32 Bit Monolithic Kernel Written in C++ and Assembly, Custom FAT 32 Bootloader
Re: ToaruOS-NIH: ToaruOS with no third-party components
The BIOS machine is probably not going to work with my loaders, but you can try to set up GRUB 512MB should be enough.Octacone wrote:lol, that thing can barely boot, let alone have UEFI.
I figured it out, but everything stayed the same.
Is 512 MB of RAM enough to run the thing? I guess I won't be having a pleasant experience, but at least some VGA (text mode) would be nice.
I'll get back to you on the EFI machine if I get my Surface working. I have a debug build you can try that may get me more information; I'll upload that somewhere when I get ~to work~ home.
Re: ToaruOS-NIH: ToaruOS with no third-party components
Any luck?klange wrote:The BIOS machine is probably not going to work with my loaders, but you can try to set up GRUB 512MB should be enough.Octacone wrote:lol, that thing can barely boot, let alone have UEFI.
I figured it out, but everything stayed the same.
Is 512 MB of RAM enough to run the thing? I guess I won't be having a pleasant experience, but at least some VGA (text mode) would be nice.
I'll get back to you on the EFI machine if I get my Surface working. I have a debug build you can try that may get me more information; I'll upload that somewhere when I get ~to work~ home.
Btw I managed to compile your toolchain, +1 for the simplicity.
Right now I'm messing around with your OS. Dude it took me like 10 minutes to find a way to change the terminal background color.
Still couldn't find the name of that gray/brownish thing that sits below the application title.
OS: Basic OS
About: 32 Bit Monolithic Kernel Written in C++ and Assembly, Custom FAT 32 Bootloader
About: 32 Bit Monolithic Kernel Written in C++ and Assembly, Custom FAT 32 Bootloader
Re: ToaruOS-NIH: ToaruOS with no third-party components
Sorry, haven't had much luck... Surface had a problem with ExitBootServices, but I don't have a solution for it yet (and the kernel was seizing later during boot anyway) - will look into it more this weekend.Octacone wrote:Any luck?
Yeah, a huge benefit of a from-scratch OS is all you really need is the compiler. I'm hoping to strip more of the required utilities away from the build to help get things closer to being self-hosted with just a gcc+binutils port...Octacone wrote:Btw I managed to compile your toolchain, +1 for the simplicity.
I don't know how to take that... a suggestion that I improve themability? The terminal in particular isn't specifically set up for good color theming.Octacone wrote:Right now I'm messing around with your OS. Dude it took me like 10 minutes to find a way to change the terminal background color.
Menu bar. Color is currently hardcode in lib/menu.c:menu_bar_render, default is #3B3B3B (so while it may appear brownish, it is actually a pure gray), which matches the color of the default decoration themes.Octacone wrote:Still couldn't find the name of that gray/brownish thing that sits below the application title.