Thank you for the wiki

[cardboardaardvark]

Hi,

I'm new. New to the forum and new to OS development. I wanted to say thank you very much to the people who contributed to the wiki. It was extremely helpful for getting this system administrator with a lot of Perl experience and some C++ experience to the point of having the start of a toy kernel implemented in C++ with runtime features actually ticking away counting uptime and showing a clock on the VGA screen.

It wasn't a quick process to get here but the wiki was so much of a time saver. Perfect way to get started with the x86 GDT/IDT/PTD/PTE so I can actually see things happening with out having to wrap my head around the official Intel references which can be a little, ahem, overwhelming at the very start. I never went to college for CS (I studied welding instead) and I swear I got about a quarter's worth of education out of the wiki in only a few weeks.

So thanks again from this grizzled sysadmin starting his new hobby of trying to come up with an OS design that isn't just another Unix clone.

Thanks again. And here's a screenshot of TalOS (yes, I've been playing too much Skyrim) ticking away inside qemu:

[thewrongchristian]

So thanks again from this grizzled sysadmin starting his new hobby of trying to come up with an OS design that isn't just another Unix clone.

Welcome to the club. Be prepared for it to suck your time out of your life if you stay interested!

BTW, you can't go much wrong with being just another Unix clone. As you do stuff, you'll get "a-ha" moments as you realize why Unix does things the way it does (you'll also get many many WTF moments as well, Unix isn't perfect!) And you'll have a ready source of user software to port, which may not be the case if you're using a non-Unix like API.

[cardboardaardvark]

BTW, you can't go much wrong with being just another Unix clone. As you do stuff, you'll get "a-ha" moments as you realize why Unix does things the way it does (you'll also get many many WTF moments as well, Unix isn't perfect!) And you'll have a ready source of user software to port, which may not be the case if you're using a non-Unix like API.

Aye, I've thought about the lack of existing software to port if I don't supply a POSIX API and how that is going to be a detriment. Probably going to wind up with no one but me using it. But also, even if I do make another Unix clone, probably no one but me will ever be using it.

I saw a post around here yesterday where the OP was pointing out most people just get a clock going and walk away. And if anyone goes past that they wind up making a tiny Unix implementation and then walk away. No one actually tries anything different.

It's likely not as bad as "no one" ever tries to do something different but I'm not surprised to see that it is rare. I've got some ideas and it is really what I wanted to explore. As I've been reading about kernel design and from the little I know about Windows I find that I'm more inclined to think in terms of how Windows/VMS solves problems than how Unix does it. Which is rather surprising to me seeing as I've been using Unix as my primary workstation since 1994 and owned my own personal Sun Ultra 5 which I also used as my primary workstation.

I'm interested to see where a Windows/Unix Frankenstein winds up. One thing I very much want to do better than POSIX is how permissions work. The base model is too simplistic, extended permission attributes on files don't go far enough, and all the MAC choices for Linux and FreeBSD leave a lot to be desired.

[eekee]

Good to see another OS which isn't just another Unix clone. I've been thinking about the hardware Unix first ran on; 32KB RAM & 3 1GB disks in the PDP-11, and wondering quite how much of modern Unix's weirdness goes back to that strange ratio. Parent processes cleaning up after dead children might be part of it, it makes sense to put that task in userspace which could be swapped out. They didn't write that particular oddity into Plan 9, but it had plenty of other weirdness in pursuit of small code size for some reason.

I'm going for a unique OS design myself, and I've been looking at odd languages in the hope of speeding development: Forth and "plain English programming". Neither has worked out too well for me, though they might for others. I was drawn to Forth by someone who said it was much faster than C++ coding. Professionally, he wrote libraries for C++ systems, but he wrote them in Forth in well under half the expected time. I however struggled to manage strings and other data structures, thinking the minimal facilities the language provided were enough. They weren't, and I needed to learn how to design my own. You need to understand how to design a lot of different things to be a fast Forth programmer. I would have learned faster if I'd used a Forth system with more modern facilities, seeing how they implemented things.

Plain English programming with CAL-4700 is essentially compiled pseudocode. Here, my problem is that I don't think the same way as the compiler authors, and thus it's taking time to learn to espress my self in their pseudocode. On the positive side, it was very easy to learn to read the code of the IDE and compiler; a big set of real-world examples.

[cardboardaardvark]

I've been thinking about the hardware Unix first ran on; 32KB RAM & 3 1GB disks in the PDP-11, and wondering quite how much of modern Unix's weirdness goes back to that strange ratio.

A while ago I started a project that needed a microcontroller to drive a machine I was building. I grabbed an Arduino, tossed out their IDE/runtime, and targeted it directly with gcc and avr-libc. I was quite surprised to see so much Unix coming out of avr-libc. Then it clicked: this $5 microcontroller I'm working with is a substantial fraction of the hardware that Unix was originally created for. Pretty amazing.

Good to see another OS which isn't just another Unix clone.

I'm giving it a shot anyway. I'm still at the phase where it is a drastically over complicated clock. I'm moving ahead though. The kernel main loop now pulls jobs out of a queue and if no job is present does a hlt. I multiplexed the PIT and wrapped it with an interface that lets you create timers with arbitrary initial and reoccurring delays with a precision of 1ms. In the interest of minimizing the amount of time where interrupts are disabled the PIT ISR checks to see if it already has a pending job and if not puts one in the queue to check for expired timers then exits. In the interest of future concurrency if expired timers are found they too are submitted as jobs to run.

I broke the status bar in the console into two parts: memory info and the clock and the memory info updates more frequently than the clock. From left to right the memory info goes: size of kernel program break, available physical pages, total physical pages:

I've been looking at odd languages in the hope of speeding development

I've got the same goal. What I really want is a kernel that lets me play around with experimental APIs in user space and I'm happy to sacrifice performance and a bunch of other things to get there. I don't expect my first try at making a kernel to wind up anywhere beyond proof of concept quality, not even prototype, so even if successful at coming up with a new user space API I like I fully expect to throw the kernel away in the future anyway. Might as well embrace that.

At the moment I've got newlib and GCC 14.2's libstdc++ running in hosted mode in the kernel and I'm running GCC with C++23 as the language version. It definitely lets me move a lot faster than C. I'm traditionally a Perl programmer and lately more of a Python programmer and, unfortunately, I definitely can't move as fast with C++ as I can with those languages.

I am very very tempted to run an experiment and see how hard it would be to get Perl running in kernel space. Though while I'm still down at such a low level I don't yet see any benefit with Perl beyond std::shared_ptr, std::string, std::vector, std::list, etc. Maybe in the future.

My current task is getting ready to have the kernel be able to execute user mode processes. I've implemented the trampoline described over here in assembly and got it running with the linker placing it at the normal place. I've built a linker script to park the trampoline only at the top 1 megabyte of the address space as talked about here (the trampoline when stripped of debug info is less than 1KiB, which amuses me).

My current sub-task is finding the ELF section header for the trampoline so I can get the physical and virtual addresses for it and map them in the page table directory. It's been more difficult than I expected to find the section name strings but I expect to solve that soon, get the trampoline mapped, and start finding the new things that will be more difficult than I expected.

Once I got the PIT multiplexed I decided to run it at a silly frequency (5khz, I might turn it up more) to expose reentrancy issues early. Wow. Never in my life have I seen malloc() go reentrant. That gave me a good laugh.

I've already got spinlocks implemented and present in the interest of future concurrency. So going reentrant isn't catastrophic. It does a cause a deadlock but also leaves a nice easy to debug stack trace for GDB. I solve the reentrancy issue at the moment by disabling interrupts in sections of code that are common inside and outside of ISRs.

After I solve spreading my trampoline around in the address space I think I'll go back and make a class of spin lock specifically for parts of the kernel that are common inside and outside of ISRs and have that mutex assert on locking that interrupts are disabled.

This is certainly a fun hobby.

[cardboardaardvark]

a class of spin lock specifically for parts of the kernel that are common inside and outside of ISRs and have that mutex assert on locking that interrupts are disabled.

This worked out quite well. I couldn't use assert() because the logging system itself uses a mutex so what I did instead was go into a silent infinite loop. It causes the kernel to hang if it happens and leaves a nice backtrace to figure out what's going on with GDB.

To implement this I now have 2 spin lock types: a general purpose one and one for use when an ISR might be doing the locking. In my interrupt handler I set a flag using an atomic variable that is true the whole time the interrupt handling system is running. The general purpose mutex checks for this flag and hangs if it is true. The ISR mutex checks the interrupt enabled bit in the CPU and hangs if it is true. This trivial implementation won't survive SMP but it's good enough for now.

All the known instances showed up immediately when I started using this and it'll catch this accidentally happening in the future as changes are made with the benefit of not having to wait for the race condition to happen.

[eekee]

I've been thinking about the hardware Unix first ran on; 32KB RAM & 3 1GB disks in the PDP-11, and wondering quite how much of modern Unix's weirdness goes back to that strange ratio.

A while ago I started a project that needed a microcontroller to drive a machine I was building. I grabbed an Arduino, tossed out their IDE/runtime, and targeted it directly with gcc and avr-libc. I was quite surprised to see so much Unix coming out of avr-libc. Then it clicked: this $5 microcontroller I'm working with is a substantial fraction of the hardware that Unix was originally created for. Pretty amazing.

You're giving me ideas I don't need. It occurred to me that an Arduino Mega (or at least the Elegoo I bought for the kit it came with) coupled to an SD-card would have a similar ratio of RAM to swap space, though I guess the SD-card's write speed would prevent it swapping out all of userspace every 250ms. Cache the SD-card with a 512KB SRAM (perhaps with another Arduino) and it might work better. The PDP-8 and -11 had the upper half of RAM as kernel space if I understand right. The Mega's ROM is much larger. This leaves the Mega's RAM for userspace, where it has twice that of the PDP-8's and half of the PDP-11s -- a substantial fraction indeed! I even have a name for all this: Ardunix. But I really want to get on with a project I'd actually use. Wait, I could actually use my Mega... lol

I'm giving it a shot anyway. I'm still at the phase where it is a drastically over complicated clock. I'm moving ahead though. The kernel main loop now pulls jobs out of a queue and if no job is present does a hlt. I multiplexed the PIT and wrapped it with an interface that lets you create timers with arbitrary initial and reoccurring delays with a precision of 1ms. In the interest of minimizing the amount of time where interrupts are disabled the PIT ISR checks to see if it already has a pending job and if not puts one in the queue to check for expired timers then exits. In the interest of future concurrency if expired timers are found they too are submitted as jobs to run.

Sounds like you're doing it the right way around. Every now and then, someone will post about the issues of rushing to build the GUI when the internals don't work too well. It's better to get the deep stuff working well first. Does that still apply if you're "planning to throw the kernel away" after using it to design APIs? Well, I guess very bad kernel code might lead to poor choices when designing APIs. I don't know though, I'm just going on the mistakes I'd make.

I broke the status bar in the console into two parts: memory info and the clock and the memory info updates more frequently than the clock. From left to right the memory info goes: size of kernel program break, available physical pages, total physical pages:

This all reminds me of HForth, in which which both the clock and redrawing the graphical screen are handled by background processes under the multitasker. They can be regular processes because there's no user/kernel-space division in HForth.

At the moment I've got newlib and GCC 14.2's libstdc++ running in hosted mode in the kernel and I'm running GCC with C++23 as the language version. It definitely lets me move a lot faster than C. I'm traditionally a Perl programmer and lately more of a Python programmer and, unfortunately, I definitely can't move as fast with C++ as I can with those languages.

Good to know you can work faster in C++ than C. My last use of C++ was getting confused by a bad tutorial, then I got into Plan 9 (C and shell script only), then tangented off into APL and Forth before I got around to trying C++ again.

I am very very tempted to run an experiment and see how hard it would be to get Perl running in kernel space. Though while I'm still down at such a low level I don't yet see any benefit with Perl beyond std::shared_ptr, std::string, std::vector, std::list, etc. Maybe in the future.

Well, Linux has Lua in the kernel if I remember right. I don't know about Perl, but Python has PyPy which is written in Python (or rather RPython) and JIT-compiles to native code. I'm still waiting for someone to write an OS in Python/RPython, compiled with PyPy. Another idea I don't quite have time for.

My current sub-task is finding the ELF section header for the trampoline so I can get the physical and virtual addresses for it and map them in the page table directory. It's been more difficult than I expected to find the section name strings but I expect to solve that soon, get the trampoline mapped, and start finding the new things that will be more difficult than I expected.

Haha! There are always unexpected difficulties! I'm going to be tackling ELF from the code generation side so I can write a "plain English programming" kernel which can load ELF libraries.

Once I got the PIT multiplexed I decided to run it at a silly frequency (5khz, I might turn it up more) to expose reentrancy issues early. Wow. Never in my life have I seen malloc() go reentrant. That gave me a good laugh.

I've heard about reentrant malloc(). I don't want to think about it until I really have to!

I've already got spinlocks implemented and present in the interest of future concurrency. So going reentrant isn't catastrophic. It does a cause a deadlock but also leaves a nice easy to debug stack trace for GDB. I solve the reentrancy issue at the moment by disabling interrupts in sections of code that are common inside and outside of ISRs.

Nice! You make it sound easy.

This is certainly a fun hobby.

It is! Though I think your coding first and designing APIs second is more fun than my trying to design so much up front. I keep finding new design questions and problems. I just got a Nintendo DS as a nice little handheld package with buttons I can use as a chording keyboard, but now I remember it doesn't have a MMU. "Ardunix" might turn out to be slow, but it doesn't need a MMU to protect against accidental overwriting. Then again, maybe I'm making too much of accidental overwriting. Some kids seem to think you can't multitask without an MMU, but it used to be normal.