Curiosity about multi-architecture OSes

[Ethin]

So I've always been confused about this. x86 seems to be the "exception" to this (or maybe its the other way around). I've looked at other CPU architectures like RISC-V and ARM and wondered how people can actually write an OS for them at all. From what I can tell, there's no "standard" way of device detection like there is on x86, or any way of detecting what memory address goes to what device or anything like that, and all these other architectures seem to be MMIO based (which is a good thing). I know that PCI/PCIe is architecture-agnostic, but I'm still confused.
Take Linux, for example. From what I've read (primarily this answer on stack overflow, ARM, at least, has nothing like the BIOS for detecting memory or anything like that. Taking embedded systems as another, such as the recently released Raspberry Pico, I've noticed that all of them appear to have predefined memory maps. But if I'm writing an OS for the generic Aarch64 architecture, I'd say its safe to assume that if my OS is device agnostic I wouldn't know what device I'm running on, and therefore wouldn't know what memory map to use. So I'm just curious how that process actually works. I can't seem to find *any* information on doing this, which is weird. You can find guides for writing an OS for the RPi, but I doubt Linux's ARM port was written with the Pi in mind. So what exactly am I missing and how did people even figure this stuff out when there seems to be a lack of standards to rely on?

[Ethin]

Okay, so I literally found an answer: device trees. Is this the only way though or are there others?

[kzinti]

You got this backward: it's way easier on non-x86 PCs. Device Trees are great.

[pvc]

One way would be setting things up at compile time.

I like to break my kernel directory structure into 3 parts:

• architecure specific code (roughly translates to a CPU; like x86, x86-64, m68k, arm, arm64, etc.)
• machine specific code (like pc, raspi2, raspi4, amiga-ocs, amiga-aga etc.)
• and everything else that's common for all architectures and machines

Then your build system must be able mix and match these parts together.
For example, you could buld your system with:

Code: Select all

ARCH=arm64 MACH=raspi4 make all

Using this sheme, your interrupt handler stubs would be considered architecture specific (since they will be most likely written in assembler).
IRQ controller initialization code would definitely go into machine specific part (for example you expect PIC or APIC on a PC, but on Raspberry Pi 4 you expect GIC-400).
Finally, your interrupt handler body would go into common part (for example, the key you pressed is decoded and put into some event queue).

[eekee]

I've wondered about this too. It certainly was bad in the past, you had to design for each specific hardware revision. Now, however...

You got this backward: it's way easier on non-x86 PCs. Device Trees are great.

Are there device trees for every SoC or even board out there? If so, great!

Regarding @pvc's point of compiling for hardware, I'm sure it's a common way. Everyone knows about Linux's `make config`; I'm sure it was vital before device trees. Plan 9, which initially ran on a variety of non-x86 hardware, has these files in its kernel source tree which are plain-text lists of drivers. (There's some indentation for some reason.) You pass the name of the file you want to `mk` to build the kernel. It's not just drivers; these files also make the difference between building terminal & "cpu" kernels & optionally other detals. Example file. I forget where the syntax is documented. This works well enough that 9front's main kernel dev says firmware should not do anything beyond loading the kernel.

Some commercial systems relied on loading drivers, with the user having to specify the right ones. In practice, I suppose this is about the same annoyance factor as compile-time configuration. Note this includes MS-DOS because it wasn't possible to probe for many peripherals on ISA systems.

Finally, it used to be normal for systems to probe for RAM with a write-read test. (This may be a firmware thing rather than an OS thing.) Early 8-bit Atari ROM code did this. It relies on RAM being kept strictly at separate address ranges to any MMIO, or perhaps the MMIO being disabled until after the probe. The early Ataris (400 and 800 models) had RAM up to the 48KB boundary only, with MMIO above that but below the ROM. (6502 only supports MMIO, it doesn't have IO ports.)

On the flip side, there was OpenFirmware for... was it Alpha or SPARC? Apple infamously got it wrong, but everybody got standards wrong before there were open source implementations to compare with or simply port. Although... Apple might have got it less wrong if they'd hired Forth programmers instead of schmucks who'd never seen Forth before. I looked at some of the code; it was unbelievably bad! The bit I looked at was just one definition; the editor for the startup script. Good Forth practice says definitions shouldn't be longer than 3-5 lines. This thing filled 3/4 of a 133x75 character terminal, so it about 100 lines, several of which were very long. It didn't work at all.

[Ethin]

@kzinti, so I assume I'd set up a build system whereby the builder would specify what system they're building it for, and then that would include the device tree, and my kernel would read the FDT and figure out the physical system architecture from there?

[zaval]

Device Trees are horrid hack by linaro, full of linuxisms and completely messy. ARM is getting rid off of it and moving quickly to ACPI. And UEFI, of course. so while now you would have tough times trying to use these standard mechanisms for what you said, it will improve in future. RPi, for example does support a decent edk2 port of UEFI.

btw, bzt, you use RPi, right? did you try UEFI on it? what is the state on RPi4? what doesn't work? from the essentials, I mean. have you stumbled upon something, that you needed to be there, but it didn't work? I still don't have these boards, so I am trying to use uboot's implementation on the board I have - the state is like this: from 4 boards I tried, only one had GOP support, but this pretty outdated version had the bug in the reporting DeviceHandle device in the Loaded Image Protocol instance of the loader image - it wrongly reported different partition, not the one from where the loader has been loaded and; the newer version doesn't have GOP and (dispaly support at all). on top of that, ext4 FS driver of uboot still has a bug failing file reads, if the file position isn't 4-byte aligned. But still, usable kind of. below is the link to the first run of the UEFI OS loader on the Pine64+ board. which is the biggest (visually) achievement using uboot. pity, such complex things as display, event, timer functions etc worked, but silly file reads failed.

PS. yeah, right, "insert video ID string" and get nothing.
let's make it a link again!

[eekee]

Device Trees are horrid hack by linaro, full of linuxisms and completely messy. ARM is getting rid off of it and moving quickly to ACPI.

ACPI is a horrid hack where a bug in your interpreter can destroy some casually interested user's hardware! And "Just use this well-debugged interpreter" is only an appropriate answer for about half the people who want to develop operating systems. I wouldn't be surprised if ACPI dates from when Microsoft and Intel were married (and I'm not entirely sure the divorce was entirely real). I think I'll stick with ACPI-free microcontrollers whether or not they have device trees... although to be honest, I'm thinking of "just using" other peoples libraries anyway. A desire for encryption is pushing me that way.

I'm not really surprised to hear device trees are terribly messy. I was having one of my hopeful moments when I wrote my previous post, but now I remember 9front recently tripped up when RP4's device tree was arbitrarily changed.

[kzinti]

Both ACPI and Device Trees are horrible hacks. Both have bugs and don't work properly. This is just the way it is, hardware is messy.

But at least device tress don't have this non-sense byte code / interpreter requirement and is much easier to work with and patch as needed.

[kzinti]

@kzinti, so I assume I'd set up a build system whereby the builder would specify what system they're building it for, and then that would include the device tree, and my kernel would read the FDT and figure out the physical system architecture from there?

I do the same thing pvc described above... Basically I have a subfolder for architectures and one for machines. Then I invoke "make" with what I want (otherwise it auto-detects the current machine). I used to support RPI 1-3 at some point but dropped that. But I fully intend on bringing the glory of ARM / AARCH64 back.

For example, my folder structure looks like this:

Code: Select all

/kernel/arch/x86
/kernel/arch/armv7
/kernel/arch/aarch64
/kernel/machine/bios
/kernel/machine/efi
/kernel/machine/rpi2
/kernel/machine/rpi3

Building looks like this:

Code: Select all

make ARCH=ia32
make MACHINE=efi
make ARCH=x86_64 MACHINE=bios

The makefile knows which bits to use from the folders above.

DeviceTree is parsed at runtime just like ACPI is. But you probably don't want to include drivers for all the hardware your OS support on a given machine. i.e. for RPI3, only include what you need for RPI3.

[zaval]

ACPI is a horrid hack where a bug in your interpreter can destroy some casually interested user's hardware!

you say, with device trees it's impossible, because almost 100%, one will get lost to find it there or anything, slightly more complex, than UART, and thus - will not be able to make that mysterious hardware explode? because, I don't see how your direct writing into those registers, written, say, in forth or assembly is any more safe, than doing the same through interpreting AML, put there by the vendor, what implied it tested it. ACPI is a standard, and it lets you find about what this machine is; DT, as made in ARM, is a halfassed non-standard, cluelessly filled with more and more linux cruft, turning it into an absolute no go. and no, it's not even close to the full blown OF. it's not there, just millions of .dtbs with all kinds of variations, everyone puts, your OS would need to have a whole database of these poor attempts to play well (in front of "bad" ACPI).

[eekee]

@zaval: Ah, you're right.

[bzt]

So I've always been confused about this. x86 seems to be the "exception" to this (or maybe its the other way around). I've looked at other CPU architectures like RISC-V and ARM and wondered how people can actually write an OS for them at all. From what I can tell, there's no "standard" way of device detection like there is on x86, or any way of detecting what memory address goes to what device or anything like that, and all these other architectures seem to be MMIO based (which is a good thing). I know that PCI/PCIe is architecture-agnostic, but I'm still confused. Take Linux, for example. From what I've read (primarily this answer on stack overflow, ARM, at least, has nothing like the BIOS for detecting memory or anything like that.

This is not entirely true. They all have BIOS (or equivalent). Not the IBM PC BIOS, but their own Basic Input Output System.

When x86 was first marketed, it wasn't just the chip. An entire machine come with it, defining a standard which we call IBM PC compatible. This included its firmware too, the IBM PC BIOS.
ARM on the other hand only sells the license to create the chip, therefore there's no standard for the machine itself. This doesn't mean there's no firmware, it's just different on all boards.

There is a way to detect memory, for example on Raspberry Pi you use a MailBox call for that and a system register to get the MMIO base address.

Both ACPI and Device Trees are horrible hacks. Both have bugs and don't work properly. This is just the way it is, hardware is messy.

True! Plus ACPI is an industry standard, so manufacturers are expected to fix the problems and provide valid and correct tables to describe the actual hardware. Device Tree on the other hand are nothing more than a structure in the Linux kernel, manufacturers couldn't care about it less, they don't provide binaries (dtb) for it, the Linux kernel developers do (and they don't know the hardware like its manufacturer).

The whole Device Tree is a useless piece of sh*t, I simply don't understand why the Linux developers come up with that instead of simply generating ACPI tables in a file for the boards that lacks them (OpenFirmware is long gone, both SPARC and PowerPC machines are extinct dinosaurs, only found in a very very small corners of the market). With Device Tree you essentially have to duplicate big parts of the kernel for no good reason. It's not better than ACPI, just encodes the same information differently. If they had an ACPI parser in place, why reinvent the wheel?

But at least device tress don't have this non-sense byte code / interpreter requirement and is much easier to work with and patch as needed.

No, but it is extremely difficult to parse it (lot harder than ACPI, which has some pointers and some well-defined structs), and there's no guarantee that its format or its parsing library API won't change with the next Linux kernel version... (stable API nonsense, you know...)

Further reading ACPI and device tree. These moroons are planning to EMBED Device Trees as-is in the ACPI table instead of writing a simple DTB2ACPI converter... We are gonna die!

Cheers,
bzt

[nullplan]

So I've always been confused about this. x86 seems to be the "exception" to this (or maybe its the other way around). I've looked at other CPU architectures like RISC-V and ARM and wondered how people can actually write an OS for them at all. From what I can tell, there's no "standard" way of device detection like there is on x86, or any way of detecting what memory address goes to what device or anything like that,

You are confusing x86 with the PC platform. There is no reason an x86 computer couldn't use a different platform, it's just that with x86, the PC is so prevalent it wouldn't make much sense. But it is possible, anyway. The PlayStation 4 is an example of a non-PC x86 computer. And it still does most of the things PCs do.

For the other architectures you always have the same problem: Reading the CPU manual will not get you far outside of the CPU. The CPU manual will tell you how external interrupts work, but you don't even know what is on the other side of that interrupt line. Some architectures have more than one choice. No, you need to read the documentation for the microcontroller you are working with, and then the mainboard you are working with. And then you can "discover" the hardware, by hardcoding the locations of things in the kernel. Device trees merely parameterize this approach. You still need all the drivers, but now the DTB tells you where everything is. And the DTB is loaded by the bootloader.

and all these other architectures seem to be MMIO based (which is a good thing).

x86 is the only architecture with special IO instructions as far as I know. PowerPC used to have them (eciwx, ecowx), but they have been phased out. I seem to remember some architecture with special I/O instructions, but those were just optimized access to a specific area of normal address space. Actually opening up a second address space to do I/O on is x86 specific.

Device Tree on the other hand are nothing more than a structure in the Linux kernel, manufacturers couldn't care about it less, they don't provide binaries (dtb) for it, the Linux kernel developers do (and they don't know the hardware like its manufacturer).

Argument invalid: Some device trees are maintained by the manufacturers themselves. For example, the MPC5121 device tree in the PowerPC directory came straight from Freescale. Plus, for ACPI we have been plagued by buggy ACPI tables and not-quite correct AML for decades now. A device tree can be fixed inside of five minutes, an ACPI table generally requires a BIOS update.

The whole Device Tree is a useless piece of sh*t, I simply don't understand why the Linux developers come up with that instead of simply generating ACPI tables in a file for the boards that lacks them

You are linking a talk in that post directly comparing DT and ACPI. Have you seen slides 13 and 14? Note that the ACPI example is just the second device from the DT. So four lines of DT have turned into 18 lines in ACPI, and they include things like an _HID and a UUID, both of which you need to figure out somehow.

That is the reason. Yes, they both encode the same information, but the ACPI stuff is horribly overcomplicated.

(OpenFirmware is long gone, both SPARC and PowerPC machines are extinct dinosaurs, only found in a very very small corners of the market).

I win my bread on a PowerPC machine. Admittedly, no OpenFirmware though. But news of PowerPC's demise have been greatly exaggerated.

With Device Tree you essentially have to duplicate big parts of the kernel for no good reason.

Are you talking about the parser required? That is not a big chunk of code. Not for Linux, anyway. The biggest block of code in the Linux kernel is the drivers directory.

If they had an ACPI parser in place, why reinvent the wheel?

"Why reinvent the wheel" asks the microkernel developer on the osdev forums after he got done writing yet another bootloader. When GRUB and Minix are readily available.

Anyway, ACPI is deservedly not very well liked among kernel developers, because it is overcomplicated, and has been used to try and shut out operating systems other than Windows. When they started with DTs, ACPI was a PC-only phenomenon, and the kernel developers wanted to keep it that way. So they made their own.

We are gonna die!

Eventually. Not of this, though.

[bzt]

Device Tree on the other hand are nothing more than a structure in the Linux kernel, manufacturers couldn't care about it less, they don't provide binaries (dtb) for it, the Linux kernel developers do (and they don't know the hardware like its manufacturer).

Argument invalid: Some device trees are maintained by the manufacturers themselves. For example, the MPC5121 device tree in the PowerPC directory came straight from Freescale.

Erhm, invalid how? Just because one manufacturer contributes to the Linux kernel, DT is still a Linux thing (not multiOS like ACPI, but I think there's an incompatible FreeBSD port too), and the other manufacturers don't give a sh*t about providing DTBs (in contrast to ACPI tables which are in memory so manufacturers must provide it in the firmware regardless to the OS).

Plus, for ACPI we have been plagued by buggy ACPI tables and not-quite correct AML for decades now. A device tree can be fixed inside of five minutes, an ACPI table generally requires a BIOS update.

We are constantly plagued by overcomplicated and buggy sh*t (and newer generations sadly doesn't even realize that there's a simpler better way). And you can just as well fix an AML in five minutes. The problem is, if it's wrong, how do you know what are the correct parameters (for both ACPI and DT)? You simply don't know that, and playing trial-and-error could brick your machine (for both ACPI and DT).

You are linking a talk in that post directly comparing DT and ACPI.

Yes, of course I know, that's the reason why I've linked it... What is your point here?

Admittedly, no OpenFirmware though. But news of PowerPC's demise have been greatly exaggerated.

Sadly no. The market is dominated by x86, and now with the rise of mobiles ARM. All the others' share are just marginal (sadly, I liked PowerPC a lot).

Are you talking about the parser required? That is not a big chunk of code.

I was talking about the dependencies and difficulties of porting dtc to your OS and integrating libfdt into your kernel specifically. Agreed, using ACPICA is not better in this at all...

"Why reinvent the wheel" asks the microkernel developer on the osdev forums after he got done writing yet another bootloader. When GRUB and Minix are readily available.

DO NOT mix software (like a boot loader) and firmware (like tables in memory with some bytecode). And to answer your question,

1. GRUB couldn't boot a higher-half kernel into clean 64 bit mode, its Multiboot protocol is complicated and despite the name not multiplatform. Not all toolchains can provide an aligned section at the beginning of the binary (with 32 bit pointers to absolute addresses btw). Its tools have tons of dependencies with the most exotic ways to broke (like this for example). Its binary is such a huge beast that it is impossible to port it to many many platforms (Raspberry Pi included, someone has tried, but failed, they rather reimplemented a minimalistic Multiboot version, which still is huge and slow compared to BOOTBOOT. No wonder Raspberry doesn't use Multiboot ).

2. I don't like Minix, it's badly designed. I can do better even if I'd waken up from sleep. (Minix was an educational system, that's why; even though Minix3 tries to put usability as first priority, it has way too much baggage). One example: walking a path requires twice as many task switches as many mount points there are on that path. For every path operations! Issuing an open() could easily generate tens of task switches. For a single mount point in path there's at least 6 in the VFS layer: user -> FS, FS -> root filesystem driver, root file system driver -> FS, FS -> mounted file system's driver, mounted file system's driver -> FS, FS -> user (and here I haven't counted the task switches caused by accessing lower layers like block device drivers, just the VFS...)

Anyway, ACPI is deservedly not very well liked among kernel developers,

It's not the matter of like it or not (I for one, hate it). There's no other way to detect hardware on many architectures, so you MUST support it, whether you like it or not. You simply don't have a choice. Duplicating everything with DT was a choice, a bad one.

because it is overcomplicated, and has been used to try and shut out operating systems other than Windows.

Citation needed! How on earth could some tables and bytecode in memory shut out an OS??? That just makes no sense. Nobody said you MUST use the official ACPICA library either, just take a look at OpenBSD or this topic.

Cheers,
bzt