Developing hobby OS on Raspberry Pi 3

[bzt]

Hi,

I'd like to share some useful tutorials. There was a time when hobby OS development was limited to the x86. Thankfully this is no more the case.

https://github.com/bztsrc/raspi3-tutorial
Tutorials about low-level programming the Raspberry Pi 3 in AArch64 mode. All the relevant OS related topics are covered from setting up the toolchain through reading a file from the SD card to implementing a chainloader and parsing an initrd.

https://github.com/s-matyukevich/raspberry-pi-os
Although it's not finished yet, this is an awesome source for OS developers. This is not just a step-by-step tutorial, but also a very good documentation on how Linux does things on the Raspberry Pi 3, and gives you an insight on how to implement the same in a simpler (hobby OS level) way.

https://github.com/rsta2/circle
The first two were written mostly in C (with minimal Assembly). Now this library is for C++ developers. It not only provides the mandatory C++ runtime, but also implements many Raspberry Pi 3 functionalities. Even if you don't want to use someone else's library for your OS, this is a really really great source for information. It is very feature-full, includes IRQ handling, timers, USB keyboards and storages, virtual memory mapping, and even drivers for gamepads.

Cheers,
bzt

[StudlyCaps]

Hey, I've not used the rPi much, but I remember reading that the graphics adaptor was totally undocumented which really turned me off it. Is that still the case?

[Schol-R-LEA]

Hey, I've not used the rPi much, but I remember reading that the graphics adaptor was totally undocumented which really turned me off it. Is that still the case?

There is some documentation, and they've released more over time, but last I heard the released information isn't sufficient to write a complete driver. For the most part, the expectation is that you will be using a proprietary blob driver provided by Broadcom, which AFAICT is what is used by Raspbian and other Linux ports, as well as the RPi version of RISCOS. It is not clear to me how generic this blob driver is, but the fact that RiscOS can use it does indicate that at the very least, some non-Unix systems can adapt it to their needs.

Comments and corrections welcome.

Personally, while I am doing some projects with a RPi 3B (not 3B+, sadly), I am seriously thinking of getting something with a more open GPU, such as the NanoPi M4, NanoPC T4, the RockPro64, or the Libre Renegade Elite. While the openness is only one part of this (for example, all the ones I am considering use the RK3399, which is significantly more powerful than the SoCs used on current RPis, all of them can be gotten with 4GiB RAM, and all can use eMMC storage, but they are all rather more expensive as a result), it is certainly a factor to me.

[zaval]

Honestly, none of ARM SBCs have documented GPU. For example, the famous RK3399, the most interesting chip for now, because of the board prices and capabilities (it's hexa core, dual clustered CPU, 2 Cortex-A72 + 4 Cortex-A53, PCIe, USB3 etc. RPi is not even close to that), even in the leaked documentation, doesn't have anything on GPU. In the "open" documentation it does have a GPU part, it's funny, for example the chapter, promisingly called Register Description, consists of this:

The GPU base addressis 0xff9a_0000.

What else you might need, eh?

But HDMI and eDP transmitter controllers are kind of documented. this is really what's needed for being able to draw on the screen something.

Speaking of those blobs, it's probably an OpenGL ES implementation, am not sure, it would be of interest for anything except 3D acceleration. Anyway, it's rather the fact, that Linux often sucks with 2D graphics (desktop) too on these boards compared to Android on the same boards - for some reasons the former can't use libraries provided for the latter. despite it's all linux. doh. apparently SoC/board vendors care much more about Android. but libraries are not compatible between even different linuxes.
What I know, vendors provide some open source "driver", doing little (but that doesn't mean that the driver isn't a horrible mess one can't even dream about figuring things out over there), and closed source user space libraries, doing everything. For example, this link has a lot of material about it regarding ARM Mali GPUs. Open sourse tons of "doing almost nothing" are there as well.

even though everybody and his/her dog tell you that this is relevant only to 3D, you see by your eyes how linux GUI is sluggish even on things like Odroid-XU4, RockPro64, Tinkerboard etc and how Android is not! Maybe because there are yet some 2D engines, separate from GPU, that can accelerate 2D drawing, and again, they are better supported on android than on linux. anyway, it's not even close to the OS-independent using these blobs, as you can see on the example of andorid/linux incompatibilities. I don't know what that RiscOS does with linux drivers, probably they mimick linux inside to use it. 2D "engines" may be or not open and well documented. I stumbled across something in the Allwinner R40 SoC, but it's so messy and twisted, can't say about it much yet, apart from that chip has a very complex graphical "pipeline".
Broadcom's VideoCore thing is rather a peak of anti-design hostility to the hobbyists, both by their horrible design - GPU is the real main CPU in the system, whereas ARM CPU is a toy controller - and their unwillingness to realease documentation. For the system programming, broadcom SoCs are trash, so in my opinion, people wanting to do system programming thing are better off to switch to something else as soon as possible. especially given how numerous this range is and that that "huge" RPi community is a little of help for the osdev. they don't do OS developing anyway and won't help you much.

Personally, I hope, that in the nearest future, I can be more elaborative with this respect (display). At least for Allwinner SoCs. Basically, Freescale (NXP) iMX6 SoCs have "kind of" the same HDMI TX IPs, but they are documented, so, I hope, this way, - to figure out how to do it on the Allwinner SoC I have. RK3399 as well, as I've said. But it's a "leaked" pdf. where I am now, it's early for touching HDMI inappropriately.

By the way, about the price of RK3399 boards. They aren't that much more in price. Given the capabilities they have. On the example of RockPro64. They put the normal PCIe 4 lane slot and that means opening the possibility of connecting any PCIe expansion cards that are happy with 4 lanes. They sell PCIe-SATA bridge card, bringing SATA here. They have rather inexpensive PCIe-M.2 adapter, with which you can connect freaking NVMe SSDs! it's >1GB/s of the speed. Finally, this board got right UHS-I support for SD cards, a very important improvement. ~70MB/s compared to the sluggish pre UHS-I ~23MB/s of the maximum throughput, on the interface often serving the role of the main storage on SBCs. This UHS-I problem is really a very disappointing example of screwing up things. by SoC vendors. they have been licensing UHS-I capable controllers for their SoCs for ages, but then designed incapable PMICs and their reference design boards (that in turn encouraged everybody else to do the same). Not enough voltage regulators for providing a dedicated one for the SD controller, so that it could switch from the 3.3V to 1.8V signaling, needed for UHS-I. Or just f&cking it up without the reason. Almost all current SBCs have UHS-I capable controllers and almost none of them can provide voltage switch due to f&cking up by design. A bitter example is RK3328, with the specifically designed for it RK805 PMIC. *sigh Fortunately, with RK3399, things have changed. Most RK3399 boards (but not all!), provide voltage switch capabilties. Given SD cards play a very important role in the SBC (main storage often), it's not a trivial problem.

It's a pleasant thing that osdev people start to have a look at ARM boards, the only irritating thing is that they unwisely narrow their perspective to only RPi, which is way not the best out there by any means. People, come on, look around, there are so much better things to play with! By the way, Sinovoip, the vendor of Banana Pis (I have Banana Pi M2 Ultra), is about to release a 24 core server ARM board! Of course, it won't be 100$ of price, but also, I am sure, it would a competitive price still. Here is the link with the video included, where they compile linux on that beast.

[eryjus]

Hi,

I can say that I am trying to get my 32-bit kernel ported to RPi2 and I am not having any success with getting a pixel written to the screen. I have been researching and pounding my head for about a week on this and have not gotten any closer.

There are several issues I am having with this:
* There is no documentation. Not only is there no documentation, but Broadcomm does not feel the need to release more information or update the current inaccurate information. Broadcomm intended the SoC to be used for education purposes. That much is clear. But they never intended it to be used for bare metal education. This is what you will determine when you read between the lines.
* The firmware is not stable or static. In fact, when the firmware is updated, they routinely break APIs. My personal opinion on this is that they really want to pull the RPi out of the bare metal hobbyists. Instead, they make sure that Raspbian works (and only Raspbian) works -- and hide the details.
* qemu does not faithfully emulate the BCM2835/6 SoC. There are big holes in what is implemented and what works on qemu does not work on real hardware -- and what works on real hardware does not work on qemu.

Hey, I've not used the rPi much, but I remember reading that the graphics adaptor was totally undocumented which really turned me off it. Is that still the case?

I'm sorry to say, things have not improved. I want to port my kernel to another platform and I was hoping this could work....

[Schol-R-LEA]

I don't know if it helps any, but there does seem to be a certain amount of support from Broadcom for the RiscOS port, probably because they were getting a lot of flak for focusing solely on Raspbian and related Linux distros. They may also be looking at it as a sop towards the UK's education market (which was, after all, the original target market for the Pi), where the BBC Micro and the Acorn Archimedes are still well-remembered (RiscOS was the OS for the Archimedes, the system the ARM ISA was designed for initially). Now that RiscOS is open source, it was an obvious choice for a non-Unix OS to support on an ARM-based system. The fact that a number of ex-Acorn devs and designers, including Sophie Wilson, work for Broadcom now may also play a part in it.

Whether this makes a difference to us in the OS-dev community isn't clear to me; probably not. But at least we can compare the RiscOS driver source to the Raspbian drivers to try and glean a few more details, I guess.

Trivia: If memory serves, RiscOS was originally named 'Arthur', though I am not sure if that applied to the OS as a whole or just the desktop environment. The name was changed in the early 1990s, around the same time the 'A' in "ARM" was switched from "Acorn" to "Advanced". The Archimedes was Acorn's intended successor to the BBC Micro, which was itself based on the earlier Acorn Electron and build by Acorn for the school systems as a collaboration with the BBC (as you could probably have guessed), initially going along with an educational television program. Steve Furber and Sophie Wilson designed the ISA initially with the hope of leapfrogging the competition with a super-fast RISC design, which was the new hotness in 1985.

[StudlyCaps]

The GPU base addressis 0xff9a_0000.

What else you might need, eh?

Yep, that's what I thought
I like the idea of developing an OS for a ARM SOC, it would be neat to have a target platform anyone could pick up for less than 100 bucks. The overall attitude of the companies involved however is discouraging. If I'm developing system code for a platform, I expect that the basic API has available documentation, but I guess that's the way technology is trending. "We sell you everything you need, please don't change anything."

[Solar]

I do understand the reasoning of the manufacturer, though. Complete documentation means disclosure of rather elementary IP, enabling others to come up with something better / cheaper by taking shortcuts in the R&D. It also means committing to things for backward compatibility purposes.

I don't say one side or the other is more "right" than the other, just that both sides do have a point. Unfortunate as it may be for those who'd like to use their own drivers.

[Schol-R-LEA]

I guess that's the way technology is trending. "We sell you everything you need, please don't change anything."

Actually, it has been the dominant model from almost the outset; openness, especially in hardware, has always been the exception rather than the rule. It only seems otherwise because of certain significant exceptions early in the personal computer market - notably the early Commodore, Tandy, and ironically enough, Apple computers. Even more ironically, the IBM PC was also relatively open, though to a lesser extent than the others I mentioned (and the story behind that is an interesting if long-ish one).

But they were exceptions. And the reasons they were exceptions mostly had to do with how the home market was developing at the time - in 1977, most home computer owners were hardware tinkerers, and it was seen as a hobby rather than a practical market. That changed rapidly by the end of that year, but the reason that (for example) the Apple II's documentation included full schematics and the assembly source code for the ROMs was because a) the average user was someone who would actually need those details, and b) no one at Apple thought the Apple II was important enough to bother hiding anything.

Prior to that, most manufacturers explicitly forbade the lessors of their hardware - no one bought computer hardware, you see, and most manufacturers wouldn't sell it unless forced, as IBM was in 1967 (?) under threat of anti-trust action by the Dept of Justice - from even opening the cases, or in at least one notorious instance, moving the TTY console to a different location. All maintenance required a Field Circus, sorry, Field Service technician to come over, and you would lose you contract if they thought you'd tampered with the hardware in any way. There was more wiggle room in mini-computers, as most of those were used with custom hardware interfaces to scientific measurement equipment, but even then you had to have permission from the manufacturer to do so, and they only gave you the necessary information (and only that much) once you'd hammered out a legal agreement for it.

Even things like printers were mostly leased, and those which were sold came with no information about how they worked; see the Computerphile video on Brian Kernighan and Ken Thompson's "202 jailbreak" and the influence that had on later typesetting systems for a taste of this.

Software was always less locked down than hardware, but the reasons for this were in the historical development of the field, combined with the peculiar economics of software development and distribution - whereas hardware requires a physical manufacturing base to produce, writing software is mostly a matter of organization, expertise, and time.

While software is very expensive to create, and even more expensive to maintain, it is possible for that expense to simply be eaten by a motivated individual trading organization off for expertise and time, or for a large number of motivated independent expert developers to leverage loose but flexible self-organization and a lot of time to amortize the cost among them to an amount which each volunteer would be willing to sacrifice, in order to get the finished product. This is part of the reason why most FOSS advocacy groups place their emphasis on libre rather than gratis - the cost isn't zero, but it is spread out among the developers, who then get the working program in exchange for the time committed and the understanding that others will get to use it, too. But it is also why few FOSS projects where the developers weren't personally invested in the final product have succeeded.

Despite all of this, FOSS is, by its nature, a side product of the commercial computer industry - there is no existing economic model (yet, anyway) by which FOSS can stand on its own, and indeed a large part of why it succeeds as well as it does is because the economic models for proprietary software are themselves deeply and fundamentally flawed. For all the money that commercial software makes some companies, that income is a fraction of what it might be; the profits inevitably falter, because each of the many ways in which it is sold or leased are all busted, bloody messes.

[bzt]

Hi,

Well, "undocumented" is a bit harsh statement. The whole thing is a big blackbox, true, but the interface is easy to use, and we also have lots of tutorials and example code. Best to start at https://github.com/raspberrypi/userland if you want fancy stuff like OpenGL or hardware video/image decoder. It is much more likely you weren't able to locate the documentation (or the errata, which is needed very much).

For the most part, the expectation is that you will be using a proprietary blob driver provided by Broadcom, which AFAICT is what is used by Raspbian and other Linux ports, as well as the RPi version of RISCOS

That's true. Only the interface is documented, the one used by the OSes you mentioned. Wether you need to dig deeper for a driver is a different question. Imho not neceassairly.

If I'm developing system code for a platform, I expect that the basic API has available documentation

And you have it. Just look around on the raspberry pi's github. Github wiki has lots of information too.

I can say that I am trying to get my 32-bit kernel ported to RPi2 and I am not having any success with getting a pixel written to the screen.

For that you don't need a GPU driver at all. Linear framebuffer is easy to set up, just take a look here. All you need to do is write some bytes at an MMIO address (MailBox property interface call) and you're good to go, you don't have to dive into the internals of the VC chip. You can also clip and scroll the framebuffer with that. Think about the MB interface as the VBE for RPi.

* There is no documentation. Not only is there no documentation, but Broadcomm does not feel the need to release more information or update the current inaccurate information. Broadcomm intended the SoC to be used for education purposes. That much is clear. But they never intended it to be used for bare metal education. This is what you will determine when you read between the lines.

That's not true. They (both BCom and ARM) suck at documentation I give you that, but there are plenty, and people have provided lots of erratas and tutorials to help you. As noted before, the SoC itself is not really documented, only it's interface (which should be enough for programming).

* The firmware is not stable or static. In fact, when the firmware is updated, they routinely break APIs. My personal opinion on this is that they really want to pull the RPi out of the bare metal hobbyists. Instead, they make sure that Raspbian works (and only Raspbian) works -- and hide the details.

I don't know where you get this. I'm programming RPi on bare metal level for quite some time now, and I have never experienced the firmware being unstable or changing. The worst I've seen was that they had to move the base address of the MMIO region on newer models (which can be detected by reading some simple ARM system registers btw). But the devices as well as the MailBox interface are stable and haven't changed (only expanded with new functionalities). As a matter of fact the newest RPi3 can run kernels written for RPi2.

* qemu does not faithfully emulate the BCM2835/6 SoC. There are big holes in what is implemented and what works on qemu does not work on real hardware -- and what works on real hardware does not work on qemu.

That's true. Qemu can boot your kernel emulating both RPi2 and RPi3, you'll have sufficient ARM support, UARTs and framebuffer works fine, but that's all. Many real RPi devices are not yet implemented. I've also noticed that emulation is less strict than real hardware (for example you can set up a vector code table anywhere on qemu, while real hardware has strict aligment requirement for the vbar register).

I'm sorry to say, things have not improved.

Yes, they have. Things now are much better as they were about 2 or 3 years ago. Let's just say there's still a lot of room for further improvements. These days you can find all the things needed by a hobby OS, but the information is not as well structured or detailed as for the IBM PC.

Cheers,
bzt

[linguofreak]

My understanding is that:

1) The GPU drivers were originally entirely a binary blob, but this changed because the RPi became popular with a customer base that didn't like binary blobs, and there are now open source graphics drivers for the Pi for Linux.

2) The GPU handles other chipset functions than graphics, for instance, the GPU firmware holds the CPU in reset and fetches the boot code from the SD card, then releases the CPU from reset once there is code in memory for it to run. My understanding is that some of these ancillary functions do not have documented interfaces, and Linux needs a proprietary blob driver to make use of them.

[bzt]

My understanding is that:

1) The GPU drivers were originally entirely a binary blob, but this changed because the RPi became popular with a customer base that didn't like binary blobs, and there are now open source graphics drivers for the Pi for Linux.

The real question is, does this really matter? For example, BIOS is also a binary blob, but do you care? You know the interface (interrupts with arguments and return value in registers and CF) and you use it. As long as the interface is consistent, do you really need to know how it is implemented under the hood? On RPi, you have a well documented interface to interact with the firmware (Mailbox) which hides all GPU specific code from you, similarly as how VESA hides the video card details from you.

2) The GPU handles other chipset functions than graphics, for instance, the GPU firmware holds the CPU in reset and fetches the boot code from the SD card, then releases the CPU from reset once there is code in memory for it to run.

Yes, this is how RPi boots. But again, do you really care? Have you ever disassembled a BIOS to figure out how the boot sector is loaded? Or is it enough to know that it loads the first sector at 0:7c00 and executes that with drive code in DL? The same stands for RPi firmware. You don't need to know or care how your boot code is loaded into memory.

My understanding is that some of these ancillary functions do not have documented interfaces, and Linux needs a proprietary blob driver to make use of them.

That's true, that Linux requires a binary blob (device tree, equivalent of FADT+DSDT on ACPI), but that's not proprietary at all. All sources are publicly available, and you don't have to use any Linux specific stuff in your kernel if you don't want to.

There are some GPU features (like hardware accelerated JPEG and MPEG decoder) which are not well documented and require a binary driver under Linux to utilize, but nothing that you have to worry about on a hobby OS. All the features you'll need (serial, keyboard, LFB, EDID, SD-access, USB, network etc.) are available without a proprietary driver, and have a fair documentation (not perfect, but the doc is there).

Yeah, it would be better if the firmware itself would be Open Source, that would align with RPi philosophy better. But you simply can't buy such a SoC, so there has to be a compromise. I think the situation is not worse than with BIOS or UEFI. I mean I have never heard anybody complaining about why is UEFI hiding how it loads FS0:/EFI/BOOT/BOOTX64.EFI for example.

Cheers,
bzt

[zaval]

Yeah, it would be better if the firmware itself would be Open Source, that would align with RPi philosophy better. But you simply can't buy such a SoC, so there has to be a compromise.

Not everyone cares about open source, but everyone gets angried when he/she can't find documentation on how to program stuff. no GPUs out there are documented at all. this AlienCore is one of the worst cases. It's not "some features" not documented, it's its MAIN features not documented.

How is that that "you can't buy such a SoC"? You can! Just don't buy this freaking sh1t from broadcom, everything else is a normal implementation and is much better documented. And there are really very well documented things like NXP iMXn SoCs. Everything is better than broadcom.

Those who are unwilling to share documentation for bringing software support for their hardware, may take a hike. No compromises needed. Just somehow, people out there think, that there is nothing except RPis. It's not truth.

I think the situation is not worse than with BIOS or UEFI. I mean I have never heard anybody complaining about why is UEFI hiding how it loads FS0:/EFI/BOOT/BOOTX64.EFI for example.

It is worse. It is one of the worst. Because not only nothing is available on how to use that GPU as an "accelerator", 3D engine, which SHOULD BE ITS SOLE role as it goes by its name, but they made it a monster, Alien, sitting inside of the platform controlling everything and one needs to do that communication, very stupid and unnecessary, they exposed this f&cking "interface" for setting up everything. Come on! On normal platforms, for configuring stuff CPU is used. It is it Central, not that closed, hidden, ugly bullshit. It bugs just from the conceptual point of view, you constantly ask WTF is this, WTF is this for etc? Not to mention it brings loads of portability issues, - really, noone else (fortunately) is using this ugly approach, so one needs to introduce bloody hacks inside of the OS to get this mediocrity run. Or design it the "RPi" way, which is highly hard to port due to this VC "mailbox" idiocy. RPis aren't ARM computers. They are VC computers with a toy ARM controllers. Thus, they are bad choice for those people from the osdev willing to get close with ARM computers. Because the people will need to learn a lot if irrelevant things that won't be needed for programming real ARM computers. Worse, if they finally make something running on RPis and they will want to broaden their target park, they will be very unpleasantly surprised they have created something that is very not easy to port to other ARM machines. at least to say.
UEFI doesn't hide what you say at all. First, everything related to the specified behaviour, is described in a freely available specification. Second, the implementation is open as well. It's not even close to compare.

[bzt]

Not everyone cares about open source, but everyone gets angried when he/she can't find documentation on how to program stuff. no GPUs out there are documented at all. this AlienCore is one of the worst cases. It's not "some features" not documented, it's its MAIN features not documented.

You're right about that. My point is, ARM/VideoCore/RPi documentation has reached a point where we have a sufficient documentation. Not as much and well organized as for the IBM PC, no argue on that, but at least there's enough for implementing a hobby OS. (Which is the whole point of this topic, to let the community know about this)

How is that that "you can't buy such a SoC"? You can!

But they didn't. Maybe you should ask the engineers at the RPi Fundation for an answer. I guess this was the cheapest, but I haven't seen their contract with BCom. Don't forget that unlike other computers, RPi has a very low, fixed price!

Those who are unwilling to share documentation for bringing software support for their hardware, may take a hike. No compromises needed. Just somehow, people out there think, that there is nothing except RPis. It's not truth.

Agreed. But the thing is, RPi is the best known by far and easiest to buy (and probably the cheapest not counting the Chinese replicas, which tend to get lost during shipping).

It is worse. It is one of the worst.

I don't understand your point of view. It is a non-common design choice, I give you that, non the less it's not as obscure as you say. UEFI for example is much worse, because although there's a documentation, you cannot be sure that the manufacturer had it implemented correctly. There's a good chance that a certain protocol works on one machine, but freezes another (talking from experience). Hell, you wouldn't even know if GOP happen to be implemented on the GPU directly! Let me remind you, neither UEFI implementation nor VC implementation are Open Source, only their interface are documented. With RPi at least you can be sure there are no non-compliant implementations.

Because the people will need to learn a lot if irrelevant things that won't be needed for programming real ARM computers.

That's just not true. In fact, you can only compile for the ARM processor (in lack of a GPU bytecode license), and every aspect of interfacing with the platform is technically a driver. And a mailbox call is not more complicated than any BIOS or UEFI call. Seriously, who cares how does the power off function is executed as long as you can call it from the ARM CPU using a well-defined interface???

To sum it up, if you don't like the RPi, don't code for it, let us others have fun with it.

Cheers,
bzt

[StudlyCaps]

To sum it up, if you don't like the RPi, don't code for it, let us others have fun with it.

While I don't have any present interest in coding for the RPi, other people voicing criticism for decisions made by the manufacturers/designers doesn't mean you can't use it. "If you don't like it don't use it" is one thing, but this seems more like "if you don't like it don't discuss it".