Which architecture to start with?

[obfusc8or]

Hello. I'm new here.

I'm considering writing my own experimental single address space hobby OS, utilizing heavily a hobby programming language. I don't know which architecture I should write it for first.

I think x86 looks like a very complicated and unpleasant architecture, and I like to believe that it is nearing the end of its lifespan in a couple of decades. ARM might be a bit nicer and have somewhat less historical backwards-compatibility baggage.

I'm a fanatic supporter of "permissive" open-source licensing (let's not talk about it now ), and thus I'm very interested in the BSD-licensed open-source RISC-V architecture. I think there are no physical CPUs available to consumers yet, though, but there are some emulators and simulators.

I've also considered creating my own simple instruction set architecture and a simulator, possibly even building a CPU from logic ICs.

So, what do you think: which would be the best way to go for now?

[onlyonemac]

This is an experimental OS, so you don't need to worry (yet) about support for processors into the far future. So use whatever processor is going to easiest for you to perform your experiment on. I don't know much about ARM or RISCV, but on the x86 you can set up a flat memory model (there are plenty of tutorials for it, I think there's even one on the OSDev wiki).

If you design your own CPU, you can obviously design it around the design/goals of your OS, but be warned that, while rewarding, designing a CPU is a challenging task that is likely to become somewhat a project in itself - you also need to think about the hardware in your system, such as a serial interface, video/sound card, disk drives, and this is likely to be very difficult if you want to support modern hardware (for example, interfacing with a floppy disk drive is considerably easier to interfacing with a USB flash drive). I have designed a CPU before and contemplated a few other designs, but I didn't build it because I got stuck with all the supporting hardware (I wanted a standalone system, so booting over a serial port wasn't satisfactory for me).

[Brendan]

Hi,

I'm considering writing my own experimental single address space hobby OS, utilizing heavily a hobby programming language. I don't know which architecture I should write it for first.

There's no point writing code you can't test; so the first question that needs to be asked is: what architectures do you have test machines for?

Note that for "single address space" you're going to want something that supports massive virtual address spaces. For a normal OS you can have 1000 processes with 3 GiB of space each, so a 32-bit CPU is mostly fine (or at least it was 10 years ago - more recent software, especially things like in memory databases, tend to need more than 3 GiB of space all by itself). For "single address space" you'd need a 64-bit CPU just to do what a 32-bit CPU could have.

If you limit yourself to "common 64-bit CPUs" (and ignore things that few people have ever seen, like Itanium, Sparc, etc) that mostly only gives you 2 choices to choose from - 64-bit 80x86 and ARMv8.

I think x86 looks like a very complicated and unpleasant architecture, and I like to believe that it is nearing the end of its lifespan in a couple of decades. ARM might be a bit nicer and have somewhat less historical backwards-compatibility baggage.

It takes a little work (e.g. enabling A20, switching to long mode, etc) to get past the backwards-compatibility baggage of 80x86; but once you've done this you no longer have to care about it.

For ARM, there's no backward compatibility, which means you need to modify your code to support different ARM CPUs. Worse, different SoCs have different devices and there's no way to auto-detect. In the end the "cleaner" ARM ends up being far more work (unless you only support one specific SoC, but in that case you can expect the OS to be obsolete before it's usable).


Cheers,

Brendan

[Brendan]

Hi,

If you design your own CPU, you can obviously design it around the design/goals of your OS, but be warned that, while rewarding, designing a CPU is a challenging task that is likely to become somewhat a project in itself - you also need to think about the hardware in your system, such as a serial interface, video/sound card, disk drives, and this is likely to be very difficult if you want to support modern hardware (for example, interfacing with a floppy disk drive is considerably easier to interfacing with a USB flash drive). I have designed a CPU before and contemplated a few other designs, but I didn't build it because I got stuck with all the supporting hardware (I wanted a standalone system, so booting over a serial port wasn't satisfactory for me).

The nice thing about software is that once you've got something usable you can create as many copies as you like for free.

Hardware isn't like this - "creating copies" means manufacturing costs. To manufacture a CPU that is slightly comparable to modern ARM and 80x86 means spending several millions dollars, and then trying to recover the costs by selling CPUs. It's something that large companies like AMD have enough trouble with; and isn't something a single person with a pocket full of transistors can realistically consider viable.


Cheers,

Brendan

[~]

16-bit x86 PC at the BIOS and DOS level (MS-DOS and FreeDOS).

Chances are that if you want to develop an OS, you have always had access to interesting demo projects and an ample documentation for the platform which even a normal user can recognize.

Places like Programmer's Heaven, CodeProject.com, StackOverflow, some Geocities websites (like the SiliconValley area), and other old websites from the Windows 98 era (Iczelion, P2P networks like eMule, and any other old Internet resource or book) will teach you most of the things that were made with true enthusiasm and without hiding details, so that information is very standard.

Start to a 386 level, with standard hardware (ISA, PCI, IDE/ATA, PS/2, speaker, timers, PICs, VGA, SATA...).

Then learn about executable file formats, learn C (it's easy if you start by learning algorithms, memory management and the C library with practical snippets of all kinds).

You can practice using Unreal Mode and Protected Mode at any point.

Then you can switch back and forth from JavaScript/HTML5 to Operating System level to make simple tests.

You should also learn the WinAPI, the Windows syscalls and libraries in Assembly and C as a minimum (then in C++), device driver and hardware programming under those systems, install standard emulators like Bochs (2.3.7 is good to run Windows 98), DOSBox and VirtualBox.


Also learn to use NASM, YASM, FASM, and compile under Linux, MinGW, Windows 98, Windows XP and Windows 7.

To learn from all sides you would need to install all versions of Visual Studio and the free GNU compilers. Then learn how to install the different libraries, SDKs, tools and other resources to compile anything you can imagine (Firefox, Blender, ReactOS, game emulators like ZSNES, z26, Stella; eMule, Audacity, free snippets in all languages...).

You would also need to virtualize or emulate in your same machine:

Your own Operating System,
Windows 98/9x,
MS-DOS,
FreeDOS,
Windows XP,
Windows 7,
All versions of Visual Studio including Community Edition (you'll need them since different projects use different versions and fail to compile if you use another, unintented tool),
the different Linux distributions,
Linux from Scratch

You will do good to get a big enough disk for you development machine and for being able to install those things.

A 2.5" or 3.5" disk of around 2 Terabytes would be a good minimum to first install all that you need and then use the tools and Operating Systems in a practical and immediate way.

Also, several 5-Terabyte or 8-Terabyte disks for storing and archive your data would be ideal (and if possible, upload what is unrecoverable and personal to Archive.org in case the disks fail, or burn it to DVDs, although that can be extremely slow and those plastic discs can get degraded, scratched and unreadable over time).

[onlyonemac]

16-bit x86 PC at the BIOS and DOS level (MS-DOS and FreeDOS).

Except you'll only have 1 MB of memory, split into 64 KB chunks. It's "single address space" alright, but usually "single address space" implies "flat address space".

Places like Programmer's Heaven, CodeProject.com, StackOverflow, some Geocities websites (like the SiliconValley area), and other old websites from the Windows 98 era (Iczelion, P2P networks like eMule, and any other old Internet resource or book) will teach you most of the things that were made with true enthusiasm and without hiding details, so that information is very standard.

Start to a 386 level, with standard hardware (ISA, PCI, IDE/ATA, PS/2, speaker, timers, PICs, VGA, SATA...).

Then learn about executable file formats, learn C (it's easy if you start by learning algorithms, memory management and the C library with practical snippets of all kinds).

You can practice using Unreal Mode and Protected Mode at any point.

Then you can switch back and forth from JavaScript/HTML5 to Operating System level to make simple tests.

You should also learn the WinAPI, the Windows syscalls and libraries in Assembly and C as a minimum (then in C++), device driver and hardware programming under those systems, install standard emulators like Bochs (2.3.7 is good to run Windows 98), DOSBox and VirtualBox.


Also learn to use NASM, YASM, FASM, and compile under Linux, MinGW, Windows 98, Windows XP and Windows 7.

To learn from all sides you would need to install all versions of Visual Studio and the free GNU compilers. Then learn how to install the different libraries, SDKs, tools and other resources to compile anything you can imagine (Firefox, Blender, ReactOS, game emulators like ZSNES, z26, Stella; eMule, Audacity, free snippets in all languages...).

You would also need to virtualize or emulate in your same machine:

Your own Operating System,
Windows 98/9x,
MS-DOS,
FreeDOS,
Windows XP,
Windows 7,
All versions of Visual Studio including Community Edition (you'll need them since different projects use different versions and fail to compile if you use another, unintented tool),
the different Linux distributions,
Linux from Scratch

You will do good to get a big enough disk for you development machine and for being able to install those things.

A 2.5" or 3.5" disk of around 2 Terabytes would be a good minimum to first install all that you need and then use the tools and Operating Systems in a practical and immediate way.

Also, several 5-Terabyte or 8-Terabyte disks for storing and archive your data would be ideal (and if possible, upload what is unrecoverable and personal to Archive.org in case the disks fail, or burn it to DVDs, although that can be extremely slow and those plastic discs can get degraded, scratched and unreadable over time).

I don't think you need to read every programming tutorial, learn every API, compile every application, study every operating system, and install every toolchain across several 8 TB hard disks just to write a hobby operating system with a single address space.

[~]

The fact is that you won't be doing just that, and if you try to write an OS, no matter how simple, you will surely won't stay just with your simple kernel but also trying to learn just about everything (computer math, algorithms, protocols, hardware programming, low level details for everything you program, etc...).

I'm personally trying to understand absolutely everything to make it more portable and make tutorials about it to make things simpler, as they were in the 90's and 2000's when the interesting things started and become less interesting after the more closed nature of the more recent programs starting with Windows XP, but it can be reverted if the environment is populated again with the highest quality tutorials that show you everything you need to know about how it looks like setting up, developing and compiling everything.

[Kazinsal]

Recommending real mode x86 as a serious starting point is as brilliant as recommending people seriously consider using FAT32 on floppies. It's a terrible mess and while you'll learn a bunch of concepts, you'll also learn a bunch of bad habits from the hacky mess that you can't apply to working in a more modern environment. Telling people they need to know the ins and outs of every OS and API is equally absurd.

I don't know what's worse. That advice or people coming here once or twice a week with no idea how pointers and arrays work.


OP: I don't think RISC-V is ever going to make it to mass production, and doing an entire CPU design from scratch for something like this is a gargantuan amount of effort. Stick with something common and real.

Also something to keep in mind: there is a larger number of people around here who know x86 well than there are for ARM.

[~]

I have been thinking about implementing a programming layer that makes everything look like the BIOS and a set of standarized classic raw hardware with 32 and 64-bit capabilities, so you can even create windows, drive hardware and make your regular programs under Windows and Linux directly with Assembly snippets as simple as a COM program.

I call it LowEST, and it could help bring scripting to the Assembly, C and native code programming (think about truly scripting Assembly language natively... if you describe a fully-standarized PC platform for it, programming in this way will be really easy, as if all PC computers were the same since the 8086 in all of its technical aspects and programming details).

There's far too much at the Real Mode level as to say that it's obsolete (in fact, the PC platform fundamentals), so it's ideal to start from a known and proven good point and style of programming and implementing hardware and software, be it physical or virtual.

You will find most key concepts there and simplified for a newbie (like graphics tricks, entering Protected Mode, programming and thinking in terms of standarized hardware and software).

At the Protected Mode side you will only find UEFI and finished OSes.

Real mode in the other hand takes you to write data directly that is already 4-Gigabytes in size without any protection with Unreal Mode, and you'l learn about the BIOS, booting and entering Protected Mode.

Wouldn't it be defeating the purpose to learn about Operating System Development to suggest that it's useless to know about 16-bit Real Mode?

[alexfru]

I'm considering writing my own experimental single address space hobby OS, utilizing heavily a hobby programming language. I don't know which architecture I should write it for first.

Anything that Linux can run on should be good enough technically. Think a 32-bit CPU with MMU or better.

I think x86 looks like a very complicated and unpleasant architecture,

You don't have to use all of its complicated features. My Smaller C compiler doesn't use even a half of x86 memory operand types. There's plenty of room for improvement, but you probably won't care about generating optimal machine code for a very long time. You'll have to get the design and the first implementation done before you think of implementing various optimizations.

ARM might be a bit nicer and have somewhat less historical backwards-compatibility baggage.

There's quite a bit of variation in ARM (in terms of features and instructions available) and it has extremely ugly instruction encoding, not complex, simply ugly, with bits of immediate and other operands smeared across the instruction bytes. So, it may be not as nice as you may think. Though, if you generate assembly code, most of this ugliness can be delegated to the assembler to deal with.

So, what do you think: which would be the best way to go for now?

MIPS is good. Not because I work for the company, but because it's a well-known arch, it's rather regular and if you want a board with it to play, there are many options. There are inexpensive boards capable of running RetroBSD, OpenWRT, Linux, Android.

[mikegonta]

Recommending real mode x86 as a serious starting point is as brilliant as recommending people seriously consider using FAT32 on floppies

Please lighten up and stop recommending it. Pretty soon everyone will be taking that technical toy novelty seriously.

[iansjack]

16-bit x86 PC at the BIOS and DOS level (MS-DOS and FreeDOS).

Worst. Answer. Ever.

You only have to read these forums to see the number of problems Real Mode causes. It's a horrible and inconsistent architecture that will only teach you bad habits. If you really want to use an obsolete processor then go for a decent one (6809, 68000, PowerPC). Better still, target a modern processor that has a future and supports a reasonable address space via paging. In practice this means x86_64 or ARM.

In reality the processor is probably the least important factor as most of an OS is architecture independent. More difficulties arise with ancillary chips. For this reason, x86_64 is the most logical choice as the whole infrastructure is more consistent, and better documented, than any other. Also, the hardware is cheap and widespread.

This answer presupposes that you are interested in the operating system more than the processor as such. If your interest was the latter I would assume that you wouldn't ask the question.

[rdos]

No reason to mess with real mode or any of the ugliness it's created (DOS extenders). Real mode only have an ugly memory partitioning scheme (segments), and you cannot make any good use of it. The 32-bit x86 environment is very nice and neat, but unfortunately it has a few problems that Intel or AMD never solved, and today the processor is optimized for flat mode, which is really ugly. I'd go with X86_64, and use the unintended feature that code can only directly access a 4G address-space (rip-relative addressing), and then use the higher 16 bits of the address space for segmentation. That way you can write relatively independent drivers that won't mess with each others unless you want them to. Of course, linking it all into a flat 64-bit image is just as bad as the 32-bit flat memory models.

[Schol-R-LEA]

Recommending real mode x86 as a serious starting point is as brilliant as recommending people seriously consider using FAT32 on floppies

Please lighten up and stop recommending it. Pretty soon everyone will be taking that technical toy novelty seriously.

I think that was Kazinal's point - that recommending it is a bad idea - but I guess the sarcasm didn't carry over. Kaz is saying that recommending writing a real mode OS without a compelling reason is just as bad an idea as using floppy FAT32 in a production design.

[Schol-R-LEA]

The fact is that you won't be doing just that, and if you try to write an OS, no matter how simple, you will surely won't stay just with your simple kernel but also trying to learn just about everything (computer math, algorithms, protocols, hardware programming, low level details for everything you program, etc...).

I'm personally trying to understand absolutely everything to make it more portable and make tutorials about it to make things simpler, as they were in the 90's and 2000's when the interesting things started and become less interesting after the more closed nature of the more recent programs starting with Windows XP, but it can be reverted if the environment is populated again with the highest quality tutorials that show you everything you need to know about how it looks like setting up, developing and compiling everything.

I would recommend against that; while there was a lot of information at the time on things now consider proprietary, I can tell you that there was a lot less information overall, and the quality of the information is on the whole awful. Even at the time, it was terribly misleading to those of us who were trying to learn from it. Much of it was straight-up misinformation, deliberately spread by people who wanted to make OS dev seem even more mystical and portentous than it is, but mostly, it was a huge pile of confused, poorly written and sloppily researched garbage. Relying on that material will set you back by years, and may even make you project impossible to complete.