Why hardware has to be so complex?

[pranavappu007]

I am new to low level programming, I would say I'm a beginner in assembly and mediocre at C. Doesn't know to do complex stuff in assembly, knows to move around stuff and call subroutines and increment counters
Fairly new to osdev, and ideas of making hobbyist OS. Tried to make a toy OS of my own, and it does work(somewhat). But what I learned through this 'experience' ( ), if any, is the actual hardware and communications and configurations of them is so complex than normal programmers would ever encounter, that only elite programmers would be able to make something useful out of it.

Why the processor, communications and stuff has to be so complex? Why something simple like real mode with a simple MMIO + an out command to initialize an operation won't work? If setting up a structure like GDT wasn't hard enough, you need to set IDT for interrupts(I didn't bother to ). If you want to go 64-bit mode, you need to implement paging, which I don't even know enough to talk about. Implementing a PS/2 keyboard driver was hard(and I did it with polling still), getting time for rtc was even harder. Coding actual kernel in C was faar better compared to this, and was far more satisfying to see results with each step of improvement. Then came to filesystem, my boot drive was a USB, and I needed to switch to real mode and use BIOS. Hell broke loose(eventually made it working, until it decides to crash again). Then there is advanced topics like multi threading, ACPI, VGA controller, PCI and USB, sound, networking etc which I might never have the courage to touch.

I wonder why all of this has to be this tough? Is there no way to simplify it a bit without loosing much efficiency? Does the actual designers even think about the programmers that work on these? Or this level is only meant for elite ones??

[nexos]

You are probably right that it is meant for the elite. Big companies like Intel or concerned about Windows and Linux following suit with there standards, but care less about OS developers like you and me. Things such as paging and stuff are required for security and paging. Hardware level security causes complexity, yet is very important. My OS is going to be modern, but I am doing it for fun, not to compete with Windows or anything.

[nullplan]

You are probably right that it is meant for the elite.

Hanlon's razor (never attribute to malice that which can be adequately explained by stupidity)! In this case, the GDT stuff is more complicated than what other architectures have, but that is easy to say with the benefit of hindsight. The stuff was created for the 286 in the early 80ies. It did not end up meshing with where computer architecture was going, but the Intel guys could not see the future. Besides, these days the GDT is just filled with a few segments, most are constant, some aren't, and then that code is just left alone.

This stuff is not hard because you have to be elite to do it. It is hard because of lack of foresight and design. Some stuff is hard because of hardware complexity, like the transfer queue stuff with USB and AHCI, but it does allow some remarkable things to happen. I personally would not want to have to react to USB bus cycles in CPU.

[nexos]

In the case of the GDT, nullplan is right. But when it comes to ACPI, that is different. I am highly critical of ACPI and think that you truly have to be elite (like Microsoft) to use it fully. My OS uses the MP tables instead.

[iansjack]

I think you overestimate the difficulty of a lot of stuff that you talk about. Most of these things aren't hard so much as unfamiliar. Once you are familiar with the concepts of interrupts, paging, etc. then it's all a lot easier than when it's a foreign language. But, you've got to practice, just like playing a musical instrument. Why is playing the piano so hard - why can't I just sit down and knock out a tune? And, if it's so hard, how come so many idiots can do it?

The one thing that can be hard is programming hardware for which there is inadequate documentation, but that's not true for most of the basic stuff. In particular, the processors are documented in amazing detail. So rather than thinking it's all too hard, get stuck in and read about these things and play with them. But, in the end, an OS is a highly complex piece of software. As a comparison, consider a language compiler - another complex piece of software. There's nothing hard about that by design - after all, there's no special hardware or language constructs involved, just straightforward programming. And you're free to design it in any way you choose, so you can't complain that the design is too difficult. But try writing a language compiler.

In the end very little that is worth doing is easy. If it was, it wouldn't be of much interest. There's nothing special about OS programmers - they're just prepared to take the time to learn new concepts and master them.

[pranavappu007]

That's why it's hard. It's hard to familiarize. You can't just make an OS like windows or linux even if you take years. I don't think you can make a even a kernel as hard as linux, on your own. I was into this for more than 3 months, and I still find GDT hard and IDT not yet completely possible.

For example, I had programmed a text editor inside my 'OS', and it was hard to program. Bugs and mistakes will occur when trying to add every single features. But it's just basic C and if you look at it and debug it, eventually you will find it as an increment error or flaw in algorithm or something.

Hardware is not like that. You have set up something, suddenly it crashes. You have no idea where it happened. At best you can put print commands to find out the code which it crashed. If you are lucky, it's just a stack issue or something. Or else, start guessing and good luck. At end out of sheer luck you will find that some weird structure you loaded and working for a while, that one bit you couldn't understand it's use to set it properly, is wrong in for this specific operation and should be changed without breaking any other part.

Interrupts and communication is better in that case as it's entirely upto the programmer to decide whether to set it up( but you NEED exceptions and stuff and it's still half as hard as setting up the entire thing) But still, it would be nice to have a simpler alternative for the ones who don't need 'Advanced Security' and other bells and whistles, to get familiar with it. And the fact that some CPU(older) can go to unreal mode and such is so horrible, espicially you won't notice an error if using newer hardware.

[pranavappu007]

IMy OS uses the MP tables instead.

Didn't know yet. Read on wiki. Didn't understand anything except used for multithreading activation and is older and OEM independent than ACPI. Thankyou for info anyway.

In this case, the GDT stuff is more complicated than what other architectures have, but that is easy to say with the benefit of hindsight. The stuff was created for the 286 in the early 80ies. It did not end up meshing with where computer architecture was going, but the Intel guys could not see the future. Besides, these days the GDT is just filled with a few segments, most are constant, some aren't, and then that code is just left alone.

In the case of a simple flat GDT, you are right. set it up and forget about it. Bt if different segment descriptor as loaded with different selectors and GDT is complex, that is a pain to set it up and also remember(or else you can mess up the offsets). And then there is LDT and stuff.
My main point was not GDT, it just includes it. You are right that out of all these, GDT is probably thesimplest and easiest. But I still wonder why everything needs to be this hard? Just in the name of security? Is it impossible to implement security that is simpler to set up, or make it optional?

My OS is going to be modern, but I am doing it for fun, not to compete with Windows or anything.

Me too, never wanted to destroy windows and be the next linus or anything. But I find it so hard that it takes away all the fun. Then what is the point of doing it?

[iansjack]

Perhaps you've just chosen the wrong hobby.

We choose to ... do these things, not because they are easy, but because they are hard, because that goal will serve to organize and measure the best of our energies and skills, because that challenge is one that we are willing to accept, one we are unwilling to postpone, and one which we intend to win, and the others, too.

[pranavappu007]

Perhaps you've just chosen the wrong hobby.

uhm... probably. At least it wasn't a total waste of time, as I learned these things exist. And the experience in C without any library support may be worth it. I'll just keep my 'project' as it is, debug it a bit and be done with it,keeping it as a remainder of my abseloute limit.

Planning to get my hands on Windows API, or some framework to make apps in windows(legacy incl.) any suggestions are welcome(this forum is gneral ramblings so these topics are allowed here, right?)

[iansjack]

The Windows API is fully documented by Microsoft: https://docs.microsoft.com/en-us/window ... s-api-list

The tools to develop with it are available for free here: https://visualstudio.microsoft.com/free ... er-offers/

This book is a useful reference: https://www.amazon.co.uk/Programming-Wi ... 157231995X

[thewrongchristian]

You are probably right that it is meant for the elite.

Hanlon's razor (never attribute to malice that which can be adequately explained by stupidity)! In this case, the GDT stuff is more complicated than what other architectures have, but that is easy to say with the benefit of hindsight. The stuff was created for the 286 in the early 80ies. It did not end up meshing with where computer architecture was going, but the Intel guys could not see the future. Besides, these days the GDT is just filled with a few segments, most are constant, some aren't, and then that code is just left alone.

I often muse what Intel might have achieved with the 286 had they gone in a different direction with regards to the protection model, given their transistor budget (I think the 286 was ~125K transistors).

I was thinking something along the lines of them keeping their segment registers as per 8086, translating pointer offsets using the segment<<4 as a base, to a 20-bit linear address, then just sticking a paging unit in to translate to physical addresses. A simple, software managed, 16 entry TLB would have given 64KB of reach at any one time, easily enough to cache working set translations of programs of the time.

They'd have been compatible with 8086 memory model, while allowing page based isolation and protection, and made all our lives simpler down the line.

I'd love to know how this sort of model would have matched up against their hideously complex protection system they adopted for the 286.

[pranavappu007]

I was wondering why all the protection systems has to be hardware based. Why even run user binary directly on CPU? Even old systems in 70s ran interpreters like BASIC,with the exponentially faster CPU and other peripherals, now we can't afford the overhead of an interpreter? Even android apps are running on top of a Java VM. We can make a kernel which has complete control of the CPU and all events externally or internally, without anything from the hardware. We can even simulate time-sharing with some clever programming, I suppose. All we might need is a mechanism to catch exceptions, to manage multiple threads and their initialization.

I think experienced programmers in here can think of even more practical and effective designs than this..

[linguofreak]

I was wondering why all the protection systems has to be hardware based. Why even run user binary directly on CPU? Even old systems in 70s ran interpreters like BASIC,with the exponentially faster CPU and other peripherals, now we can't afford the overhead of an interpreter? Even android apps are running on top of a Java VM.

Part of the answer is that we're doing harder things with software now, that the old 70s/80s home computers didn't have the performance to handle on bare metal, let alone an interpreter. No matter how fast your CPU is, there will always be some problems that it's too slow for, and so interpreter overhead will always sacrifice *something*.

Another other part of the answer is that, when we aren't doing those things, we can afford the overhead of *many* interpreters. Running on my system right now, I count 1 instance of bash, 1 of java, 3 of make, 1 of python, 3 of plain old sh, and 6 of zsh. Plus, IIRC, most of my Desktop Environment is written in python, so there's probably a heap of different python interpreters running that don't have the process name "python".

And then there's the fact that a buggy interpreter can always be tricked, but, interpreted or not, if each process has its own address space, then the data of other processes doesn't even *exist* as far as that process is concerned. Admittedly, you can have kernel or CPU bugs that degrade this protection, but putting an interpreted process in its own address space prevents it from getting unauthorized access simply by exploiting an interpreter bug. There needs to be a follow up to the attack on the interpeter bug that attacks the kernel or CPU.

[iansjack]

Why do in software what can be done in hardware? The whole point of computers is to make life easier, so if protection can be built in to the hardware, perhaps augmented by software, that's the way to go. It is, IMO, far harder to build a reliable, secure system using only software than to rely upon hardware protection mechanisms like paging. Besides, paging has many more advantages than just security.

Paging is a very good example of what newbies find difficult, but is in fact very easy and elegant. (I discount segmentation problems here as it is trivial to set up a flat segment model and then forget about segmentation.)

[bzt]

with the exponentially faster CPU and other peripherals, now we can't afford the overhead of an interpreter? Even android apps are running on top of a Java VM.

Interesting that you mentioned that, you see Android does not use Java VM, because it's way too resource consuming. It's using Dalvik VM much more suitable on mobile platforms (both Java bytecode and Dalvik bytecode is compiled from Java source, but they are very different).

Otherwise you're right. Programmers are wasting resources, they don't know how computers work any more. They just import a huge library, that's it. Take a look at Wirth's law (if you don't know who that is, Wirth created Pascal, Modula and Oberon languages), or May's law: "Software efficiency halves every 18 months, compensating Moore's law". Unfortunately this has a drawback on hardware too. Most of the hardware complexity is caused by incompetent firmware implementations (EFI, ACPI, SMM, etc.) The way how an UART chip for example works hasn't changed in the last 40 years.

Paging is a very good example of what newbies find difficult, but is in fact very easy and elegant.

Yes, I agree. Here we can see that it is not the hardware that's changed, it is the mentality of the programmers. (Thankfully there's no firmware component to paging, it would be difficult to mess that up too )

Cheers,
bzt