OSDev.org

Chpetrou wrote:Hello

I finished the Bare Bones, and played with the kernel appearance and messages a bit. I mostly understood how the kernel works until now.

First, is this kernel a Microkernel, a Monolithic, or a Hybrid ? Also the mode of the processor and the memory management etc. are implemented in the GRUB boot-loader?

Second, i am trying as a starting point to make the OS work as a simple calculator, but how can i use the keyboard as an input, like in C++ "cin and cout", or C "scanf and printf" ?

Also how can i interact with hardware directly without any libraries or some kind of "help", for example you don't have some variables which count something that is happening in hardware to do something. How do you create drivers for a discontinued product?

Lastly, how can i implement something like a file manager and something like bash, and make a CLI OS like early BSD or early UNIX, and then make something like windows 1.0 which had the DoS behind and something like an early "GUI" which in fact was a graphical shell on top of DOS, or something like Mac OS System 1, or at least just a color window that each box is a program and the user can choose what program he wants? (I know that a GUI OS needs many things like display server, Desktop Environment and things like these)

Thanks in advance

Unless they state, teach and demonstrate otherwise, bare bones tutorials are monolithic by default because they contain it all in a single compiled unit, with no external libraries or stuff to load.

To use the keyboard, you need to make a PS/2 or USB keyboard driver that properly retrieves the bytes of the scancodes when you press, repeat, release and use modifiers like LEDs and Ctrl/Alt/Shift, etc... It will need concentration to write it unless you get to find a low level keyboard driver library.

If you want to make drivers for DOS or Windows, you need to learn about making DOS device drivers and TSRs. Normally you will need professional-grade specific drivers for those peripherals to initialize them (you might learn to use the most classic hardware devices, but beyond that there are so many that it's more practical to just learn to natively load the drivers they are shipped with in a CD from your OS). If you want to use new hardware under, say Windows 98 or Millenium, you need to develop VxDs, and make a VxD that allows you to use drivers from newer OSes like Windows XP 32-bit drivers. You'll need books and a lot of information and study to do it. It's possible but you need to get on par of the people who wrote things like KernelEx, ReactOS (open-source Windows XP clone) and Wine.

To make a file manager, you need at least to have FreeDOS as a base, you need the "DOSLFN" drivers to use Long File Names under DOS as if you were under a Windows command shell. You need to learn about how to make GUIs with tree controls and scroll bars, even if you plan to implement them in DOS mode.

You can use MS-DOS/FreeDOS as the base, as a shell and boot loader that can take you to a native GUI or to your own OS. From there you will have many examples on how to do all sort of things, but centering in keyboard/mouse input, FAT/exFAT/NTFS file systems and a UI (preferably graphical) to implement a good file manager with massive file management functions for DOS (and preferably being a 32-bit program) is the most useful start for an OS that starts implemented as a competent file manager and viewer.

You can implement more and more DOS, DPMI and WinAPI library functions on your own under DOS and link your program or programs against them, and you will already be doing a good amount of OS development work.

Bad advice.

Forget DOS and real mode, which are hopelessly old fashioned, and concentrate on protected mode programming. The jump from a simple bare bones tutorial to a kernel which does anything useful is a difficult one. Read the articles in the Wiki; perhaps start with the keyboard one.

iansjack wrote:Bad advice.

Forget DOS and real mode, which are hopelessly old fashioned, and concentrate on protected mode programming. The jump from a simple bare bones tutorial to a kernel which does anything useful is a difficult one. Read the articles in the Wiki; perhaps start with the keyboard one.

Forgetting about Real Mode is like not possibly having 16-bit data available in a computer. It's like having a computer capable to handle 32 and 64-bit instructions but not 16-bit ones even if they are vital for booting and also for lots of valuable historical code samples that demonstrate low-level progamming principles. I don't know how it was removed from 64-bit mode, and how the Linux people still implement emulators to run 16-bit code from it perfectly as if it was native (see 64-bit Ubuntu with Wine running old Win16 screen savers and useful scientific programs to calculate heat-transfer-related values and the such, among others).

There's nothing wrong with DOS. It has lots of basic information and great demos with code (as well as assembly optimization books and games) that will always be vital for an x86 PC, no matter how new. It's a very good filter to simplify stuff (for example, if somebody ported the X graphics server to DOS, it would allow to port the rest of GUI programs in a simplified format, good to learn).

If you reimplemented the BIOS as a 64-bit system and if you internally used native drivers for INT or API services, it would no longer be regarded as old stuff. It's just at the same level of efficiency and brevity as x86 Assembly. In fact, it should be possible to patch UEFI and go back freely to it, and to a 16, 32 or 64-bit classical open source BIOS.

Hi,

~ wrote:There's nothing wrong with DOS. It has lots of basic information and great demos with code (as well as assembly optimization books and games) that will always be vital for an x86 PC, no matter how new. It's a very good filter to simplify stuff (for example, if somebody ported the X graphics server to DOS, it would allow to port the rest of GUI programs in a simplified format, good to learn).

The key to duck shooting is to aim where the duck will be, not where the duck is now. Aiming for real mode/DOS is like aiming where the duck's grandparent's hatched.


Cheers,

Brendan

It can still be used as a boot loader for a system that doesn't require installation or altering partition or boot record structures but instead just resides in a known subdirectory in the root directory, and that is booted from external USB or disc media to be removed after boot.

In this way it's more capable than boot loaders like GRUB and it's not intrusive. It can perform more tasks in an easier way even at boot time, than GRUB or LILO than I know.

It's already just a shell for the BIOS. It could probably be used to patch UEFI and get back to an open-source BIOS that is 16, 32 or 64-bit.

It's like an extended boot one single step away from raw hardware with drivers for all the typical hardware, so it's still very convenient to simplify applications that come from open source. It still has support for updating tools like DJGPP and NASM, it seemingly is getting more updated recently and it would help clean up the existing code base specially for big projects like web browsers.

If it has all applications and drivers ported to it and as open source, I would prefer to use it instead of Windows or Linux due to its simplicity and direct hardware access. It would be like running on bare metal, no matter if DOS was used as the loader. I would like to do that same thing with Windows or Linux but it was never possible, so I have to prefer it over those bigger systems.

It will surely be a very healthy thing to see all existing applications ported natively to MS-DOS/FreeDOS. Everything becomes dramatically less bulky always. It makes programs more readily portable to any other OS, considerably faster, small and independent from system libraries potentially absent among OS versions slightly apart, resulting in programs that historically never stop running properly (something rare to see in a normal Linux distro... sadly I don't remember to have been able to just copy the binaries of a program from a Linux distro and still being able to run them in any other Linux installation).

Nobody said that more modern functions and drivers wouldn't be used in place of those from DOS itself. In fact they should be easily loadable and unloadable per application without having to reboot. But if they aren't needed, why not just use what the minimalist DOS already offers if that's enough to process a task?

And why not just use the raw hardware without BIOS or much heavier UEFI functions if raw hardware is enough to run and complete a task, as is constantly done in laboratories? It should be easy to reach raw hardware with no system instead of favoring UEFI and similar stuff like direct hardware acces that is impossible to fully enable in Windows or Linux, block that possibility.

Okay, I debated this, but I think I will post my 2 cents about this topic, and hope that a war does not start.

First off, modern machines no longer have 16-bit capabilities, and with UEFI, which starts in 32- or 64-bit mode, there is no reason to learn 16-bit nor is it required for boot loaders. With 64-bit machines using UEFI systems, 32- and 64-bit protected mode is the standard. Anticipate the duck's path, right?

With that said, I also think that using DOS is a good way to learn programming, though 16-bit segmented real mode is a waste for most and a tough chore to learn and then to break.

To use another idiom, don't use a canon to kill a mosquito. For the newbie, a simple hammer to pound the nail is better than rigging up a compressor, a hose, oiling the gun, filling the rack with "bullets", depressing the trigger, then tapping the board.

For example, when booting UEFI, it's easy to type: and watch it work, but you have to have the library that goes along with it. Learning what the library does and how it actually prints the string is the learning part.

Anyway, my 2 cents worth, take it or leave it.

Ben

BenLunt wrote:16-bit segmented real mode is a waste for most and a tough chore to learn and then to break.

This is one of the main reasons that I think it is a bad idea to spend any significant amount of time learning 16-bit programming on x86 processors. If you must write a bootloader (again, IMO, a bad idea for beginners) you need very little 16-bit code. The aim should be to get into one of the non brain-dead modes of the processor ASAP. This certainly means completely ignoring DOS, which is not a prerequisite for a boot loader.

BenLunt wrote:Okay, I debated this, but I think I will post my 2 cents about this topic, and hope that a war does not start.

First off, modern machines no longer have 16-bit capabilities, and with UEFI, which starts in 32- or 64-bit mode, there is no reason to learn 16-bit nor is it required for boot loaders. With 64-bit machines using UEFI systems, 32- and 64-bit protected mode is the standard. Anticipate the duck's path, right?

With that said, I also think that using DOS is a good way to learn programming, though 16-bit segmented real mode is a waste for most and a tough chore to learn and then to break.

To use another idiom, don't use a canon to kill a mosquito. For the newbie, a simple hammer to pound the nail is better than rigging up a compressor, a hose, oiling the gun, filling the rack with "bullets", depressing the trigger, then tapping the board.

For example, when booting UEFI, it's easy to type:

Code: Select all

IntializeBase(handle, table);
Print(L"Hello, World!\n");

and watch it work, but you have to have the library that goes along with it. Learning what the library does and how it actually prints the string is the learning part.

Anyway, my 2 cents worth, take it or leave it.

Ben

As far as I know, there are no x86 CPUs without 16-bit mode.

The only thing that is achieved with UEFI is concealing even further the implementation, workings and low-level programming for the actual hardware. It's also just one more layer that prohibits direct access to the hardware. With Windows and Linux is enough to do that. It just turns any machine into garbage with a functional form but fundamentally unusable, from the point of view of the developers who waited to use the advanced electronics since there are PCs, by themselves (i.e., all of us).

It also, in the end, just blocks the DOS as an OS and an architecture. As if somebody was afraid of its simplicity and power. As if somebody knew that even drivers under it are tiny, and then they don't want the drivers easily deciphered. I don't get why they conceal the code to the drivers from the whole world when manufacturing advanced hardware is so incredibly difficult already and when there are already so many hardware companies. It just makes no sense; manufacturers will always keep making money so not publishing the code and the specifications to the hardware is just a touch out of egoism and irrational mistrust.

The BIOS has the same level of efficiency than the x86 CPU itself because it was developed along with it. It's at the peak performance and code size efficiency point. If the concern if that it's 16-bit, now it could be perfectly shipped in 32 and 64-bit versions by turning NASM into a cross-assembler that automatically adjusts the assembly stage and the generation of data and instructions to the target CPU mode.

The system should always start in BIOS mode. UEFI should be shipped in a MicroSD card on the motherboard itself as a secondary OS that loads other OS. An image of it should be in the installation CD to repair it. There should also be 32 and 64-bit images of the actual motherboard's BIOS. In this way we could use, for example, the BIOS in 64-bit mode. UEFI or a 32/64-bit BIOS image should be optionally loaded by the BIOS. I would be willing to use UEFI only as a library, not as a replacement of the PC BIOS.

Instead of ending up fully concealing the internals of the hardware and the electronics because all hardware peripherals are supposedly always different (which produces an useless machine for people who would otherwise be taking advantage of it by themselves), they should just standardize the hardware. UEFI should boot and start from BIOS mode and should always be fully optional in all motherboards. If something like this doesn't happen, PCs will end up becoming just tablet-like machines with a BIOS-like system that could probably end up reaching 128 Megabytes of binary bulk in place of a BIOS that is just updated to 32 or 64-bit mode (there is already some work for 32-bit BIOSes so it could be done too for 64-bit, to keep the absolute minimum simplicity at boot time, and make UEFI a loadable OS for a motherboard -- as such it could even always use the same image across the Internet -- with this I suspect that UEFI really is just using the BIOS to start in a simple way, so it's just taking the place of an OS, and is just designed to be ripped apart at run time by any other OS that wants to load itself in UEFI mode).

As I see it, this sort of platform/BIOS software problem would be easily solved by turning Intel Assembly language as used by NASM/YASM/CPU manuals, into a Cross-Assembler, where you write code in generic/portable Intel x86 Assembly, and the Cross-Assembler would produce code for x86 16/32/64-bit modes, and also would translate the x86 Assembly source code into ARM binaries directly.

If what they want is a sort of cross-platform BIOS, they could port the original and simple PC BIOS to other architectures in this way, and it wouldn't necessarily be 16-bit code. It could internally really be 32 or 64-bit, and even ARM.

Stop posting bullshit. UEFI provides you with full access to the hardware after you call ExitBootServices.

~ wrote:Cross-Assembler

Such a thing exists – it's called LLVM.

~ wrote:cross-platform BIOS

It's called UEFI.

Roman wrote:Stop posting bullshit. UEFI provides you with full access to the hardware after you call ExitBootServices.

~ wrote:Cross-Assembler

Such a thing exists – it's called LLVM.

Why not use plain, standard, vanilla x86 Assembly as if it was a native assembled language and a compiled language for other architectures? It would be more popular, and the huge advantage would be that it would finally end up standardizing Assembly programming to a single language for all CPU architectures, taking advantage of the weight that the x86 CPU already has in terms of available programs written in Assembly (and also the weight of compiled programs that could be disassembled into ASM source code to be used and recompiled).

I already demonstrated that Intel x86 Assembly language just needs a few additional data types and instruction names that automatically select the CPU operation mode and machine word size, to become portable along with a careful platform-independent design. The CPU already selects the operation mode automatically... Why not the Assembly language? You can see instructions switch CPU mode sizes with the HIEW hex editor, specially, versions 7/8 and above, which support 64-bit code (I guess I got a big part of the idea to have the Assembly language itself automatically select the CPU operation mode size from HIEW when doing this).

If you want a second assembly language, the Assembler could also use ARM assembly language and assemble for native CPUs/compile to different architectures.

Roman wrote:

~ wrote:cross-platform BIOS

It's called UEFI.

Why not use the sort of improved assembled/compiled Assembly language I mention above, and load UEFI from a secondary MicroSD card on the motherboard? The language itself would make it cross-platform even if it's written purely in x86 or ARM assembly, with the improved Assembler/Compiler tool mixture (so NASM.EXE and YASM.EXE would become a selective compiler and assembler in a single application, and differences in architectures would be left to the Assembler itself, not to ever bulkier run-time layers).

It would also allow to load a 32-bit or a 64-bit BIOS image instead of UEFI, and apart from the original 16-bit BIOS, those bulkier systems, specially UEFI, would become optional. The PC would benefit because its roots would stay untouched while being able to go from BIOS, to UEFI, or to native 32/64-bit BIOSes (there is already some work on 32-bit BIOS code so it's not impossible to extend it while keeping things simple and still standard, virtually unchanged while extending).


Current hardware manufacturers aren't keen on standardizing hardware, so trying to remove the BIOS, stop using or removing Real Mode from the 16-bit CPU and insisting in using obligatorily a system like UEFI that strangely has protection features like a final OS without first executing a normal BIOS to allow full control over the machine as has been so far, just cannot be an effort to standardize, but just getting rid of the x86 PC in favor of some other format.

1. x86 assembly language was never designed to be portable.
2. What "weight" are you talking about? Can you name a couple of programs written in x86 assembly significant these days?
3. LLVM is a huge project with a corresponding community. It works, it is widespread and used by corporations like Apple and Google. No one cares about your portable x86 dialect.
4. I see no purpose in loading UEFI or whatever from SD cards.
5. I already told you UEFI lets you be free after calling a dedicated function.

BTW, your website is [hopefully?] not accessible for me. Is that because you host it on a homegrown megalithic bare metal HTTP server running in real mode on a i286 DOS PC?

Roman wrote:1. x86 assembly language was never designed to be portable.

Probably it was, at least within its own architecture. It's really obvious how tightly optimized its designers necessarily had to optimize it while making its language software-oriented, instead of signal-oriented or machine-oriented like other architectures.

Roman wrote:2. What "weight" are you talking about? Can you name a couple of programs written in x86 assembly significant these days?

Most of the portability is already covered at compile time without creating a whole new architecture or a virtual machine if you treat existing vanilla Assembly as a compiled and assembled language at the same time, and even more if you add automatic-sized data types and instruction names, to be used for code written in the future.

DOS OSes. I guess that Windows 9x was also written in assembly. It's suspiciously fast compared to later Windows versions, even in a 386SX without FPU.

And if we take into account that we can just disassemble the code from existing x86/x64 binaries and turn it into portable ASM files, then we can clearly see that we could compile that disassembly to other CPU architectures, Without Having Any Source Code At All, without emulating, just recompiling the disassembled sources to other architectures or to 32 or 64-bit x86. That includes disassembling and natively porting/compiling old abandonware programs to any other architecture. If there are instructions that can select operand sizes across modes, then they should be coded to do so automatically in the Assembly language.

If Assembly is compiled, even features like paging or segmentation can be interpreted in the compilation stage and proper code produced by a compiler of assembly source files.

Roman wrote:3. LLVM is a huge project with a corresponding community. It works, it is widespread and used by corporations like Apple and Google. No one cares about your portable x86 dialect.

It's not a dialect. I already said that the x86 CPU can select the format, the instruction and operand size automatically in such a way that the same Assembly source code could be used across 16, 32 and 64-bit modes.

It isn't a dialect. You just need to add a few data types and registers like WIDEAX (16, 32 or 64-bit register), WIDEWORD (16, 32 or 64-bit memory variable), LODSWIDE/STOSWIDE (instructions that use the maximum word size of each CPU mode instead of hard-coding it). And so on...

It's not a dialect if all you have to do is integrate those functions into the assembler instead of having to patch the code with preprocessor directives that in the end just achieve reaching CPU functions that probably couldn't be figured out how to use, like automatically selecting machine word sizes in Assembly, which holds a really big percentage of the portability or non-portability problem. The solution is tiny as you can see:
http://archive.org/download/x86_Portabl ... rtable.asm

http://www.archefire.org/_PROJECTS_/x86 ... e.asm.html

Roman wrote:4. I see no purpose in loading UEFI or whatever from SD cards.

It allows to keep a clean architecture. It would be like loading an official OS. It would allow shipping 32 and 64-bit pure BIOSes in pure BIOS mode for those modes. It would allow the easiest update the motherboard firmware without risking brick a motherboard and being able to revert to any point of firmware updates (even custom ones).

~ wrote:Why not use plain, standard, vanilla x86 Assembly as if it was a native assembled language and a compiled language for other architectures?

So how do you load a GDT, the Page Table Register, etc.? These things just don't exist in "vanilla x86 Assembly". I thought your idea of recording programming step-by-step was pretty stupid, but this is a far, far more ridiculous idea.

What is your problem with the more advanced modes offered by modern processors? Don't you understand how they work? The idea of crippling a computer by running a 35-year old operating system on it, and ignoring the advances in processor architecture since then, makes no sense whatsoever.

iansjack wrote:

~ wrote:Why not use plain, standard, vanilla x86 Assembly as if it was a native assembled language and a compiled language for other architectures?

So how do you load a GDT, the Page Table Register, etc.? These things just don't exist in "vanilla x86 Assembly". I thought your idea of recording programming step-by-step was pretty stupid, but this is a far, far more ridiculous idea.

What is your problem with the more advanced modes offered by modern processors? Don't you understand how they work? The idea of crippling a computer by running a 35-year old operating system on it, and ignoring the advances in processor architecture since then, makes no sense whatsoever.

Proper design must be kept. Details that will always be platform-dependent must be in the low-level module of the application. It can still be compiled if the same functions are produced by the compiler, but it's reasonable to have a low-level platform-dependend module always. There will always be native assembly and assembly code that can be compiled to any architecture. Solving the full platform-independent portability for x86 assembly (and very probably ARM) is nearing the size of the code shown here:
http://archive.org/download/x86_Portabl ... rtable.asm

I have nothing against them (32/64-bit modes... I use them along with 16-bit).

I'm just pointing out that Assembly language, x86 and ARM together, can become an assembler as well as a compiler, fully portable language. They already did most of the work, and they would require no virtual machines to achieve portability.

The problem are also the 32 and 64-bit OSes there which don't allow full access (it's impossible to tell Linux or Windows, for example, to run every program in Ring 0 if you really want or need it, if you use a Registry value or configuration flag, or a special driver shelved on the CD).

FreeDOS ported to pure 32 and 64-bit modes, keeping the original 16-bit version (and emulating it in 32 and 64-bit modes with an internal emulator handy) would solve it.

x86 CPUs still have 16, 32 and 64-bit modes. If they remove any of them, specially the 16-bit one, is no longer an x86-capable CPU (the wildcard X would no longer cover it). That's the grace of the x86 architecture: Adding modes without getting rid of what it already had.

~ wrote:The problem are also the 32 and 64-bit OSes there which don't allow full access (it's impossible to tell Linux or Windows, for example, to run every program in Ring 0 if you really want or need it, if you use a Registry value or configuration flag, or a special driver shelved on the CD).

That's because, unlike DOS, those operating systems provide a full API to allow userspace software to do everything it should ever need to do, and running applications in ring 0 would only be giving them more room to cause trouble for the rest of the system.