Page 2 of 3

Re: Some Questions about continuing with OS and hardware

Posted: Tue Aug 23, 2016 9:44 am
by Roman
Running programs in ring 0 is pointless and horribly unreliable and insecure. If you want to tinker with hardware you can write a kernel module.

Re: Some Questions about continuing with OS and hardware

Posted: Tue Aug 23, 2016 9:50 am
by ~
It's not the same in any way.

It would be valuable if you wanted to see how everything works without a debugger. It would be useful and trivial. As useful as packing and distributing the full original toolchain and development environment used by the developers of each open source project instead of expecting everyone to have development tools properly configured for every project in every platform.

Disassembly technologies are part of archaeologic, historic, unknown meaning interpretation and investigation abilities, and apparently are a big part of platform independence for software and hardware (like PCI, USB...).

BIOS services are designed to be non-intrusive and working without the need to restrict access to anything so that's what makes them attractive to use and to port to 32 and 64-bit.

I guess I should give a try to see how the most simple and lightweight option goes at the assembly side, make assemblers more portable (which would also help disassembly technologies for all kinds of data and code, not just executables but also data maps where several possible options can be taken for each data field/buffer) and actually compile assembly to foreign architectures, while there are others developing VM projects and new languages, and share the implementation results, don't you think?

Re: Real Mode (was: Some Questions about continuing with OS.

Posted: Tue Aug 23, 2016 2:33 pm
by Brendan
Hi,

A) Pure assembly is virtually never portable; not "portable between different CPUs", not "portable between different modes of the same CPU", and not even "portable between different OSs that use the same mode on the same CPU".

B) If you're going to use some sort of translator to convert assembly into something that can run; then assembly is an inferior choice, because it's full of the idiosyncrasies for a specific case that no longer apply, and because it was never designed to do this efficiently. Any alternative that's actually designed for this purpose (LLVM byte code, CIL/.NET, Java byte-code, ....) are all superior.

C) If you're still going to use some sort of translator to convert assembly into something that can run (despite the fact that it's inferior); then using assembler macros to do the conversion is incredibly stupid because there is no chance of any kind of optimisation being done on the generated code.

D) There are only 4 reasons to use assembly:
  • To allow you to do things that portable languages can't do; which is destroyed by the entire "portable assembly" concept
  • To achieve "hand optimised for a specific CPU" performance; which is destroyed by the entire "portable assembly" concept
  • For education; where you can learn assembly for the actual CPU instead of learning a mangled mess.
  • For fun; where it doesn't matter how idiotic or retarded something is someone might think it's fun (but that doesn't make it less idiotic or less retarded)
E) For 80x86 "real mode vs. something not deprecated"; this is actually 2 very different subjects being combined. The first is "real mode vs. protected mode or long mode" (which includes using real mode code that relies on "wrappers around UEFI functions"; and includes using protected mode for code that relies on "wrappers around BIOS functions"). The second is compatibility and BIOS vs. UEFI.

F) For "BIOS vs. UEFI", if you care you're doing it wrong. For a well designed OS, the only code that cares what the firmware happens to be is the early boot code, and in that case I'd recommend having multiple boot loaders for multiple types of firmware (e.g. one for BIOS, one for UEFI, one for OpenFirmware, one for coreboot, ...).

G) For "real mode vs. protected mode or long mode"; there's is very little difference between real mode code, 16-bit code (for a protected mode environment) and 16-bit (for a long mode environment) - the main difference (and the only difference for normal "application code" that doesn't/shouldn't touch things like IDTs, etc) is how segment loads work. However, both real mode code and 16-bit code (for protected mode and long mode) is painfully limited due to 16-bit addressing, and for this reason real mode code and 16-bit code should be avoided unless there's an extremely good reason (e.g. you're writing early boot code for BIOS).



Cheers,

Brendan

Re: Real Mode (was: Some Questions about continuing with OS.

Posted: Tue Aug 23, 2016 3:32 pm
by ~
Another reason to learn about 16-bit segments is being able to manually write (without code or resource compilers nor linkers) DOS and Win16 MZ/LE/NE EXEs as easily as if we were writing COM programs, knowing exactly how to span more than 1 64K segment, while knowing what's going on in precise technical terms. Difficult memory partition is seen in every single existing CPU, from a 6502 or a microcontroller, to 64-bit or higher CPU modes, so learning segmentation shouldn't be surprising.

Another reason to use pure assembly is to master all internals completely to understand how all libraries work. It not only applies to executable code but also to file formats, where all data can be treated as an execution data map (values from each field and buffer are corresponded with one or more functions to execute according to what they have... in this way we can add more functions to a pointer array to handle the different values... the most fundamental is corresponding a value or a value range into a direct or scaled array index to avoid modifying the base code and adding too many conditionals/IFs, like scripting native code). If such "executable data maps" are used, programming in assembly becomes much easier, and also higher-level languages, requiring many less IF constructs and becoming more directly reactive to the values know and the actually present ones.

In any case, compiling x86 assembly to other CPU will always tend to be more microscopic than using higher level languages. If we know and master the file formats used by the programs themselves and the data they handle, then programming in assembly language isn't a problem in itself nor more difficult than other languages because we will have all the normally hidden information at hand, in a clear-to-read format. Only system-level instructions and features would be difficult to port to other architectures, but not impossible when it comes to manage the system in the same way as a given CPU, in software; and it would be better to put such system-level functions in a platform-dependent module even though it can be made portable by an assembler that compiles to other CPU instructions.

If the C language can be portable with all of its confusing syntax and nested expressions, then Assembly can perfectly be done. We just need to fine-tune and extend its programming paradigm and write well-crafted code, as if we were defining an electronics circuit and its microcode. Then it will be much easier and reasonable to master it and write full applications in it that can be ported anywhere, with platform dependent and independent components perfectly separated in two packages.


I have created a code file full of macros to implement data, registers and instructions that will adjust automatically depending on the target mode or platform by using their names. Ideally they should be a part of the language. It's already part of the CPU in that most of the same opcodes are interpreted as the same instruction across CPU modes, so it's portable across modes. Now it's the assembly language itself the one that needs to be slightly improved so it also adjusts automatically, like the CPU does across modes (and which effect you can see by switching between 16, 32 and 64-bit modes in HIEW32 7 or 8).

At this level, optimizations should no longer be a mysterious art but the optimizing combinations of instructions should even receive their own mnemonic names as if they were single instructions.

Re: Real Mode (was: Some Questions about continuing with OS.

Posted: Tue Aug 23, 2016 3:46 pm
by Kazinsal
No one in their right mind cares about DOS linkers and executables anymore. That technology is so old and obsolete Microsoft decided it wasn't worth supporting going forward on modern architectures eleven years ago.

I don't understand your obsession with trying to revive such ancient and dead technology. It was supplanted for a reason, and that reason is because it sucked like explosive decompression.

Re: Real Mode (was: Some Questions about continuing with OS.

Posted: Tue Aug 23, 2016 3:48 pm
by iansjack
~ wrote:If the C language can be portable with all of its confusing syntax and nested expressions, then Assembly can perfectly be done.
The truth is that you are not talking about making assembler portable (an obviously impossible task because of the fundamental differences between different processors), but in creating a layer on top of assembler; a higher-level language. You are many years too late in doing this; languages such as BCPL and C were created many years ago and serve exactly the purpose that you are looking at. And the truth is that well written C libraries are likely to be far more efficient, and perform better, than an assembler programmer can manage (unless they are an extremely good assembler programmer). You can take it that, after many years of development, there are efficient libraries available.

You are trying to reinvent the wheel. Unfortunately, your concept of the wheel is a square one rather than the well rounded examples that already exist. By all means go and develop your ideas; but I think that they, and your rather quaint ideas about DOS, have little place in these forums - perhaps stick to the "General Programming" category if you really must, but they are only an unnecessary distraction in the forums aimed at operating system development.

Re: Real Mode (was: Some Questions about continuing with OS.

Posted: Tue Aug 23, 2016 4:16 pm
by ~
Kazinsal wrote:No one in their right mind cares about DOS linkers and executables anymore. That technology is so old and obsolete Microsoft decided it wasn't worth supporting going forward on modern architectures eleven years ago.

I don't understand your obsession with trying to revive such ancient and dead technology. It was supplanted for a reason, and that reason is because it sucked like explosive decompression.
Even 64-bit Linux is capable of running Win16 and giving the impression that they are native. What's good about DOS is that if somebody writes a driver and if it runs in DOS, it will be tiny, easy to understand, the device will be well implemented and well oriented to final applications while keeping the intended power it should offer, and would run well in bigger OSes, and the same for existing applications, even the more modern web browser under DOS. A program can expand itself to 32 or 64-bit and with its own functions without having to get through Windows or Linux or having them monopolize the whole machine (bad for a machine intended for very heavy development at all levels where you need to switch from low to high-level development without delays to be efficient).

If Windows/Linux at least allowed to run the whole user space and programs in Ring 0, at least for a special user, it would become as easy to use and flexible as DOS: We could use a few library functions to initialize something totally or partially, and we could perform the rest of the execution on the raw hardware and our own functions, avoiding having to learn libraries for something we could do with our own knowledge and resources.

iansjack wrote:
~ wrote:If the C language can be portable with all of its confusing syntax and nested expressions, then Assembly can perfectly be done.
The truth is that you are not talking about making assembler portable (an obviously impossible task because of the fundamental differences between different processors), but in creating a layer on top of assembler; a higher-level language. You are many years too late in doing this; languages such as BCPL and C were created many years ago and serve exactly the purpose that you are looking at. And the truth is that well written C libraries are likely to be far more efficient, and perform better, than an assembler programmer can manage (unless they are an extremely good assembler programmer). You can take it that, after many years of development, there are efficient libraries available.

You are trying to reinvent the wheel. Unfortunately, your concept of the wheel is a square one rather than the well rounded examples that already exist. By all means go and develop your ideas; but I think that they, and your rather quaint ideas about DOS, have little place in these forums - perhaps stick to the "General Programming" category if you really must, but they are only an unnecessary distraction in the forums aimed at operating system development.
It's not impossible. Think about porting an application that has been properly encapsulated and defined, which is intended to write on the screen, or functions to perform arithmetic operations for a calculator. If you think about how to compile x86 assembly to other native executable code, you will find that it's basically the same as compiling C, and we aren't using anything exotic, just clean x86 assembly well-defined to be cross-platform. If for example we are masters in using SSE and we want to use it in other architecture, it would be very useful if we already wrote an x86 application.

If it wasn't possible to compile one assembler into a different object code, then it wouldn't be possible to use PCI, accelerated 3D graphics cards, IEEE-754-compliant FPUs or USB in architectures other than x86 PCs either.

About DOS, I know that many companies have tried to simply erase it due to the fact that everything that is written for it is so tiny that it's very easy to disassemble and port to other architectures and OSes, specially drivers. But technically it's the second stage of the BIOS. It's just a shell for the BIOS that can take us, for example, to a portable LILO or GRUB, which doesn't require actually altering any installed hard disk, or start another OS from a subdirectory. We can either use DOS functions or implement our own (why not implement DOS-like services for 64-bit?).

As I said, given the current impossibility to run everything purely in Ring 0 under the current major OSes, it's good to learn what is in DOS.

_______________________________________________________________________________________
_______________________________________________________________________________________
_______________________________________________________________________________________
_______________________________________________________________________________________
Just think about the fact that if we turn the WinAPI, the Linux kernel components and other OSes' components into mere libraries to link applications against, and whether we run programs booting them directly or launched from DOS, and if we add programs like Firefox and a layer to load Win9x/XP/Win7/Linux drivers while any program loads, under DOS, then it would no longer look like DOS, but like a custom machine with a custom system which we can learn, run and configure as we like, without having to use any particular OS at all, but DOS with SATA and other modern drivers, and maybe ported to pure 32 and 64-bit layers running in Ring 0, as well as the BIOS (BIOS32, BIOS64) is a good minimum, in practice more reasonable than booting every user program in place of an OS (although that's also a good test to know how little BIOS/OS services and libraries it needs to exist and run in memory all by itself).

If there are people from the 80's and 90's who can better understand how to achieve it (running everything with a minimum base as FreeDOS but without a particular OS to run programs), than with other OSes (used by uninteresting home and office tasks), then that's the best choice for them. It has good things to offer like simplifying code and expressing it in considerably lesser instructions; there are things to do and things to offer like happens with people from other OSes, UNIX or not.

In fact, remember how many old games were designed to run without OS at all (maybe they didn't want to pay for OS licenses to keep profits if any, and the profit of keeping an excellent concise programming level), and today they are still able to run directly as boot code, and use the minimum imaginable resources (like these 8086/8088 games and demos: Zaxxon, PacMan, JansFlame fire effect, 2DROT/3DROT... try to set up and boot old programs without OS... they all can run without OS and almost using no BIOS services).

That's the idea with DOS: Tilting to raw hardware or to existing OSes turned into libraries for any and every other OS so we can use whatever we want, not just traditionally-packed OS software.

Re: Real Mode (was: Some Questions about continuing with OS.

Posted: Tue Aug 23, 2016 4:54 pm
by ~
If you aren't thinking to use any existing kernel, if you aren't thinking to port major existing applications, if you aren't thinking to write that many user applications, if you only really care about porting the standard C and C++ libraries and the standard toolchain (compilers, assemblers, linkers, loaders), then you probably don't want to implement an end-user OS, but maybe another sort of test bed for a very particular OS that only you know well, probably for machines that aren't x86 and which wouldn't be affected by using UEFI because they have already had it, probably like the Raspberry PI or highly-capable Arduinos, phones and tablets at the very least, etc. Probably you are also thinking to implement a hardware device to run it all on, using existing electronics and CPUs, and available electronics-manufacturing services for the printed circuits, etc.

Re: Real Mode (was: Some Questions about continuing with OS.

Posted: Wed Aug 24, 2016 12:39 am
by iansjack
~ wrote: If it wasn't possible to compile one assembler into a different object code, then it wouldn't be possible to use PCI, accelerated 3D graphics cards, IEEE-754-compliant FPUs or USB in architectures other than x86 PCs either.
I begin to wonder if you actually understand what assembler is, or appreciate the differences between different CPU architectures.

What on earth has the native language of the CPU got to do with PCI, 3D graphics cards, USB, etc.?

Re: Real Mode (was: Some Questions about continuing with OS.

Posted: Wed Aug 24, 2016 1:34 am
by tsdnz
iansjack wrote:
~ wrote: If it wasn't possible to compile one assembler into a different object code, then it wouldn't be possible to use PCI, accelerated 3D graphics cards, IEEE-754-compliant FPUs or USB in architectures other than x86 PCs either.
I begin to wonder if you actually understand what assembler is, or appreciate the differences between different CPU architectures.

What on earth has the native language of the CPU got to do with PCI, 3D graphics cards, USB, etc.?
The guys are right mate, not sure if you really understand the difference or where you are trying to head.

What on earth has the native language of the CPU got to do with PCI, 3D graphics cards, USB, etc.?

Very true, nothing.

Ali

Re: Real Mode (was: Some Questions about continuing with OS.

Posted: Wed Aug 24, 2016 1:39 am
by BrightLight
iansjack wrote:What on earth has the native language of the CPU got to do with PCI, 3D graphics cards, USB, etc.?
LOL! =D>
~, please stop posting this DOS sh*t everywhere, and stop spamming your weird assembly dialect everywhere in big font. The latest version of DOS was released years before I was even born. No one needs DOS anymore. And current OSes use 32-/64-bit modes for a reason. They also use ring 3 instead of 0 for a reason as well. If you want to program in DOS for your personal entertainment, then go right ahead. But TBH, you can't expect this forum to agree with you; it's too obsolete and outdated. How do you feel, for example, using your PC without speakers and in 640x480x16 colors screen resolution when it supported 1366x768 and 3D acceleration?
No one can port the X Window System, neither all our current apps to DOS; the accessible memory isn't even enough. BTW, DOS doesn't support multitasking; it cannot even run the X Window System and GNOME or KDE for example at the same time. Add that to my average load (Chrome, VLC Media Player, gEdit, Terminal). Try fitting those in 1 MB of low memory (really, even 128 MB wouldn't be enough to run all that.)
Programming in assembly has nothing to do with DOS/BIOS, and programming drivers has nothing to do with DOS. And, as already stated, if you want to tweak with your hardware, consider writing a Linux kernel module. It's not hard, and you can feel like you're in ring 0.
And one last thing: don't try to make assembly portable. Ever. It'll always be a disappointment at the end. Assembly was designed to be closely attached to one specific architecture, and be highly optimized for it (if you know what you're doing.) If you want a portable language, consider C (or Rust if you want something newer.)
EDIT: Forgot to say this. Emulators for DOS exist for one reason: let you run your old programs. FreeDOS exists to serve that same purpose; to run your old programs, not to write new programs/drivers for DOS as a way to introduce yourself to system-level programming. Personally, I learned OS development initially by using the BIOS and real mode for many months, and I think it was the biggest mistake I ever made. It took be so long to adjust to 32-bit mode, but not so long to 64-bit mode because it's mostly the same as 32-bit mode (because for both, you don't depend on the BIOS for anything.) And you're here advising newcomers to make a mistake that others have made many times. Please, I don't mean no offense, but please just stop.

Re: Real Mode (was: Some Questions about continuing with OS.

Posted: Wed Aug 24, 2016 3:07 am
by Antti
It is easy to say that wasting time on real mode programming was a mistake. Nevertheless, people saying that have very often done that by themselves and think that they wasted their time on it but still totally neglect to mention (admit) the fact that they still learnt from it and it affected their point of view. In retrospect, it is easy to say that one should have learnt only the things that end up being in the final "understanding" state.

The problem is not learning and using the real mode, in this case, but learning and using it far too long and getting stuck with it.

Re: Real Mode (was: Some Questions about continuing with OS.

Posted: Sat Aug 27, 2016 10:17 am
by ~
Antti wrote:It is easy to say that wasting time on real mode programming was a mistake. Nevertheless, people saying that have very often done that by themselves and think that they wasted their time on it but still totally neglect to mention (admit) the fact that they still learnt from it and it affected their point of view. In retrospect, it is easy to say that one should have learnt only the things that end up being in the final "understanding" state.

The problem is not learning and using the real mode, in this case, but learning and using it far too long and getting stuck with it.
If we feel better in Real Mode or if we feel constantly going back using, finding and learning from resources from the 90's and from the ISA/PS/2 PC's, then it's because that's our current level of knowledge and expertise, and we would do best to just agree with ourselves that fact, and accelerate our learning to other levels by learning the basics from the environment that actually holds all of the basics, which happens to be Real Mode/DOS/DPMI/Win9x/loadlin, whatever looks slightly like software fore bare-metal standard hardware/software/firmware.

But since the 386, Real-Mode is not 16-bit but 32-bit. We just need to set up Unreal Mode and we nicely have 16/32-bit data and instructions in a single package.

I'd really like that 16/32/64-bit Unreal Mode existed. In this way it could be possible to access the 64-bit address space from 16-bit and with no protection if we don't need that CPU software feature for our hardware project.

But since we already have 16/32-bit code, multitasking and V86 mode to span more than 1 process, and techniques to load multi-segment programs or Protected Mode (with DPMI or directly, and the EXE format instead of COM, or manually loading a specially-crafted COM file with more than 64K... very easily done) we aleady have a lot of capability.

We really don't need to write that much code to write good utilities, so such techniques are good if we feel that we still need to learn from the material found for Real Mode, which as I said, can handle 16 and 32-bit code and data directly.

The question is: Will there really be a way to intensively hack the CPU so that we can use EIP and ESP/EBP in pure Real Mode so we can use huge 16-bit programs without worrying about segment registers, leaving them loaded with 0?

Could we maybe use 32-bit prefixes like...?:

Code: Select all

a32 o32 call ???
a32 o32 ret



Then we could craft a return code area where programs would return if there are problems with using 32-bit EIP/ESP/EBP in Real Mode. From it, we could jump back to where we were.

We could get down to a manual level of execution, and implementing part of the call/return mechanism in software so that we can cover what doesn't work well about using 32-bit instruction and stack pointers in 16-bit mode. Then we could assume that we are in a sort of Unreal Mode/Protected mode without protection but with 16/32-bit code/data/stack/instruction pointer.


Will there be a possible way to enable 16/32/64-bit code/data/stack/instruction pointer from Real Mode, like Short Mode?

Probably Intel meant making possible to keep disabled all of the protection and virtualization mechanism (which are possible to implement purely in software), while being able to fully enable 16/32-bit code/data/stack/instruction pointer mode and leave full-fledged Protected Mode only for tasks that really benefit from it, like OSes for which there are already a lot of programs from different players as well as menaces, and where crashes or violations to the system (like erasure of the whole hard disk) aren't wished. If that's the case, then enabling and using 32-bit also for the stack and the instruction pointer is something for which information has become forgotten/lost, and something nobody wanted to take seriously (because of the need to design a special jump/call/return area to wrap the shortcomings to use 32-bit address space for code in 16-bit automatically, needing a small patch for each application).

I guess the next thing I will do is writing some applications or an OS without critical hardware protection in 32-bit code/data Unreal Mode. Anyway, my code will be ready to be fully ported to 16/32/64-bit or other architectures or OSes, so whatever I do will be readily usable in any environment.

Re: Real Mode (was: Some Questions about continuing with OS.

Posted: Sat Aug 27, 2016 12:00 pm
by Octocontrabass
~ wrote:If we feel better in Real Mode or if we feel constantly going back using, finding and learning from resources from the 90's and from the ISA/PS/2 PC's, then it's because that's our current level of knowledge and expertise, and we would do best to just agree with ourselves that fact, and accelerate our learning to other levels by learning the basics from the environment that actually holds all of the basics, which happens to be Real Mode/DOS/DPMI/Win9x/loadlin, whatever looks slightly like software fore bare-metal standard hardware/software/firmware, including usual and useful file formats of all kinds.
If you feel better in real mode, you haven't done very much in protected mode. The initial setup is a bit more work than real mode, but the improved flexibility is worth it.
~ wrote:The question is: Will there really be a way to intensively hack the CPU so that we can use EIP and ESP/EBP in pure Real Mode so we can use huge 16-bit programs without worrying about segment registers, leaving them loaded with 0?
Had you read the wiki page on unreal mode, you would already know that the answer to this question is yes. However, it has some severe limitations that make it useless for anything serious. It would be less work to just switch to protected mode and be done with it.
~ wrote:Could we maybe use 32-bit prefixes like...?:
I'm honestly not sure if that would work. That's pretty obviously going beyond what the CPU was ever intended to support, so you'll probably stumble across some unexpected behavior.
~ wrote:Protected mode without protection
Switching to protected mode without actually enabling any protection features is trivial. You should try it, it's actually less work than setting up unreal mode!
~ wrote:Will there be a possible way to enable 16/32/64-bit code/data/stack/instruction pointer from Real Mode, like Short Mode?
No.

Re: Real Mode (was: Some Questions about continuing with OS.

Posted: Sat Aug 27, 2016 1:11 pm
by Ycep
OK, everyone who thinks that OSes shall be coded in real mode should also use stone computer:
Image