Multitasking in real mode?

[DavidCooper]

People with enough enthusiasm to get anywhere will build something.

That's true, and in time there will be better guides through the maze to deal with USB so that it's easier to see what actually needs to be done to communicate through it: EFI will have a lot more appeal once there's a beaten path to getting USB with drivers for keyboard, mouse and mass storage devices up and running, at which point the BIOS won't be missed, but we aren't there yet.

It's been 2 and half years and you're still dreaming about being "ultimately free from using the BIOS"? You should've been free from the BIOS 2 years ago.

I'm not in any great hurry to be free from the BIOS - it does all I need for now and I've saved myself many years of unnecessary slog with device drivers (in a neverending race against moving goalposts) which has all gone into something cutting edge instead. During all that time I've been able to work for many thousands of hours and to save my work at the end of each session instead of throwing it all away and having to start again from the same place every day - the BIOS actually enables me to save my work, and I love it for that.

Of course your case is special - you're writing an intelligent hex editor that's going to develop an OS by itself.

Machine Editor's an unintelligent programming system which I use to build other things, and one of those other things is the thing that brought me from linguistics into programming in the first place. I never came into this to build an OS - that's just something that was forced on me as a necessary extra to support the A.I. (which we now have to call AGI to distinguish it from the specialised A.I. in toasters which is used to help burn your toast more evenly than in older toasters).

The reality is though, Brendan, that for all the emnity between us, I still like you and admire you greatly for your depth of knowledge, perfectionism and the amount of time you spend helping people here. You're trying to steer people in the right direction for all the right reasons, and that's what matters. I suspect that a lot of the time when you come across the wrong way you're actually being misread, because you also have a great sense of humour which shows up in a lot of the colourful analogies and turns of phrase which you come up with. The most important thing is that you make people stop and think, and that will certainly stop them making a lot of the mistakes they might otherwise have made, but it's also possible that it may on occasions help them to find ways to turn what you see as a bad route into a good one by thinking up clever ways to avoid the pitfalls. I like the diversity of what everyone's doing here and am glad they aren't all writing the same OS.

[Prochamber]

@Brendan I'm really sorry but could you please stop? You are not helping and you are getting everyone off topic. I understand and respect your point but I don't want to turn this into a debate on whether or not we need BIOS. While I understand your points I do not agree with them. I will avoid inflaming the situation further by rebutting them.

Anyway from what @bluemoon has told me, I'm guessing virtual 8086 mode will not be the answer because although it archives my goal of leaving user space programs unaffected I will have to rewrite large portions of my operating system and all interrupt handlers, both BIOS and my own.
How viable is it that I copy task memory to and from extended memory when needed? i.e. copying a 64k segment before task switching

[Antti]

I've saved myself many years of unnecessary slog with device drivers

I do not think it takes years to implement a system that can do the same without using BIOS services.

[Mikemk]

@OP, what I would do is have the os find a usable segment for the program to run in, and then require that programs start something like this:

Code: Select all

push cs
pop ds

Then, when calling the program, the os does:

Code: Select all

mov ax, ? ; stack segment
mov ss, ax
mov bx, ? ; program start
xor ax, ax
call far bx:ax

[M-Saunders]

Hi,

Six years later, with over 50,000 downloads of my OS, and hundreds of emails from people grateful that I got them into OS development, I don't regret anything. So let's be fair. Real mode isn't sensible for modern OS development, but its handful of benefits can still bring lots of potential OS developers.

Are you helping these people, or providing a distraction that postpones any meaningful progress while giving them false hope?

False hope? I can't recall telling anyone that real mode is the future, or that the BIOS solves everything, or that MikeOS should be the model for all development. In fact, I've done nothing like that. Pretty much everyone who has contributed code to my project has been fully aware of the limitations of real mode. And on the odd occasion when someone has mailed me, proposing to add all sorts of modern and complex stuff to MikeOS, I always tell them that they need to go the protected mode path.

Nobody is getting any "false hope".

The shortest path between 2 points is a direct line; and doesn't begin with a pointless detour in the wrong direction.

When people are writing code for fun, as a hobby, I don't think we can ever tell them which direction they should be going in. Or that their work is pointless. If someone writes a cool little OS for, say, the Sega Genesis, you could also say it's pointless and going in the wrong direction. It's using old, out-dated hardware. But I respect projects like that -- it's not up to me, to tell people what they should be doing.

Fact is, there are still hundreds of millions of computers out there which are capable of running real mode 16-bit BIOS-happy operating systems. For hobbyist programmers who want to get something running quickly, it's usable. This forum is about "operating system development", and not specifically protected/long mode OSes, so I think we should be welcome to all projects, from experimental little real-mode projects to mighty potential Windows-killers.

Mike

[Antti]

When people are writing code for fun, as a hobby, I don't think we can ever tell them which direction they should be going in.

I strongly agree on this one. However, it only makes sense if you know what you are doing. I am sure that every member writing in this thread knows what they are doing. When it comes to beginners, I am sure that it makes sense to tell them which direction they should be going in. It does not matter whether it is a hobby OS or not. In fact, I dislike the whole "hobby" prefix. Why it could not be a "serious" project made as a hobby? Hobbyism seems to be an excuse to several shortcomings. Even if a developer wants to do a simple OS, it is not good to make fundamental restrictions that could prevent building it up to a not-so-simple OS.

I think it is not pointless (as a hobby) to do any OS that suits well to target hardware. An OS to Sega Genesis could be a good thing as a hobby.

[bluemoon]

As a side note, to make it clear to OP, and this has been stated again and again - there is a point where a set of features implement with real mode is more complex than protected mode or long mode.

For very simple hello world, real mode may be a little easier to get started, but if you add multi-tasking, GUI, design non-blocking IO, kqueue and such into your list you'll find the restriction of real mode and BIOS is limiting your work and you spend more and more efforts working around the limitations. To visualize this, imagine how much more complex to implement a web browser (even lynx) with real mode than protected mode - you may not even store full page content within a single 64K segment.

[Mikemk]

As a side note, to make it clear to OP, and this has been stated again and again - there is a point where a set of features implement with real mode is more complex than protected mode or long mode.

For very simple hello world, real mode may be a little easier to get started, but if you add multi-tasking, GUI, design non-blocking IO, kqueue and such into your list you'll find the restriction of real mode and BIOS is limiting your work and you spend more and more efforts working around the limitations.

Actually, the purpose of protected mode is that limitations not present in real mode are built in - not the other way around.

you may not even store full page content within a single 64K segment.

Can it be stored in a 4GB segment?

My point is that in real mode, you can do anything you can do in pmode + more. Want to have 4GB address space? activate a20. want a gdt and segmentation? It can (fairly easily) be emulated by software, possibly even at full speed. So why do people use pmode? It's easier, but more difficult to activate. real mode doesn't need to be activated, but more advanced stuff is harder to implement. There's your difference.

As to the question of new vs. old:
Suppose I have an old windows mobile 6 device (It's true). Would I be interested in apps for Windows phone 8, iOS 6, or android jellybean for it? no, because they won't run. I'd want apps made for it on it, even though there out dated.

[Brendan]

Hi,

@Brendan I'm really sorry but could you please stop? You are not helping and you are getting everyone off topic. I understand and respect your point but I don't want to turn this into a debate on whether or not we need BIOS. While I understand your points I do not agree with them. I will avoid inflaming the situation further by rebutting them.

It's impossible to understand my points but not agree with them. Convince me that you are not about to waste several years of your life on code that you will inevitably discard.

Anyway from what @bluemoon has told me, I'm guessing virtual 8086 mode will not be the answer because although it archives my goal of leaving user space programs unaffected I will have to rewrite large portions of my operating system and all interrupt handlers, both BIOS and my own.

There's a range of ways that virtual8086 could be used.

At one end of the range there's the "most minimal" option where virtual8086 is used in a way that almost nothing changes. All applications still run in the same virtual address space, almost all interrupts are simply reflected back to real mode, and IOPL is set to 3 so that real mode code can use "IOPL sensitive" instructions like IN/OUT, CLI/STI and HLT directly. In this case the code needed would be minimal, but the benefits would also be minimal - applications can still trash each other and trash the BIOS and trash your "kernel", and the only real benefit is that you can copy memory a little faster.

In the middle of the range would be using paging to isolate applications from each other and not much else. In this case you can change IOPL so that the BIOS can use "IOPL sensitive" instructions but applications can't, and make sure the application can't trash the BIOS or your "kernel" (by setting those areas to "read only"). You'd need to do physical memory management in protected mode and your general protection fault handler would need to be a little more complicated, and you'd need a page fault exception handler. You'd also need to intercept some BIOS functions (e.g. functions that read/write to disk) so that the BIOS actually transfers data to/from a buffer (because the addresses applications use would be virtual addresses not physical addresses). This is more work; but it gives you a lot more benefits - e.g. each application can use about 636 KiB of RAM directly (plus more that you copy into/out of the application's address space when it asks), applications are less able trash things, etc. It would also be possible to make this work for multi-CPU (e.g. so you can have 4 CPUs running 4 applications at the same time). In addition you'd be able to do things like implement a "write-through" disk sector cache to avoid a lot of disk IO (even when applications use "int 0x13" directly). The main problems are that only one task can be using the BIOS at a time (performance would still suck) and there's still many ways that applications can trash each other (e.g. using BIOS to corrupt data stored on disk, stealing keypresses from other applications, messing up other applications video, etc).

At the other end of the range is emulating the BIOS. In this case it's a full 32-bit protected mode OS, with its own drivers for everything (that allow many things to be happening at once). This gives you the highest performance possible without any changes to applications, and can prevent all the ways that applications can trash each other (and it would even work for UEFI systems). In addition, you'd be able to run real mode applications and 32-bit applications side-by-side. For this case the main disadvantages is that it'd be easier to port the applications to 32-bit and save yourself a lot of work by not bothering to emulate the BIOS.

What I'm saying here is that you can use virtual 8086 mode without rewriting large portions of your OS; but it's a compromise (more rewritten means more advantages and a better OS in the end).

How viable is it that I copy task memory to and from extended memory when needed? i.e. copying a 64k segment before task switching

That is easy; and copying about 639 KiB of RAM is equally easy (even DOS let applications use more than one 64 KiB segment). It solves none of the real problems though - applications would still be trying to "share" the same BIOS functions/devices; and can still trash the BIOS Data Area, trash the kernel, trash the file system, etc.


Cheers,

Brendan

[iansjack]

@m12 I'm not going to give a detailed response to your last post because it doesn't deserve it. I'm afraid that, IMO, it is arrant nonsense that shows a complete misunderstanding of the operating modes of x86 processors.

[Brendan]

Hi,

Six years later, with over 50,000 downloads of my OS, and hundreds of emails from people grateful that I got them into OS development, I don't regret anything. So let's be fair. Real mode isn't sensible for modern OS development, but its handful of benefits can still bring lots of potential OS developers.

Are you helping these people, or providing a distraction that postpones any meaningful progress while giving them false hope?

False hope? I can't recall telling anyone that real mode is the future, or that the BIOS solves everything, or that MikeOS should be the model for all development. In fact, I've done nothing like that. Pretty much everyone who has contributed code to my project has been fully aware of the limitations of real mode. And on the odd occasion when someone has mailed me, proposing to add all sorts of modern and complex stuff to MikeOS, I always tell them that they need to go the protected mode path.

Nobody is getting any "false hope".

Wrong.

Imagine if I left a stack of nice looking hamburgers on a silver tray in busy shopping mall with a sign on it that says "Free Hamburgers!". This would give hungry people the hope of good meal. They will come. They will eat the hamburgers. They won't find out the hamburgers are full of poison until it's too late.

MikeOS is the same. People that don't know better see MikeOS and start hoping they've found a good path to OS development. They start working with the MikeOS code (either building on top, or forking, or just borrowing ideas). They don't find out it's all completely wrong until it's too late.

Do you hate people so much that you're deliberately luring potential OS developers into taking the "real mode puke" dead end? Did a bad man lure you into taking the "real mode puke" dead end when you were a child?

The shortest path between 2 points is a direct line; and doesn't begin with a pointless detour in the wrong direction.

When people are writing code for fun, as a hobby, I don't think we can ever tell them which direction they should be going in. Or that their work is pointless. If someone writes a cool little OS for, say, the Sega Genesis, you could also say it's pointless and going in the wrong direction. It's using old, out-dated hardware. But I respect projects like that -- it's not up to me, to tell people what they should be doing.

Sounds like a nice theory; until one of your victims comes here wanting multi-tasking and wanting to keep programs separate from each other; and some poor bastard has to try and clean up the all damage you've done.


Cheers,

Brendan

[M-Saunders]

Hi,

all completely wrong

Do you hate people so much

real mode puke

one of your victims

Doesn't sound like you at all Brendan... I'm not sure what's got into you, or if someone else is posting in your name, but I used to respect and admire your posts. Now you just sound bitter and elitist. Shame.

Think what you want, but it doesn't change things. I know my OS is relatively simple, and shouldn't be used as the basis of a modern, all-singing all-dancing OS. But I've never claimed that. However, in the last six years, I've received hundreds of emails from people who've tried MikeOS, and lots of appreciation for providing people with an accessible introduction to the field of OS development. Many have gone on to write "serious" OSes in C, protected mode etc.

So I'm sorry Brendan, but those people who've expressed gratitude to me, for getting them on the road in OS development, matter far more than an angry person on a forum. I'll carry on getting people interested about this awesome topic, and if the approach I take angers you, well... That's something you need to deal with. Good luck, and I'm out of this thread.

Mike

[bluemoon]

So why do people use pmode? It's easier, but more difficult to activate. real mode doesn't need to be activated, but more advanced stuff is harder to implement.

You only need 3 instructions to activate protected mode, I can't imagine what's so difficult to activate it.
Even the whole bootstrap for higher-half, GDT, TSS initialization take less than 100 lines of assembly code, including blank lines and comments. IDT/exceptions may takes another 50 lines for assembly stubs.

So, I don't agree real mode is anything easier, but both mode are easy.

[DavidCooper]

What is the best way to implement multitasking in real mode?
...
Would using unreal mode or segments or copying code between extended memory help? I'm read the OSDev Wiki but it mostly covers protected mode and doesn't have many examples, so I'm a little confused.
...
How viable is it that I copy task memory to and from extended memory when needed? i.e. copying a 64k segment before task switching

I don't like idea of copying lots of stuff up and down memory at every task switch, but it may turn out to be the best option. What you need to do is work out how much stuff would actually need to be moved every second, then work out how long it would take to move it and see if the percentage of a second taken up by that is acceptable. The more often you switch task, the bigger that percentage will be, so you need to do some sums and experiments to work out the limits of what is possible. It may be viable and it may even be the best solution, but only the numbers will be able to tell you that.

I don't know much about unreal mode, but would I be right in thinking that it allows you to have program code in the bottom 640KB or memory and all the data they work with at almost any address up to the 3GB point? If so, then active programs and threads could sit in low memory while temporarily inactive ones could be moved out of there to stop them taking up space, and data would never be in low memory at all (unless it needs to go through a buffer there on loading and saving). If applications are bulky though, that isn't so easy to do, but even so, most of the time you'll just be using one application while the rest aren't doing a lot, so you could maybe organise things in such a way that the user can choose which ones keep running all the time in the background and which get put into temporary storage (coming back to life as soon as the user switches back into them).

If that is viable, the main problem would be that existing programs can't cope with the data being outside the bottom 640KB of memory, so when those are running they're going to put severe limits on how much code can multitask properly, though again that isn't necessarily important: as I write this I have a number of different applications open, but only one of them is doing any significant work at the moment and it's the one I'm typing into, so it wouldn't matter if the rest were in temporary storage as I wouldn't be able to tell the difference. It would only matter if I had music playing in an app in the background or if some app was doing a long calculation and had to work hard for an hour without having anything interesting to show me along the way, though even then it's probably rare for most users to have as many as three programs genuinely needing to do a lot of work at the same time, though there will often be lots of little threads active at once doing small stuff like supporting the GUI.

[*] I want to keep programs separate from each other.

If you mean for security purposes, that's going to be tough, unless you run all code indirectly through an emulator/monitor, relying on the speed of modern processors to hide how much slower the code is running, though you could always restrict its use to untrusted apps, and that would also allow you to have the code of those apps sitting in extended memory all the time. That could still give you a performance cost which may be worse than copying everything up and down memory repeatedly, but it could provide complete security.

Maybe you'll need to do all the above and more before you can tell what the best way of doing things is, but I'd start by getting out a calculator and doing some experiments as it could save you a lot of time if it rules out an option at the start. Writing an emulator and running everything through it would cost at least 90% of processor time, but it would get the job done. Copying everything up and down memory hundreds of times a second may cost you 1% or 100% or processor time depending on how much stuff needs moving and how fast the machine that it's running on is, so until you know how much there is to move and how long it will take to move it, you're left having to guess which is the best way forward. My guess is that moving stuff up and down memory hundreds of times a second may be fully viable on a modern machine, but that's just a guess, and you don't want to proceed on the basis of guesses.

Brendan's right about real mode OSes being something that should ordinarily be avoided, but if your driving aim is to push real/unreal mode to its limits to find out what it can do and to demonstrate that to the world, I don't think that's an entirely pointless task. By doing this kind of work you are also gaining transferable skills - everything we do is about problem solving and learning how to solve problems better and more quickly. I would say that what you are doing is in many ways an artistic adventure which is quite different from trying to build a cutting edge OS. You know what you're trying to build and why you want to build it, and so long as you understand the limitations that you've imposed on yourself, what you're doing is perfectly valid.

[Prochamber]

It's impossible to understand my points but not agree with them.

Wow. How arrogant are you? I see you chose to ignore my request to stop ranting about BIOS.

...and to everyone else:
From what's been written I see that I would need to copy a lot more data than originally planned to prevent the kernel or BIOS getting messed up and added a lot of latency. Clearly my idea of Big Unreal Mode and memory copy was not the best.

Here's another: Huge Unreal Mode.
As I understand Huge Unreal Mode allows you to extend the code segment into 32-bit address space.
If I copy the kernel segment and BIOS segment to another part of the memory and just change the segments when the task needs to execute it would mean that each task will be running a separate copy of the kernel and BIOS so they won't get corrupted. I would just need to change the segment before changing tasks. It is impossible for disk corruption to occur in this way because a lower IRQ (the PIT) cannot interrupt a higher one (hard drive/floppy disk). Even if this is not the case I could just instruct the disk API to disable multitasking before beginning any operations
I understand that this will cause my EIP to mess up when calling interrupt but I could just write wrappers for my interrupted to call the original routines and preserve the return address.

What do you think of my idea? Is it plausible? Is it even possible to change extended segment registers in unreal mode?