Your OS design

[Tyler]

Because 64-bit is "new", there's much less backward compatability mess to worry about. 64-bit chips are more standardized now than 32-bit CPUs have been for years, and 64-bit chips will probably become less standardized over time (not more standardized) as different manufacturers add their own new features, like VMX (Intel's virtualization) or SVM (AMD's virtualization).

Haven't they simply build 64-bit onto the same architecture? Given that all software and hardware would have to be changed anyway the obvious thing todo would be to start a new platform for scratch, based on the same ideas but standardized. Yet as far as i know, your operating system has to deal with all the same backward compatabilities as now plus having to check for a 64-bit system.

The backward compatability does not come from Intel, but IBM not looking past each version of there PC. It is there PC compatable architecture that creates the many dependencies, and we will continue to use that even when all systems are 64-bit.

[Candy]

Because 64-bit is "new", there's much less backward compatability mess to worry about. 64-bit chips are more standardized now than 32-bit CPUs have been for years, and 64-bit chips will probably become less standardized over time (not more standardized) as different manufacturers add their own new features, like VMX (Intel's virtualization) or SVM (AMD's virtualization).

Haven't they simply build 64-bit onto the same architecture? Given that all software and hardware would have to be changed anyway the obvious thing todo would be to start a new platform for scratch, based on the same ideas but standardized. Yet as far as i know, your operating system has to deal with all the same backward compatabilities as now plus having to check for a 64-bit system.

The backward compatability does not come from Intel, but IBM not looking past each version of there PC. It is there PC compatable architecture that creates the many dependencies, and we will continue to use that even when all systems are 64-bit.

If you have a 64-bit system, you very likely:
- Don't have a 386
- Won't have ISA slots - no ISA PnP
- Will have a PCI bus
- Will have PCI-e and/or AGP
- Will not have an A20 gate
- Will have a video card with enough memory for any arbitrary framebuffer
- Will have a network card
- Will not have a token-ring bus card
- Will have at least 64M of ram

That's a whole lot of corner conditions you can skip. There are also architectural improvements:
- Each interrupt table entry can have its own stack
- You don't have to stack-switch to a default kernel stack on each kernel entry
- You always can use SYSCALL for 64-bit programs
- You can always use CPUID
- Your processor tells you about its caches, addressing size and so on.

That allows you to skip so many design choices that are only relevant for <64bit systems.

[bewing]

Candy, most of those simplifications you mentioned are also true for P6 family 32bit systems. Especially if you just refuse to support really old legacy equipment (like ISA) with your drivers.

Because 64-bit is "new", there's much less backward compatability mess to worry about.

Well, I think I disagree. The next big compatibility issue is going to be EFI -- and both of us are going to have to completely recode our boot processes to handle it (if it actually becomes a standard), so I think we're in exactly the same boat.

The thing with 64bit CPUs is: if Intel locks itself into a stupid design, and someone else makes a significantly better mousetrap -- then the next generation of 64bit platforms may not even be "Intel compatible". We are at the beginning of an entirely new chipwar here, and I want to see how it falls out before I invest in a lot of code. 32bit CPUs are fast enough for me, at 2 or 3GHz, if I write fast, tight code.

BTW, how long is it going to take to write your OS and what sort of computers will be around when it's finished?

LOL -- I have a fair amount of free time, and I think I'm halfway done. The OS is booting to a prompt, and all my system buffers are properly initialized. Or perhaps I'm 90% finished, according to that Michael Abrash quote that Combuster posts. I've been putting in a lot of hours, trying to get it running soon, rather than just dinking around with "interesting" features. I think I could have a mini-IDE running on a GUI in 2 or 3 months. I'm building the assembler for it now.
I DO admit that it's already taken me longer to get where I am than I was hoping.

I'm thinking that in 10 years time "many-CPU" NUMA machines will be common, and 32-bit CPUs will be obsolete (except for embedded systems).

I'm guessing more like 15 years -- the Pentium isn't quite "obsolete" yet, as a design structure. In general, I agree, of course. But if I have a decent OS on it (not M$ crap), I'd be happy to still be running my 2Gz P4 machine in a decade, I think.

What happens when you've got a 30 MB file and an application appends 2 KB to the end of it? Will you store the extra 2 KB somewhere else on disk (fragment the file), or relocate the entire file somewhere else so that you can add that extra 2 KB to the end without overwriting other data and without fragmenting the file? How much time would it cost to relocate (read and write) 30 MB of data, and how much time would it cost if the file was fragmented?

An "old" file that is being actively modified is allocated new clusters from the 8K cluster pool. When the file's "aging" flag indicates that it's not active anymore, the file system manager has a low-priority daemon that rewrites files into a "properly" sized cluster sequence (in the middle of the night ).

File fragmentation is bad, but is "no file fragmentation" worse (considering that it's a general file system, not something primarily designed for "write once")?

Look at any standard general file system. You must agree that 99.9% of the files on that filesystem ARE write-once! Out of 40 thousand files, a few hundred at most are written on any day. And most of those are write-once temp files. But fragmentation is not my primary target anyway -- it's the damned empty space at the ends of all the thousands of tiny files that are less than one cluster long. Preventing fragmentation at the same time is a nice freebie for me.

How will you prevent ring 1 code from trampling on the kernel's memory area?

GDT level. Ring 1 does not use paging (only Ring 3). Ring 1's gdt entries only allow read access to any memory outside the current running app's data area, and the Ring 1 shared memory areas. There's only one excuse for those stinking segment registers, and that's to restrict memory accesses.

If you do prevent ring 1 code from trampling on the kernel's memory area, does that mean that ring 1 code will trample on other ring 1 code instead?

Hopefully not, but at least that won't BSOD the machine. I think it's mostly preventable.

[Brendan]

Hi,

Because 64-bit is "new", there's much less backward compatability mess to worry about.

Well, I think I disagree. The next big compatibility issue is going to be EFI -- and both of us are going to have to completely recode our boot processes to handle it (if it actually becomes a standard), so I think we're in exactly the same boat.

32-bit 80x86 Apple computers use EFI already.

The thing with 64bit CPUs is: if Intel locks itself into a stupid design, and someone else makes a significantly better mousetrap -- then the next generation of 64bit platforms may not even be "Intel compatible". We are at the beginning of an entirely new chipwar here, and I want to see how it falls out before I invest in a lot of code. 32bit CPUs are fast enough for me, at 2 or 3GHz, if I write fast, tight code.

It doesn't work like that - no-one is allowed to make an 80x86 CPU without signing a cross-licencing deal with Intel. This means anything any manufacturer does is cross-licenced to Intel, and Intel can choose to do the same as other manufacturers (or not). Besides this there's a lot of market pressure to be compatible with everyone else, especially for application's code (and even more pressure to be compatable with Windows, for better or worse).

The main problem isn't competition between manufacturers, but the addition of new features. Take paging in 32-bit CPUs for an example - 80386 had "plain paging", 80486 introduced global pages and INVLPG, Pentium introduced PSE, Pentium Pro introduced PAE and Pentium III introduced PAT page-level cache controls. Almost something new with each new CPU type, and all of this is from Intel alone. With paging in long mode you only really need to care about whether or not NX (no-execute) is supported, but that's only a simple masking operation (no-where near as messy as entirely different paging structures for e.g.).

BTW, how long is it going to take to write your OS and what sort of computers will be around when it's finished?

LOL -- I have a fair amount of free time, and I think I'm halfway done. The OS is booting to a prompt, and all my system buffers are properly initialized. Or perhaps I'm 90% finished, according to that Michael Abrash quote that Combuster posts. I've been putting in a lot of hours, trying to get it running soon, rather than just dinking around with "interesting" features. I think I could have a mini-IDE running on a GUI in 2 or 3 months. I'm building the assembler for it now.
I DO admit that it's already taken me longer to get where I am than I was hoping.

I thought I was well past half-way done back in 1998 - I had a CLI, a GUI, good/stable memory management, scheduler, keyboard, generic video, serial, floppy, othello, etc. In the last 7 years I've progressed a lot - I learnt how crappy my OS was, did "about half" of a much better OS and now I'm about 1% through the latest rewrite. Michael Abrash is an optimist (unless you like unmarketable code).

I'm thinking that in 10 years time "many-CPU" NUMA machines will be common, and 32-bit CPUs will be obsolete (except for embedded systems).

I'm guessing more like 15 years -- the Pentium isn't quite "obsolete" yet, as a design structure. In general, I agree, of course. But if I have a decent OS on it (not M$ crap), I'd be happy to still be running my 2Gz P4 machine in a decade, I think.

AMD's "multi-chip" machines are already NUMA. Intel's "multi-chip" machines will be NUMA in the next few years (see Intel's CSI). The only real question is how quickly "multi-chip" machines become common, which depends on how many cores they can put on a chip before it overheats or before there isn't enough bandwidth to get data to/from the cores (and if you care about servers or not, as "multi-chip" is already common for servers).

Of course 64-bit is here now - I doubt anyone buying a new 80x86 desktop/server will get a 32-bit CPU. In 10 years time how many people will be using a 10 year old computer? How many people actually use 10 year old computers now? Plenty of people own them (like me) but I doubt there's many people who actually use them for desktop/server use - maybe one or 2 being used as a gateway or router running something like SmoothWall (i.e. very similar to embedded systems).

What happens when you've got a 30 MB file and an application appends 2 KB to the end of it? Will you store the extra 2 KB somewhere else on disk (fragment the file), or relocate the entire file somewhere else so that you can add that extra 2 KB to the end without overwriting other data and without fragmenting the file? How much time would it cost to relocate (read and write) 30 MB of data, and how much time would it cost if the file was fragmented?

An "old" file that is being actively modified is allocated new clusters from the 8K cluster pool. When the file's "aging" flag indicates that it's not active anymore, the file system manager has a low-priority daemon that rewrites files into a "properly" sized cluster sequence (in the middle of the night ).

When an "old" file that is being actively modified is allocated new clusters from the 8K cluster pool, is the entire file copied into new unfragmented space from the 8k cluster pool, or is the file fragmented?

Imagine you've got 3 files on disk (A, B and C) and the disk sectors look like this:

AAAABBBBCCCC--------

If you want to append data on the end of file B, do you do this:

AAAA----CCCCBBBBB---

Or do you do this:

AAAABBBBCCCCB-------

The first option means that you've got free space fragmentation and (from an application's perspective) appending a single byte could cause unacceptable delays as the entire file is copied elsewhere. The second option means that you've got file fragmentation.

Defragging during idle time is a good idea, but it's not the same as "fragmentation is impossible", and (for a reliable and versatile OS) isn't necessarily adequate on it's own because you can't guarantee that there will be enough idle time. For example, I use one of the computers here (mostly) as a games machine - it's either 100% busy or it's turned off (or it's booting or shutting down) - there is no idle time.

For servers you can have similar problems (but worse). For example, consider a server being used for NNTP and SMPT (newsgroups and email) where a large number of files are being modified. Are you going to expect adminstrators to take NNTP and SMPT offline for a few hours each day to give the OS enough idle time to defrag?

How will you prevent ring 1 code from trampling on the kernel's memory area?

GDT level. Ring 1 does not use paging (only Ring 3). Ring 1's gdt entries only allow read access to any memory outside the current running app's data area, and the Ring 1 shared memory areas. There's only one excuse for those stinking segment registers, and that's to restrict memory accesses.

If you do prevent ring 1 code from trampling on the kernel's memory area, does that mean that ring 1 code will trample on other ring 1 code instead?

Hopefully not, but at least that won't BSOD the machine. I think it's mostly preventable.

This is better than no protection, but imagine you've got a sound card driver that occasionally trashes the disk driver's shared memory, and your file systems and/or swap space are occasionally being corrupted.


Cheers,

Brendan

[Dex]

What will the OS of the future be like and how can we OS Dev's make lots of $$$$$$$$$.
Here is Dex's guild to making big money from What will be the OS of the future.
Now i know you will all be saying, i not in it to make money, but deep down, we all dream of M$ buy outs, to get there hands on something we have come up with .

1. So first thing you need to know is the OS of the future will have no value .
So you can all forget all your code that trys to copy window or linux line of thought.
Now if the OS of the future, has no value, how can i make any money.

2. Thats simple, the OS is valueless, but what will be of great value is services.
I see the future OS being lots of device talking to each other by things like wifi and
over the net etc.
Alot of these device maybe running linux or some other embedded OS, but there will
be little value there.

3. Here are some example of today services "Google" that makes lots of money, also take games these use to be played on consol and come on disks, but more and more people are turning to on line games like "SecondLife"
Then theres "youtube" "myspace" "photo storage" etc.

4. Now this is just the beginning, anything that you do now on you PC and use software for, could and will be available on line, it could be free and make money from advertising, or subsction based etc.

5. So where does this leave us OS dev's and why will OS like windows loose value,
because the thing that keeps people using them, will run on any OS eg: linux etc.
So if you want your OS to be used in the future, you need to tale your OS to a newwork OS.
Let take loading a file off a hdd for a example, Now we have drive and need away to read/write to the File sys. But i think its would be better to use FTP and let a FTP server do the gritty work.

6. Now you may ask what do i know, well here's is some link that may help:
http://www.kottke.org/05/08/googleos-webos
http://www.boston.com/business/technolo ... re/?page=1
http://ibtimes.com/articles/20070228/mi ... search.htm

7. Just think of it like this, we are all working on OS, which other than help with answer to ?, does not help each other much. But if there were network OS our client/servers could be used by each other, does not mater if code in C or asm or pascal .
Rant over .

[Tyler]

3. Here are some example of today services "Google" that makes lots of money, also take games these use to be played on consol and come on disks, but more and more people are turning to on line games like "SecondLife"
Then theres "youtube" "myspace" "photo storage" etc.

If you ever compare the future of operating systems with the temporary craze of retards using up internet bandwidth... i may just come round to your home and have to slap you back into reality...

Whatever the future holds for operating system, i don't think (pray to god) it will ever become something other than a way of managing a single computer's resources. The "Internet Operating System" concept really needs a new name until the idea gets old and we can all forget the decade of the marketed internet. It is not an operating system, simply online services, and all dex is describing is a standardised communication, so he is definetly dreaming.

[Dex]

If you ever compare the future of operating systems with the temporary craze of retards using up internet bandwidth... i may just come round to your home and have to slap you back into reality...

When you wake up in Hospital after 2 years in a coma, you will see i am right

You can take a horse to water but you can't make him drink

[Tyler]

When you wake up in Hospital after 2 years in a coma, you will see i am right

You can take a horse to water but you can't make him drink

Ok deal, if i somehow end up in hospital then i will believe you.

Also, it is not about making them drink the water, it is the fact the horse is beign told the water is magical and does many things it doesn't.

[Candy]

Could you guys again stop egging each other on?

People have visions, let them. Dex believes desktop OSes will become extinct, let him. There are many people who believe the same; I for instance do not. I can understand why it would work and I do say it can work. I do also realise that it more strongly counteracts a few of my instincts, which I expect in others. What about some arguments from you instead of insults and threats?

[Tyler]

Could you guys again stop egging each other on?

People have visions, let them. Dex believes desktop OSes will become extinct, let him. There are many people who believe the same; I for instance do not. I can understand why it would work and I do say it can work. I do also realise that it more strongly counteracts a few of my instincts, which I expect in others. What about some arguments from you instead of insults and threats?

I thought it was more joking, but my apolgies. I think i made my argument quite clearly, the current run of "Web 2.0" (try not to throw up people) is a media farce that will wear out in no time. The concept of an operating system is to manage system resources, it will always be this unless somehow electronics evolve into a totally virtual realm... cause thats likely.

There may well be a standard for generic communication between all devices though. My operating has in place the ability to work as a minimal Resource Kernel that communicates with other systems it attaches to as a whole. This form of distibution is what i hope will happen when all devices are built with embeded systems and people are able to communicate with them all from there Desktop computer.

The idea of an internet operating system is just saying, the internet is goign to get better. I do not think that has to be said. Websites are constantly gaining new abilities and i am sure one day Googles online Wordeditor and Spreadsheet will be far bigger. This is no an operating system though, simply online services. An operating systme continues to be the software which manages a single computers resources.

[Brynet-Inc]

I myself hate the "Web Operating System" ideas.. (And the Web 2.0 nonsense..)

I'd personally slap everyone who uses an excessive amount of Javascript(AJAX junk..) if it was humanly possible..

I'd also ban proprietary junk such as "Adobe Flash" if it was possible..

Follow a w3c standard.. It's not that hard people!

A "Web Operating System" is not an Operating System, It's simply waisted time and bandwidth.

Does any agree with me on this? or am I alone with my views once again?

[pcmattman]

A 'web operating system' is useless! Think this one through, people!

What do you need to run a web server? AN OPERATING SYSTEM!

What is the point of a web operating system if users need another operating system to run the operating system? Either way, you can't call a web operating system an operating system because it depends on another operating system being installed.

( too many times have I said operating system in the above paragraph)

[Brynet-Inc]

A 'web operating system' is useless! Think this one through, people!

What do you need to run a web server? AN OPERATING SYSTEM!

What is the point of a web operating system if users need another operating system to run the operating system? Either way, you can't call a web operating system an operating system because it depends on another operating system being installed.

( too many times have I said operating system in the above paragraph)

Finally.. someone else understands why such things are a pointless waist of time.

[pcmattman]

Finally, someone else who agrees that Flash ruins websites, JavaScript shouldn't be used excessively and web operating systems are a pointless waste of time...

What's really fun is saying my last post (before Brynet-Inc's post) out loud have fun!

[Alboin]

Finally, someone else who agrees that Flash ruins websites, JavaScript shouldn't be used excessively and web operating systems are a pointless waste of time...!

I agree as well. Although, I think the Flash and JavaScript problems are due to a lack of choices. If we weren't trying to use languages that weren't meant for creating dynamic websites, then there wouldn't problems. What you need is a method of combining PHP, JavaScript, and multimedia.

Moreover, web browsers aren't really the best dynamic tool either. They were meant for displaying text. I don't see why someone doesn't just creating a new method of web application development, by creating a byte code interpreted web browser(Not the browser itself, but the sites.), where you website is downloaded in a single package, and run from there. (That is, it is the site is compiled to bytecode and run.) Therefore, the site could be divided into modules; each downloaded if needed and run. Now, there would be some issues with talking back to the server, but they could be worked out.

Just a thought....Maybe I should implement it....