How does paging is working ?

[dozniak]

In short:

PageTable[virtual_address] = physical_address + access bits

Whenever you try to access memory at virtual_address, processor will consult PageTable for this address and figure which physical_address if any, corresponds to this virtual_address.

physical_addresses need not be contiguous, as long as virtual_addresses are contiguous your application won't really notice.

[azblue]

Well i need to use segmentation, task segments etc... so what then ? Paging is useless here ? Do you mean by "long mode", a flat memory model ?

I wouldn't say paging is useless, but if you're using segmentation to separate your tasks you don't have to use paging. That's probably the way to go if you're having trouble understanding it.

Long mode isn't the same as the flat memory model; it's 64-bit mode. (Well, that's a slight oversimplification.)

GDT consist of 3 descriptors, CS always 08, DS always 16. And paging. And one TSS.

You mean in flat memory model, right ?

Yes, that's what he meant. If you're using segmentation you'll have a lot more entries.

[rdos]

Well i need to use segmentation, task segments etc... so what then ? Paging is useless here ? Do you mean by "long mode", a flat memory model ?

I wouldn't say paging is useless, but if you're using segmentation to separate your tasks you don't have to use paging. That's probably the way to go if you're having trouble understanding it.

Long mode isn't the same as the flat memory model; it's 64-bit mode. (Well, that's a slight oversimplification.)

GDT consist of 3 descriptors, CS always 08, DS always 16. And paging. And one TSS.

You mean in flat memory model, right ?

Yes, that's what he meant. If you're using segmentation you'll have a lot more entries.

Quite likely you'll have the full GDT (8192 entries). I use 0xF040 as the highest entry, partly so negative numbers would not be interpreted as valid selectors. In addition to that, you also have the LDT available, which can contain different descriptors per process. In a segmented design, it would typically be used by applications, or application data in kernel that cannot be shared.

And a modern segmented design will use 32-bit code, and paging as well. Objects that are allocated in large amounts would use a flat selector instead so the GDT doesn't get out of entries.

[AJ]

Hi,

modern segmented

That's an oxymoron. A segmented design will cause problems for modern compilers and will not be easily portable to long mode (where segment limits are ignored).

@OP: You would be far better to set up flat code and data segments and ignore segmentation thereafter. You then only need to worry about the GDT for TSS entries (normally one per core). Use segmentation if you have some pressing 'legacy' reason to do so or an unusual design. If it's the latter, perhaps you could let us know which part of you design requires segmentation, and we can advise on how it can be implemented better on modern hardware.

Cheers,
Adam

[rdos]

Hi,

modern segmented

That's an oxymoron. A segmented design will cause problems for modern compilers and will not be easily portable to long mode (where segment limits are ignored).

Not so. The OpenWatcom compiler can handle the 32-bit compact memory model, which uses a single 32-bit code segment with a single default data segment and multiple data segments for heap. It works very well for a design based on segment protection. As a device-driver model, optimal protection is achieved by letting each driver run in it's own code segment and default data segment.

[Brendan]

Hi,

modern segmented

That's an oxymoron. A segmented design will cause problems for modern compilers and will not be easily portable to long mode (where segment limits are ignored).

Not so. The OpenWatcom compiler can handle the 32-bit compact memory model, which uses a single 32-bit code segment with a single default data segment and multiple data segments for heap. It works very well for a design based on segment protection. As a device-driver model, optimal protection is achieved by letting each driver run in it's own code segment and default data segment.

According to the relevant wikipedia page, your "modern" OpenWatcom compiler dates all the way back 25 years to a release of Watcom C 6.0 in 1988 that was designed for DOS only.

Compilers tend to be designed for OSs. Modern compilers (designed for modern OSs) don't support segmentation; while ancient compilers like OpenWatcom that were originally designed for ancient OSs and therefore couldn't avoid supporting segmentation.

DOS used segmentation because you don't have a choice in real mode. Windows 3.1x and the original OS/2 (which was designed to run in the "16-bit protected mode" of 80286 CPUs) used segmentation because they were 16-bit and didn't have a choice either. Windows 9x used segmentation because Microsoft cared about supporting DOS applications and 16-bit Windows applications back then, not because it was "modern" (the same OS used "flat paging" for the more modern 32-bit Windows applications). All more recent OSs (OSs that didn't have to care about 16-bit, like FreeBSD, Linux, etc) never used segmentation at all.

Basically; no sane OS developer has ever used segmentation in protected mode because they thought segmentation was a good idea - it was always a case of 16-bit applications and/or compatibility.

Mostly; RDOS used segmentation (which was a mistake) because he didn't know any better, then spent the next 20 years trying to convince himself that it wasn't a mistake, and is now suggesting that other people (beginners) make the same mistake simply because he's successfully deluded himself into thinking it was a good idea. It's not a good idea. It was never a good idea. Do not listen to RDOS.


Cheers,

Brendan

[AJ]

As far as I can see (from their own Wiki here), the last release of OpenWatcom was over three years ago, it can't produce 64 bit binaries and even Linux support is considered "Experimental". Hardly the compiler of choice when what we are talking about is modern OS development on a modern compiler.

Also, just because it's possible to find a compiler that supports segmentation does not imply that segmentation is anything but legacy. Especially When that compiler's wiki proudly announces that the few supported platforms include 16 bit OS/2 and DOS.

Cheers,
Adam

[Edit: I got ninja'd by Brendan!]

[rdos]

Your knowledge is outdated. OpenWatcom supports 32-bit code generation, and it is one of few compilers that can produce 32-bit segmented code for OS/2. So your claims that OpenWatcom is a 16-bit compiler only for legacy is false.

Given that OpenWatcom could be ported to RDOS, and also could be adapted to generating 32-bit segmented device-driver code clearly also supports that it can be ported to new segmented platforms.

If I was to start on OS-development from scratch today, I'd make sure I had a dual-platform code-base which could be run both segmented (with paging) and flat. That's a recipe for success. Flat memory models in a monolithic kernel is a recipe for disaster.

[Brendan]

Hi,

If I was to start on OS-development from scratch today, I'd make sure I had a dual-platform code-base which could be run both segmented (with paging) and flat. That's a recipe for success. Flat memory models in a monolithic kernel is a recipe for disaster.

You can't actually be serious? See if you can name a single OS that currently has more than 1% market share, that isn't "monolithic with flat paging". I'll tell you now it isn't Windows, OS X or Linux. The only OS I can think of that comes close is QNX, but only because it's a micro-kernel (it doesn't use segmentation either).


Cheers,

Brendan

[rdos]

Hi,

If I was to start on OS-development from scratch today, I'd make sure I had a dual-platform code-base which could be run both segmented (with paging) and flat. That's a recipe for success. Flat memory models in a monolithic kernel is a recipe for disaster.

You can't actually be serious? See if you can name a single OS that currently has more than 1% market share, that isn't "monolithic with flat paging". I'll tell you now it isn't Windows, OS X or Linux. The only OS I can think of that comes close is QNX, but only because it's a micro-kernel (it doesn't use segmentation either).


Cheers,

Brendan

The fact that micro-kernel designs haven't been able to compete with monolithic designs points to the performance issues of micro-kernels.

And all the OSes you mention still have serious stability-issues even when they have been tested on millions of different installations. That's why such design is a recipe for disaster. You won't find the bugs even if you could test your code on millions of installations.

By having a dual code-base, which runs almost the same code segmented and flat, you could do extensive testing on the segmented environment and then remove most of the bugs without a need for testing on millions of machines.

In fact, using 64-bit long mode in a sane way (which Windows and Linux doesn't), you could achieve similar results. However, then you cannot just pack all your code and data into a bundle at almost the same address. You need to scatter code and memory allocations randomly over the full address space. Today, GCC still has serious bugs in the compiler that doesn't allow this to be done.

[bwat]

See if you can name a single OS that currently has more than 1% market share, that isn't "monolithic with flat paging". I'll tell you now it isn't Windows, OS X or Linux. The only OS I can think of that comes close is QNX, but only because it's a micro-kernel (it doesn't use segmentation either).

OSE. Used in the majority of mobile telephony network products. It was very big in terminal side as well at one point.

[bwat]

Compilers tend to be designed for OSs.

Compilers are designed for languages (as were architectures, the old saying was "language first, architecture second", but now that's different).
Compilers tend to be OS agnostic. The only real OS specific concern is how to handle newlines in input (these days mainly MSDOS or UNIX, but at one time there were more options - I think the LISP machine had a different terminator). Maybe we should add UNICODE support as well, that's OS specific. Otherwise, a compiler tends to be quite portable and the output is linked to OS specific routines.

Did you mean "Compilers tend to be designed for architectures" instead? That I could agree with in this day and age.

[bluemoon]

In fact, using 64-bit long mode in a sane way (which Windows and Linux doesn't)...

Are you serious? That two are the major platforms for long mode processors, if that are insane, nothing is.

[rdos]

In fact, using 64-bit long mode in a sane way (which Windows and Linux doesn't)...

Are you serious? That two are the major platforms for long mode processors, if that are insane, nothing is.

It is quite clear that both Windows and Linux either put their kernel image at the top 2G or the bottom 2G, because the compiler cannot handle putting some part of the code at 128G, another part at 256G. If either of those platforms used other positions for code, they would have fixed these bugs long ago. In essence, both Windows and Linux use 64-bit as if it was 32-bit, and thereby destroying any advantage of 64-bit mode.

[Combuster]

because the compiler cannot handle putting some part of the code at 128G, another part at 256G.

-mcmodel=large says you're a liar.