OSDev.org

eekee wrote:That makes sense. Seeing how binary size bloated up over a decade ago made me wonder how far the bloat had gone, and initial discussion in this thread seemed to be about code size.

Well, most of the working set in a 3D game goes into textures. Hi-res textures. They really do need them in RAM, though, so the transfer into (the far more scarce) VRAM is as fast as possible.

StudlyCaps wrote:Obviously you don't need 64-bit addresses, but you very much need a >4GB working set these days. I think 64bit addressing is simply the most efficient way to do that with todays technology.

True, we could use bank switching like in the "good old days", to continue with 32-bit addressing way beyond its use-by date. Nonetheless, 32-bit addressing with more than circa 768MB of RAM is already a bit of a problem on the OS side (because now the OS needs to switch out its mappings of RAM because its virtual space is a lot smaller than available RAM, if you include the necessity to access IO space). A problem entirely solved by 64-bit addressing for the next decade or two (by making virtual space more than four times as large as physical space, thus making it possible to not only map all RAM to kernel space, but also map a logical view of the kernel image and its own addresses on their side). And when that space inevitably runs out? At the moment I think I shall survive being only capable of handling up to 64 TB of RAM. And by the time that kind of memory actually does roll around, we will probably have 5-level paging, allowing my scheme to extend naturally another 9 address bits to 32 EB of RAM. And by the time that memory is no longer enough for serious work, I expect to no longer be doing any kind of recreational OS work.

Well, I'm not into 3D games or other bloated software, and so I don't think I need to transition to 64-bits.

nullplan wrote:Nonetheless, 32-bit addressing with more than circa 768MB of RAM is already a bit of a problem on the OS side (because now the OS needs to switch out its mappings of RAM because its virtual space is a lot smaller than available RAM, if you include the necessity to access IO space).

Huh? *headtilt* The last time I heard anything like this, the reasonable limit was 3.5GB to allow space for kernel and MMIO. Where do you get the 3/4GB figure from? I agree with the bit about switching mappings, that part makes sense, I'm just confused about the size difference.

nullplan wrote:...And by the time that memory is no longer enough for serious work, I expect to no longer be doing any kind of recreational OS work.

Quite.

nullplan wrote:At the moment I think I shall survive being only capable of handling up to 64 TB of RAM. And by the time that kind of memory actually does roll around, we will probably have 5-level paging, allowing my scheme to extend naturally another 9 address bits to 32 EB of RAM.

*confused dog head tilt* 5-level paging has been out for about a year now.

And you skipped an increment - 9 more bits gets us to 32 PB, not EB.

What I'm curious about is what comes after that. There's only 7 unallocated bits left. Is there going to be a 128-entry PML6? And what about physical addressing? 5-level paging extended MAXPHYSADDR to its maximum 52 bits.

sj95126 wrote:*confused dog head tilt* 5-level paging has been out for about a year now.

It has? I admittedly did not keep up. I thought Ice Lake server CPUs were delayed. And only those have 5-level. Whatever, the 4-level barrier is still way above my head for the time being, and likely will be for the coming decade.

eekee wrote:Huh? *headtilt* The last time I heard anything like this, the reasonable limit was 3.5GB to allow space for kernel and MMIO. Where do you get the 3/4GB figure from? I agree with the bit about switching mappings, that part makes sense, I'm just confused about the size difference.

Well, I did say "a bit of a problem". You see, when you, like Linux, reserve 1GB of address space for kernel space, and you, like Linux, map all physical memory into kernel space (or try to, anyway), and you, like Linux, try to reserve some space for I/O memory, then, like Linux, you will run into a problem when physical RAM + I/O space starts to exceed 1GB. I was guessing that at the time, about 256MB of I/O space were needed. I have no clue if that estimate was conservative or liberal. Anyway, RAM sizes with more than 1 binary digit set are rarely meaningful, so the largest below 1GB would be 768 MB. And 1GB is definitely too large, since however much I/O space is needed, it will be more than zero.

Now, like Linux, you can invent the "high memory" scheme, where you map only a small part of physical RAM (low memory) and all of I/O space into kernel space permanently, and then switch out the mappings for all remaining memory ("high memory") as needed. And, like Linux, a few decades later, when the whole thing is obsolete through the introduction of 64-bit systems, you might notice that the entire thing left your kernel an unmaintainable mess, and strip it out again. Myself, I won't even bother putting it in. These days, you are either on a high-memory system, then it will have a 64-bit CPU, or you are on a 32-bit CPU, then it will have little memory. 64-bit CPU with little memory might happen in certain applications, but 32-bit with lots of memory makes no sense.

Or you go a different route, forego virtual memory entirely (except as memory protection), identity map everything, and fail to have a problem until physical RAM goes beyond 3.5 GB. Viable option, but not compatible with any established ABIs out there, but it is what a surprising amount of OSes beyond the large consumer OSes went for. The nonstandard ABIs unfortunately require patched versions of popular compilers to support, or outright bespoke toolchains, with all the problems that brings with it. That is the reason why at work, I'm stuck with a compiler that doesn't support C95 to this day. Because an innovation from the year the PC CD-ROM drive found widespread adoption is too recent for the toolchain vendor.

Windows used to boot with the 2GB option by default, so they would have been able to use the old system of memory management for longer. But I honestly have no idea where the 3.5 GB figure appears in higher-half kernels. It makes little difference if the RAM exceeds the available virtual memory by a little or a lot. I have also no idea where the popular "36 bit" figure for PAE addressing comes from, since PAE can support physical addresses up to 52 bits. Anyway, with PAE and "high memory" support (essentially virtual bank switching), a 32-bit OS can, in theory, support up to 2^52 bytes of physical memory (too lazy to look up the correct unit right now). Of course, I have no idea where the switching costs get so high as to make that whole thing impractical.

nullplan wrote:

sj95126 wrote:*confused dog head tilt* 5-level paging has been out for about a year now.

It has?

It hasn't. Intel just announced another delay on Ice Lake server CPUs, and those will be the first ones to support it (if they're ever released).

Octocontrabass wrote:

nullplan wrote:

sj95126 wrote:*confused dog head tilt* 5-level paging has been out for about a year now.

It has?

It hasn't. Intel just announced another delay on Ice Lake server CPUs, and those will be the first ones to support it (if they're ever released).

Server, yes. The mobile (unsocketed) CPUs have been out for a while and reviewers had their hands on them a year ago.

Anyway, the point is, PML5 is out there. But you don't have to use it if you don't want to. PML4 is still supported if you don't set CR4.LA57.

nullplan wrote:Well, I did say "a bit of a problem". You see, when you, like Linux, reserve 1GB of address space for kernel space, and you, like Linux, map all physical memory into kernel space (or try to, anyway), and you, like Linux, try to reserve some space for I/O memory, then, like Linux, you will run into a problem when physical RAM + I/O space starts to exceed 1GB.

I use a lock-free physical memory allocator and I have reserved 40 MB for keeping track of physical memory in kernel space. I don't know exactly how much physical memory it supports, but it's more than 100 GB at least. I think the limit is 1.3 TB.

I reserve 1GB for kernel and give 3GB to user space.

sj95126 wrote:The mobile (unsocketed) CPUs have been out for a while and reviewers had their hands on them a year ago.

But those are Ice Lake client CPUs, which don't support 5-level paging. Only the forever-delayed Ice Lake server CPUs will support it.

sj95126 wrote:Anyway, the point is, PML5 is out there.

Got a link to a reliable source on that? I haven't seen any CPUs that claim to support it.

Outside of consumer applications, there are lots of scientific and/or enterprise programs that just *need* > 4 GiB to function properly. That is not because programmers are lazy but because distributed computing is hard and there are inherent barriers: there are conjectures that certain problems just *cannot* be broken up without duplicating work (consider the problem of doing a depth-first traversal of a graph).

Korona wrote:Outside of consumer applications, there are lots of scientific and/or enterprise programs that just *need* > 4 GiB to function properly. That is not because programmers are lazy but because distributed computing is hard and there are inherent barriers: there are conjectures that certain problems just *cannot* be broken up without duplicating work (consider the problem of doing a depth-first traversal of a graph).

The primary issue is not if a problem can be broken up but if memory accesses can be localized to smaller areas at any given time, rather than being completely random. If memory accesses are random with large data objects, there will be a huge number of page cache misses that will require the processor to look up all four paging levels for many accesses, leading to extremely poor performance. However, if memory accesses can be localized at any given time, then banking can also be successfully employed, and then you really don't need a 64-bit address space. This basically boils down to either lazy programmers (they don't bother to create algorithms that won't need a random 64-bit address space) or poor algorithms that will perform poorly regardless of hardware and processing power available.

That's simply not true. There are *inherent* barriers for some problems that prevent you from breaking them up into small pieces. For example, similar to the P vs. NP problem, it is also an open question whether P = NC (a class of "well-parallelizable" problems). Our expectation is that P != NC, i.e., some problems cannot be parallelized efficiently. ("Parallelizing well" is necessary but not sufficient for "distributing well", since if there is a distributed algorithm, I can just emulate it on a shared memory machine to obtain a parallel algorithm.)

You can take my example of depth-first graph traversal and try to come up with a good algorithm for distributed memory. If you can't, according to your claim, you (and all the other people who have looked into this) are either too lazy or this is a problem that will perform poorly in general. I don't think that either of the two cases is true.

(Yes, random memory access performs not great on shared memory x86 machines, but it performs *way* worse in distributed memory. Furthermore, there are shared memory architectures that support random access well, for example the Cray XMT, or the newer https://www.emutechnology.com/, which both migrate threads between cores instead of migrating data between caches, or the PIUMA architecture that allows asynchronous memory access; this was published two weeks ago by Intel.)

IMO 64 bit addressing is necessary now. My OS maps the complete physical memory to 0xFFFF800000000000. It would take to much address space doing that with 32 bit addressing, even with things like PAE. I wouldn't call programmers who take advantage of it lazy, as I like to work smarter, not harder . With x86_64 in particular, you get way more registers, and for me that is the biggest advantage. Back when I maintained a 32 bit version of my OS, the 32 bit version was a lot slower then the 64 bit version. Why? Because in C with 32 bit x86, parameters are passed on the stack, while on x86_64, they are passed in registers.

@nullplan, thanks for the detailed reply. I have to admit the combination of all the factors does seem complex enough to cause serious problems just because it's complex. I just don't understand all the factors, as usual. I do remember the issues of dividing memory were a cause for some complaint in the 80s, even on simpler computers than the IBM PC. Now you mention Linux "high memory", I remember the term because I was amused at Linux using this old DOS term. I once thought paging would fix all these issues, but I can in a general way understand that managing the page table could become complex enough to cause serious difficulties. The more things change, the more they stay the same! This goes for ABIs too: the choice of RAM or ROM at address zero was an issue in 8-bit computers.

Until I wrote this post, I didn't understand why any architecture would prefer address 0 to be writeable, but I think I do now. My reason for not understanding was null pointers; the most common form of pointer error as I've heard over a span of literally decades. (That's not even counting Charles Moore writing about a similar problem in 1970.) However, I'm now thinking it may have been outweighed by physical limitations on the performance of position-independent code. Position-independence requires an extra addition in many instructions, fast addition costs transistors, and transistors in CPU design could not be considered free until some point in this century. All sorts of ABIs relocate programs on load but this doesn't help when program loading needs to be fast, as it does in Unix. I personally thought the impact of this relocation on shell script performance was harmful in the 00s when I was writing a shell-script web server. (I've completely changed my views on shell scripts; I now think dividing the commands into many separate programs is a stupid idea.)

I still don't precisely know why mainstream ABIs both require program relocation and make demands on the memory map, but can understand it in terms of legacy cruft and the difficulty of controlling complexity.

In my opinion, 32-bit addressing can do an impressive amount of things. Many smartphones are 32-bit, and they can still play streams, render 3d scenes, have a UI, etc. Switching to 64-bit addressing is necessary only if you think you'll have more than 4 GiB worth of active processes (including the kernel). You can pageswap to store inactive processes that won't be woken up for a long while.