Anyone had any real experience on 64-bit machines?

[kernel64]

Has anyone had a development kernel running on an Intel processor with EM64T or an Athlon 64?

In your experience, was it worth it? Are there annoying or severe limitations that make things especially difficult compared to 32-bit mode?

My development plan is to compile a 64-bit version of an embedded systems library, like RedHat Newlib, and get a sucky monolithic kernel running ASAP with user program support so I can learn as much as possible in the least amount of time, then from that experience properly design a kernel. I favour microkernels, although they are hard to design and get working in a reasonable time frame, so I may end up with a properly designed monolithic kernel instead like BSD or Linux. I would be happy with that. I could then work on a microkernel and port my TCP/IP networking stack, file systems, etc to the microkernel, so it's like incremental development and problems arising are likely to be in the microkernel and not in new (and not thoroughly tested) user mode support code like drivers, file systems and networking developed alongside the microkernel. Retro-fitting is not that difficult as we have seen with the system emulation that Mach is capable of in Apple Mac OS X, with its BSD usermode environment.

Are there any experienced 64-bit programmers here that can give a few words of advice on what to watch out for when writing 64-bit code? (What would warn options on compilers warn about when told to look for code that is not 64-bit clean?)

One thing I thought about was simply the size of pointers in long mode (64-bits of course) and maybe the C compiler would cause int types to be 64-bits instead of 32. (Or maybe not, because long longs are already 64-bits in most cases).

Thanks for your input.

[Solar]

Hmmmm...

We recently switched a major C++ application from 32bit Solaris to 64bit Solaris... and tell you what, all we did was a recompile.

But that was application level code, of course, and code that doesn't make explicit use of any 64bit features (obviously). OS-level code and specialized stuff (e.g. RDBMS) will feel more impact, of course.

[Candy]

Has anyone had a development kernel running on an Intel processor with EM64T or an Athlon 64?

In your experience, was it worth it? Are there annoying or severe limitations that make things especially difficult compared to 32-bit mode?

I've been mucking about with my own OS to go to 64-bit mode and as soon as I have a day off I'll start to convert it. As of now I have a working 64-bit bootloader that says "Hello 64-bit world!". That didn't give too many problems, but I've only tested it on bochs.

I favour microkernels, although they are hard to design and get working in a reasonable time frame, so I may end up with a properly designed monolithic kernel instead like BSD or Linux. I would be happy with that. I could then work on a microkernel and port my TCP/IP networking stack, file systems, etc to the microkernel, so it's like incremental development and problems arising are likely to be in the microkernel and not in new (and not thoroughly tested) user mode support code like drivers, file systems and networking developed alongside the microkernel. Retro-fitting is not that difficult as we have seen with the system emulation that Mach is capable of in Apple Mac OS X, with its BSD usermode environment.

That's quite a high goal you're setting there. Just get it running with threads and processes, without a completely messed up code base and you're ahead of most here already.

Some pointers on 64-bit stuff that's unlike the 32-bit world:

No segments. You can use fs and gs as base registers, but they function only as a third "base register" you can add to a normal calculation.

Always PAE. You always have to use PAE.

[note target=solar] the following isn't standard conforming, but the current situation on AMD64. I'm not trying to be conforming[/note]

Ints are 32-bit. Longs are 64-bit. Pointers are 64-bit, of which the top 17 bits have to be equal. Long long is also 64-bit.

Any code that casts a pointer to an int is wrong. Cast pointers to this standard type equivalent to a pointer that Solar can tell you about (don't know which myself, think it was like ptrint_t or something like that).



Just get a base system running, muck about with MSR's and make a decent abstraction layer. Then design the OS, and mostly ignore the sublying base logic.

[AR]

Any code that casts a pointer to an int is wrong. Cast pointers to this standard type equivalent to a pointer that Solar can tell you about (don't know which myself, think it was like ptrint_t or something like that).

#include <stdint.h>
uintptr_t (unsigned integer version of pointer)
intptr_t (signed integer version of pointer)

Unfortunately, stdint doesn't appear to be part of the freestanding environment.

[kernel64]

Thanks for your replies. Candy: yes, these are high goals, but I'm 25 and have the rest of my useful life to make them happen. ;D

Actually, I skipped a few bits. I'm starting with modifications to Minix and also with my own DOS-like monotasking kernel that nobody but me will ever see. Very small and simple. But it's entirely throw-away and I'm itching to move on from there as soon as I sort some things out.

I'm hesitating to start with a 64-bit kernel. I have no goals for portability personally, but if anyone does want to port my kernel when it has a mature and stable code base I want some good design and lots of C code in there to make that possible. I am looking at the pros and cons of starting with a 64-bit kernel and sticking to it, as opposed to developing on a 32-bit machine and having a 32-bit kernel as a starting point. I suppose it would be more difficult to go back to 32-bit once I've started out with 64-bit, but like I said any kind of porting to different or even slightly different architectures is not one of my personal goals. I would prefer a 64-bit system if it's not too much of a hassle.

I had a lot of conflicting information when first researching: some say the Athlon 64 and EM64Ts are not "real" 64 bit processors but Motorola/IBM G5s et al. are, and some say yes the x86-64s are "real" 64-bit CPUs. And from a very experienced and accomplished OS developer: the AMD 64 is more "real 64-bit" than the EM64Ts, because the EM64Ts can't access as much physical memory as the AMD64s and the EM64Ts need bounce buffers for PCI DMA while AMD64s don't or somesuch... I'll have to look into that further. What I'm saying is I don't want a system full of hacks because of hardware limitations. There'll always be hardware limitations and hacks, but I don't want a system full of them just to get it working at all...

[Solar]

Unfortunately, stdint doesn't appear to be part of the freestanding environment.

<stdint.h> is not required in a conforming freestanding environment. That does not mean a freestanding environment is not allowed to provide <stdint.h>.

[Brendan]

Hi,

Pointers are 64-bit, of which the top 17 bits have to be equal.

A minor correction...

Pointers/addresses are 64 bit, of which the top N bits have to be equal (where N depends on the "number of linear addressing bits" value you get from CPUID function 0x80000008 in EAX bits 8 to 15).

I assume the CPU manufacturers have done it this way so that they can add more linear addressing bits when-ever they like, and perhaps support a different number of bits for different CPUs (e.g. 52 bits for server CPUs, 48 bits for desktop, 40 bits for mobile CPUs and maybe only 32 linear addressing bits for embedded 64 bit CPUs).

It would be a shame to hard-code "the top 17 bits have to be equal" into your OS and then need to go back and change it all later...


Cheers,

Brendan

[Candy]

Hi,

Pointers are 64-bit, of which the top 17 bits have to be equal.

A minor correction...

Pointers/addresses are 64 bit, of which the top N bits have to be equal (where N depends on the "number of linear addressing bits" value you get from CPUID function 0x80000008 in EAX bits 8 to 15).

I assume the CPU manufacturers have done it this way so that they can add more linear addressing bits when-ever they like, and perhaps support a different number of bits for different CPUs (e.g. 52 bits for server CPUs, 48 bits for desktop, 40 bits for mobile CPUs and maybe only 32 linear addressing bits for embedded 64 bit CPUs).

IIRC, the virtual address space (into which pointers point) is fixed at 48 bits.

[Brendan]

Hi,

It's almost as if AMD changed their mind half way through writing the documentation. For example, (AMD64 Architecture Programmer's Manual, Volume 2: System Programming"):

From section 1.1.2 Memory Organization, Virtual Memory

Protected Mode - This mode supports 4 gigabytes of virtual-address space using 32 bit virtual addresses.

Long Mode - This mode supports 16 exabytes of virtual-address space using 64 bit virtual addresses.

And from section 1.1.3 Canonical Address From

Long mode defines 64 bits of virtual-address space, but processor implementations can support less. Although some processor implementations do not use all 64 bits of the virtual-address, they check bit 63 through the most-significant implemented bit to see if those bits are all zeros or ones.

After reading this, you'd think virtual address spaces can be as large as 64 exabytes (2[sup]64[/sup]), but some implementations may support less. IMHO this is deliberate, and not a marketting scam or an error.

When you get down to the part about paging ("section 5.3 Long-Mode Page Translation"), it appears that only 48 bits of virtual address space are possible using the PAE paging mechanism.

Then you switch over to Intel's "64 Bit Extension Technology Software Developer's Guide Volume 1 of 2" where you find similar things, like this (from Section 1.1 64-Bit Extension Technology):

In the 64-bit sub-mode of 64-bit extension technology, applications can access the following features:

• * 64-bit flat linear addressing

And then from Section 1.6.8 Address Translation:

As the first implementation of the IA-32e technology supports a maximum of 48 bits of linear address space, this paging option supports 2[sup]36[/sup] 4KByte pages spanning a linear-address pace of 2[sup]48[/sup] bytes (56 terabytes).

Both manufacturers have been very careful not to rule out any future extensions or alternatives to the "long mode PAE paging" system to support more than 48 bits of virtual addressing. This could be done with a fifth level of paging (a "PML5"), by extending the existing PML4 to a 2 MB page or by creating something entirely different.

For example, if I were a CPU manufacturer I'd be thinking of using 64 KB pages for everything. This would result in a 3 layer paging system, where there's one 64 KB page directory pointer table, and up to 65536 page directories with up to 65536 page tables each. For each virtual address, bits 0 to 15 would select the offset in the final page, bits 16 to 32 select the page table entry, bits 32 to 47 select the page directory entry and bits 48 to 63 select the page directory pointer table entry.

Anyway, IMHO an OS developer would be wise to follow the CPU manufacturers, and not to rule out any future extensions or alternatives to the existing paging system. That way when the CPU manufacturers do support more than 48 bit virtual addressing (and sooner or later they will), only the memory manager/s would need to be changed to take advantage of it.

If you start designing an OS today it's very likely that it won't be ready for actual use within ten years, so allowing for larger virtual address spaces from the start isn't as silly as it sounds.


Cheers,

Brendan

[Candy]

Is true.

However, you can't fit 64k entries in 64kbytes. You would only be able to fit 8k entries. So, you'd end up with a 4-level design again.

I think they might be looking at reverse tables or open mapping (letting the kernel control the tlb directly).

[Brendan]

Hi,

However, you can't fit 64k entries in 64kbytes. You would only be able to fit 8k entries. So, you'd end up with a 4-level design again.

Doh - no wonder it all looked so tidy! :-

y"]I think they might be looking at reverse tables or open mapping (letting the kernel control the tlb directly).

Now there's an interesting idea - make the TLB entirely explicit, rather than implicit with explicit cache flushing.

Something tells me this was done on some other architecture (can't remember which) - you'd get something like a "TLB exception" if the address wasn't in the TLB, and the kernel would need to create the TLB entry and return (or handle the exception in another way).


Cheers,

Brendan

[Candy]

However, you can't fit 64k entries in 64kbytes. You would only be able to fit 8k entries. So, you'd end up with a 4-level design again.

Doh - no wonder it all looked so tidy! :-[

When you have a very good idea that nobody seems to have realised yet, expect something to be wrong with it.

Still, 64k pages is a good idea. It would still allow 64k * 8k * 8k * 8k * 8k = 256EB which is only a factor 4 oversize. Which means that if you double the entry size (which is good for more capabilities etc.) you can make it fit exactly. 64k per page, 16 byte entries.

Wanna send an open letter to Intel and AMD?

I think they might be looking at reverse tables or open mapping (letting the kernel control the tlb directly).

Now there's an interesting idea - make the TLB entirely explicit, rather than implicit with explicit cache flushing.

Something tells me this was done on some other architecture (can't remember which) - you'd get something like a "TLB exception" if the address wasn't in the TLB, and the kernel would need to create the TLB entry and return (or handle the exception in another way).

Of course it was done before, but I tend to not like it. It gives the same recursive problems you would get with paging, or you'd have to make the kernel a monolithic page in memory or such. In any case, it won't make the design any prettier.