OSDev.org

I understand that after entering long mode, you MAY run in a "long" (e.g. 64bit enabled) code segment, but the AMD64 manual says that segments work exactly like in non-long mode for non-long descriptors. My question is, does segment limit checking still work in long mode? I understand that if you are in a 64bit segment, then all segments are flat 64-bit (theoretically) segments, but if you enable long mode and then load a 16-bit or 32-bit code segment, and then load a data segment with a 3GB segment limit, will the CPU prevent code from accessing beyond 3GB? Or will it give the 32bit code a flat 4GB limit on account of limit checking being disabled in long mode?

I already made a PC emulator, it wasn't especially good, but it works enough to boot DOS.
You can find it (together with my experimental 64bit OS) here:
http://code.google.com/p/vm64dec/downloads/list.

It only supported real-mode. Now I want to make a better emulator that uses dynamic translation, and is faster than QEMU. To do this, I will have to rewrite my disassembler/decoder called crudasm and it's a lot of work but, the idea is, I want code to run at 3GB linear but with all data segments (e.g. all segments besides CS and SS) having a limit such that accesses beyond 3GB linear are impossible

This will work for most real-mode code. I plan to use a software MMU when the guest switches to protected mode and activates paging hardware. I will only support 32bit guests initially. But the host is in 64 bit long mode. This means that if segment limits don't work in 64bit long mode, then my scheme will not work at all!

These registers will be virtualized: RFLAGS, RSP, RIP, all segment registers, RBP. All other registers will map to themselves.

I couldn't find the answer in the AMD64 manual, it is ambiguous at best. Intel64 manuals have a lot of info in them, but they are murky on the subject too as far as I can tell.

Willow
P.S. Still no luck getting my 64bit OS at the above-mentioned link to start up the Application Processors.

wrschlanger wrote:My question is, does segment limit checking still work in long mode?

Intel docs say they don't do runtime checking of the limit. And that FS & GS are treated little differently. Chapter 3.2.4, November 2008 edition.
Chap 3.4.4 quote: "The processor checks that all linear-address references are in canonical form instead of performing limit checks". Canonical probably means that highest 12 bits of linear addr are 0.

wrschlanger wrote:"but if you enable long mode and then load a 16-bit or 32-bit code segment,

You will get error after changing CS(far jump). Probably #GP.

wrschlanger wrote:and then load a data segment with a 3GB segment limit, will the CPU prevent code from accessing beyond 3GB?

I never tried. fastest way to answer the question is to try to do it yourself.

geppyfx wrote:Canonical probably means that highest 12 bits of linear addr are 0.

Actually it means they are sign extended (i.e. the same as the most significant bit that is part of the address).

You will get error after changing CS(far jump). Probably #GP.

You can load 32-bit code/data segments.

JohnnyTheDon wrote:

You will get error after changing CS(far jump). Probably #GP.

You can load 32-bit code/data segments.

Mine single core Athlon64 (one of the earliest) either reboots or hangs unless I use this code descriptor: besides this is what docs say, never go against them

L (64-bit code segment) flag
In IA-32e mode, bit 21 of the second doubleword of the segment descriptor indicates whether a code segment contains native 64-bit code. A value of 1 indicates instructions in this code segment are executed in 64-bit mode. A value of 0 indicates the instructions in this code segment are executed in compatibility mode. If L-bit is set, then D-bit must be cleared. When not in IA-32e mode or for non-code segments, bit 21 is reserved and should always be set to 0.

PS: I have never tried to deal with compatibility mode

PS: I have never tried to deal with compatibility mode

That is what I was referring to. The entire point of compatibility mode is to load 32-bit/16-bit segments.

If you read the docs completely, you will also find this passage:

In IA-32e mode of Intel 64 architecture, the effects of segmentation depend on whether the processor is running in compatibility mode or 64-bit mode. In compatibility mode, segmentation functions just as it does using legacy 16-bit or 32-bit protected mode semantics.

This means if you have any 32-bit or 16-bit segments, you can simply transition into compatibility mode and then use those segments.

An easy way for kernel code to transition into compatibility mode is using the 'sysret' instruction which is supported on all x86-64 processors. Although I have never tried it, I believe call gates work as well.

but the code have to be executed as 32bit code. No xmm8-15 or r8-r15 or even 64bit rax,rbx... . Very sad. So this is no long mode at all. You may as well stick with pre x86-64 architecture CPU.

The OP was asking what the behavior of the processor was when 32-bit code segments were loaded. In any case, compatibility mode is very useful because it allows binary compatibility with 32-bit programs.

JohnnyTheDon wrote:The OP was asking what the behavior of the processor was when 32-bit code segments were loaded. In any case, compatibility mode is very useful because it allows binary compatibility with 32-bit programs.

The OP was asking about either 16bit or 32bit so my assumption that he wanted to remain in long mode was more correct than yours about compatibility mode. For all we know he may not have known about compatibility mode.

How could he remain in 64-bit mode and use a 32-bit code segment? And he is emulating a processor, not writing software to run on one. He would need to know about compatability mode to simulate long mode completely.

@OP: The bottom line is, while in long mode all you need is a 64-bit code segment, and the base and limit of that segment is ignored. If you use compatability mode, you can run 32-bit and 16-bit protected mode code using 32-bit and 16-bit code and data segments just like you would in protected mode. You are in fact in compatibility mode after you enable long mode, and you can return to this mode at any time. So you will definitely need to emulate compatibility mode, and the Intel manuals have more information about the differences between protected mode and compatibility mode.

JohnnyTheDon wrote:How could he remain in 64-bit mode and use a 32-bit code segment? And he is emulating a processor, not writing software to run on one. He would need to know about compatability mode to simulate long mode completely.

Exactly, we don't know what he knows, he didn't explained in details and this is one of his first posts(look dates). I conclude offtopic starting with post #7 to be closed since it doesn't bring anything useful to those who will be doing forum search later.