OSDev.org

Posted: **Wed Dec 23, 2020 12:16 pm**

I finally finished my bootloader for the FAT16 fs and it works perfectly... on QEMU. For some reason, it boots perfectly with QEMU but when I burn the image to a USB with rufus and try to boot it with a real machine it won't work. I achieved to track the error to this line:

Code: Select all

mov dx, word [dword eax + FAT_OFFSET]

What this does is loading a specific index of the fat table (EAX) in the DX register. It is a strange error since this line is included in a function that is called multiple times. It throws the error (hangs) only when the value of EAX is 8, or at least this is what I discovered with my testing. In my code EAX increases by 2 every time the function is called. Anyway, I tried to load that address manually with:

Code: Select all

mov dx, word [dword FAT_OFFSET + 8]

But it works perfectly in real hardware without hanging.
I was told that in some machines 32 bits addresses cannot be accessed unless unreal mode is enabled but since sometimes it works I don't think this is the case.
Obviously, only that one line of code cannot underline any error so I'm posting also the function that includes that line. To give you some more context I'm also linking the GitHub repo folder that contains this code. It is in boot/include/fat16.inc and the functions of this file produces this error when they are called from boot/include/fat16_ext.inc which contains some functions called in boot/loader.asm around line 213.
I spent a few days trying to track this error but this is all I could find. I hope is an easy solution.

Code: Select all

;*********************************************;
; Read the current selected cluster in memory ;
; Parameters:                                 ;
;   ebx => Buffer to load file to             ;
;*********************************************;
LoadNextCluster:
    ; zero out registers for calculations
    xor ecx, ecx
    xor dx, dx

    ; convert the cluster in lba
    mov ax, word [cluster]
    sub ax, 2
    mov dl, byte [bpb_SectorsPerCluster]
    mul dx
    xchg cx, ax
    add ecx, dword [first_cluster_sector]
    
    ; sets the others parameters and read the disk
    mov dl, byte [bpb_DriveNumber]
    mov al, byte [bpb_SectorsPerCluster]
    call ReadSectorsLBA

    ; get next cluster from fat table
    xor eax, eax
    mov ax, word [cluster]
    mov dx, 2
    mul dx ; since fat table is an array of words (2 byte)
    mov dx, word [dword eax + FAT_OFFSET]

    mov word [cluster], dx

    ret

Posted: **Wed Dec 23, 2020 1:24 pm**

What is the value of FAT_OFFSET?

Posted: **Wed Dec 23, 2020 2:04 pm**

Octocontrabass wrote:What is the value of FAT_OFFSET?

FAT_OFFSET = 0x00020000

Posted: **Wed Dec 23, 2020 2:46 pm**

That offset doesn't fit in 16 bits, so you must either extend the segment limit (using unreal mode, protected mode, or long mode) to access it, or choose a different segment:offset combination.

BIOSes sometimes use unreal mode, so you may see it work even when it normally shouldn't. BIOSes may switch between ordinary real mode and unreal mode at any time without warning.

QEMU doesn't emulate segment limits correctly. You should see similar issues in a more accurate emulator.

Posted: **Wed Dec 23, 2020 2:49 pm**

All segment limits are set to 0xffff initially, so that won't work.

Posted: **Wed Dec 23, 2020 3:03 pm**

Octocontrabass wrote: BIOSes may switch between ordinary real mode and unreal mode at any time without warning.

Oh, I didn't know that. So I think what I need to do is to enable unreal mode myself before accessing this memory.

Octocontrabass wrote: QEMU doesn't emulate segment limits correctly. You should see similar issues in a more accurate emulator.

Do you have any suggestions on which one to use? Maybe BOCHS? I plan to debug the C kernel with gdb so I'd like to switch to an emulator that also supports it.

Gigasoft wrote:All segment limits are set to 0xffff initially, so that won't work.

Enabling unreal mode I'm gonna set up a GDT witch should take care of this.

I'll let you know how it goes after testing with unreal mode. Thanks.

Posted: **Wed Dec 23, 2020 3:46 pm**

Load a segment register with 2000h and skip the + FAT_OFFSET part as 32-bit addressing doesn't work in real mode.

Posted: **Wed Dec 23, 2020 3:51 pm**

Bonfra wrote:So I think what I need to do is to enable unreal mode myself before accessing this memory.

Or use a different segment:offset pair. For example, set a segment register to 0x2000 and then you can access everything from 0x20000 to 0x2FFFF using a 16-bit offset.

Bonfra wrote:Do you have any suggestions on which one to use? Maybe BOCHS? I plan to debug the C kernel with gdb so I'd like to switch to an emulator that also supports it.

Do you need it? Most operating systems don't use segment limits once they switch to protected mode. (They set the segment limits to 4GB, which is the same as having no limit.)

Posted: **Wed Dec 23, 2020 3:57 pm**

@rdos: 32-bit addressing works in real mode *IF* you know you are running on an 386+. It is just that you have to be careful when the effective address exceeds the limit for the segment it is accessing. As others have said using unreal mode changes the limits to 4GiB so that 4GiB address space can be addressed even in real mode.

Posted: **Wed Dec 23, 2020 3:59 pm**

QEMU (without KVM) doesn't check segment limits for purposes of performance. Enable KVM with -enable-kvm option and that should be picked up. BOCHS will throw an error on the console when you exceed segment limits.

Posted: **Wed Dec 23, 2020 4:06 pm**

MichaelPetch wrote:@rdos: 32-bit addressing works in real mode *IF* you know you are running on an 386+. It is just that you have to be careful when the offset exceeds the limit for the segment it is accessing. As others have said using unreal mode changes the limits to 4GiB so that 4GiB address space can be addressed even in real mode.

No it doesn't. The segment limits are set to 0xFFFF when the processor boots into real mode, and so any offsets above 0xFFFF should cause a fault. If you switch to protected mode, set the limits to 4GB and return to real mode without setting them back to 0xFFFF, then large offsets will work. However, this is a hack.

Still, in this case the obvious solution is to load es with 0x2000, use an es: override and skip the FAT_OFFSET part.

Posted: **Wed Dec 23, 2020 4:10 pm**

MichaelPetch wrote:QEMU (without KVM) doesn't check segment limits for purposes of performance. Enable KVM with -enable-kvm option and that should be picked up. BOCHS will throw an error on the console when you exceed segment limits.

Then QEMU simply is a faulty x86 emulator. As long as it is not emulating long mode, segment bases & limits must be evaluated to create correct emulations. I guess it is not so strange that no emulator can run RDOS as most have bugs in their segmentation logic.

Posted: **Wed Dec 23, 2020 4:14 pm**

Octocontrabass wrote:Or use a different segment:offset pair. For example, set a segment register to 0x2000 and then you can access everything from 0x20000 to 0x2FFFF using a 16-bit offset.

This is exactly what I did and it works beautifully:

Code: Select all

push es
push word 0x2000
pop es
mov dx, word [dword es:eax]
pop es

MichaelPetch wrote:Enable KVM with -enable-kvm option and that should be picked up.

Awesome! I'll activate this feature. I don't want to be surprised by these things happening anymore.

Thanks to everyone.

Posted: **Wed Dec 23, 2020 5:11 pm**

rdos wrote:
MichaelPetch wrote:@rdos: 32-bit addressing works in real mode *IF* you know you are running on an 386+. It is just that you have to be careful when the offset exceeds the limit for the segment it is accessing. As others have said using unreal mode changes the limits to 4GiB so that 4GiB address space can be addressed even in real mode.
No it doesn't. The segment limits are set to 0xFFFF when the processor boots into real mode, and so any offsets above 0xFFFF should cause a fault. If you switch to protected mode, set the limits to 4GB and return to real mode without setting them back to 0xFFFF, then large offsets will work. However, this is a hack.

Still, in this case the obvious solution is to load es with 0x2000, use an es: override and skip the FAT_OFFSET part.

32-bit Addressing and addressing an effective address beyond the segment limits are not the same thing. 32-bit addressing is legitimate and allowed by the 386+ processors in real mode and was designed that way. Did you miss the part where I said "It is just that you have to be careful when the offset exceeds the limit for the segment it is accessing". If you use 32-bit addressing and exceed the LIMIT of the segment being referenced then THAT IS A PROBLEM. But if you use 32-bit addressing and the effective address remains within the limits then that is perfectly acceptable. If you have a memory address [123+eax+esi*4] and the computed effective address falls within the segment limits there is nothing wrong with that even in real mode.

Unreal mode (back in the 80s it wasn't called that of course) has long since gone from hack to mainstream. Originally Intel didn't document this behavior (switching back to real mode was considered undefined and for some batches of 386s switching back to real mode wasn't supported at all). Then Microsoft started embracing unreal mode with things like HIMEM.SYS in the 1980s and some companies pushed back on Intel's view of switching back to real mode. Eventually Intel fully embraced switching back to real mode and it finally documented the behaviour publicly by the end of the 80s. For the last 30 years the behaviour of the descriptor caches has been mainstream and public knowledge and Intel now defines this behaviour in its documentation. If you want to consider using publicly documented behaviour of the 386 a hack so be it. I consider it a useful tool at times.

As others have correctly pointed out, on real hardware *some* BIOSes put the processor in unreal mode to do work before reaching the bootloader and don't put the segment limits back to 0xffff. The result is that on some systems the limits are 4GiB by the time the boot sector is loaded and starts executing. Yes, even BIOSes often rely on unreal mode to do work. You can't rely on this behaviour but is often one reason why some hardware operates differently than others if you are exceeding the limit of 0xffff. If you want to access beyond the 0xffff limit and wish to remain in real mode then explicitly entering unreal mode is a viable option.

On a side note: I tend not to use any instruction besides what is supported by an 8086 (not even 80186 instruction additions like push immediate, pusha, popa etc). That includes using 386 32-bit addressing. Once I know I am operating on a 386 then I have no problems using 386 features. So in this case, I'd make the same recommendation someone else mentioned - put 0x2000 in a segment register and uses 16-bit offsets from that segment. If you are writing code that you know will have to be on a 386 then of course as a designer using 386 instructions. 386 features or 32-bit addressing would be your choice.

Posted: **Thu Dec 24, 2020 3:20 am**

MichaelPetch wrote: 32-bit Addressing and addressing an effective address beyond the segment limits are not the same thing. 32-bit addressing is legitimate and allowed by the 386+ processors in real mode and was designed that way. Did you miss the part where I said "It is just that you have to be careful when the offset exceeds the limit for the segment it is accessing". If you use 32-bit addressing and exceed the LIMIT of the segment being referenced then THAT IS A PROBLEM. But if you use 32-bit addressing and the effective address remains within the limits then that is perfectly acceptable. If you have a memory address [123+eax+esi*4] and the computed effective address falls within the segment limits there is nothing wrong with that even in real mode.

Of course. It's all about limits and not if 32-bit instructions & addressing is supported in real mode or not. Besides, this is the worst thing with long mode not supporting 64-bit registers & addressing in protected mode.

MichaelPetch wrote: Then Microsoft started embracing unreal mode with things like HIMEM.SYS in the 1980s and some companies pushed back on Intel's view of switching back to real mode.

The HMA area exploited with himem.sys is not depedent on unreal mode. It exploited the "feature" that if you load a segment register with FFFF then you could access 64k - 16 bytes above the 1 MB barrier. Some systems let this address wrap around to zero while others didn't, which is also why we have the A20 address line hacks.

As for switching back to real mode, it was the 286 processor that had a specific way to enter protected mode that couldn't be undone. AFAIK, it's still impossible to get back that way. Intel added another possibility in the 386 processor that was possible to use to get back to real mode with. They also added the V86 mode to the 386 processor to be able to emulate real mode, as well as the 32-bit extention and paging. I think the 386 processor was a master-piece in good design, but unfortunately, software & compilers largely have been unable to use it as it was meant to be used.

MichaelPetch wrote: As others have correctly pointed out, on real hardware *some* BIOSes put the processor in unreal mode to do work before reaching the bootloader and don't put the segment limits back to 0xffff. The result is that on some systems the limits are 4GiB by the time the boot sector is loaded and starts executing. Yes, even BIOSes often rely on unreal mode to do work. You can't rely on this behaviour but is often one reason why some hardware operates differently than others if you are exceeding the limit of 0xffff. If you want to access beyond the 0xffff limit and wish to remain in real mode then explicitly entering unreal mode is a viable option.

Based on the VBE entrypoint, I don't think BIOSes rely a lot on unreal mode. If they did, we would not be able to run the BIOS in V86 mode since that mode has fixed 64k limits for segments, and practically every BIOS (except the new i3 that switches to protected mode) does support running the VBE interface in V86 mode.

OSDev.org

Memory reading problem in real hardware.

Memory reading problem in real hardware.

Re: Memory reading problem in real hardware.

Re: Memory reading problem in real hardware.

Re: Memory reading problem in real hardware.

Re: Memory reading problem in real hardware.

Re: Memory reading problem in real hardware.

Re: Memory reading problem in real hardware.

Re: Memory reading problem in real hardware.

Re: Memory reading problem in real hardware.

Re: Memory reading problem in real hardware.

Re: Memory reading problem in real hardware.

Re: Memory reading problem in real hardware.

Re: Memory reading problem in real hardware.

Re: Memory reading problem in real hardware.

Re: Memory reading problem in real hardware.