How to load files from FAT12 image in Protected Mode?

[arjob]

To run my OS in native hardware, I write the 1.44 floppy image on a pen drive, with with the `dd` command, and boot from this pen drive. The BIOS finds a BPB and emulates a floppy drive.

In Real Mode, the custom bootloader implements a FAT12 `LoadFile` routine using the BIOS INT 0x13. The bootloader loads the 2nd stage bootloader, which in turn loads the `Kernel Loader`, switches to PMODE, and jumps to the `Kernel Loader`.

Now in the `Kernel Loader`, the BIOS or the real mode `loadFile` routines are no longer available. The floppy emulation layer by the BIOS is also no longer available.

I am looking for any suggestion, how to implement FAT12 driver in Protected mode. At this stage I however really do not want to write a whole USB Controller driver in order to talk to the pen drive.

Would it be easier if I use MBR or CDROM image, instead of a floppy image?

[alexfru]

The kind of image has nothing to do with BIOS availability.

Your options:

• convert the floppy FS into a RAM FS, so you can still write to it, but the changes won't persist (this may be OK for a lot of things)
• make your protected mode code friendly with BIOS (very broadly: switch to/from real or virtual 8086 mode, let BIOS regularly handle timer interrupts, use I/O ports and memory below 1MB), so you can keep using it whenever you want to
• adopt UEFI

I'd seriously consider UEFI over BIOS. Because... future.

[arjob]

The kind of image has nothing to do with BIOS availability.

Your options:

• convert the floppy FS into a RAM FS, so you can still write to it, but the changes won't persist (this may be OK for a lot of things)
• make your protected mode code friendly with BIOS (very broadly: switch to/from real or virtual 8086 mode, let BIOS regularly handle timer interrupts, use I/O ports and memory below 1MB), so you can keep using it whenever you want to
• adopt UEFI

I'd seriously consider UEFI over BIOS. Because... future.

Thanks, but how does other OS did it in the past. How does it talk to disk drives very early in Protected Mode? (say CDROM or USB stick).

[iansjack]

They implement drivers for the hardware.

[arjob]

The kind of image has nothing to do with BIOS availability.

Your options:

• convert the floppy FS into a RAM FS, so you can still write to it, but the changes won't persist (this may be OK for a lot of things)
• make your protected mode code friendly with BIOS (very broadly: switch to/from real or virtual 8086 mode, let BIOS regularly handle timer interrupts, use I/O ports and memory below 1MB), so you can keep using it whenever you want to
• adopt UEFI

I'd seriously consider UEFI over BIOS. Because... future.

Can you please explain how UEFI would be usable here? Would it help is if instead of using the USB pen drive, I create an CD/DVD image?
How would anyone address this problem of access to a block device with in Protected Mode? I am looking for an easier option, but not really want to scrap the BIOS bootloader for now.

[arjob]

They implement drivers for the hardware.

Yes I understand that, but for a hobby OS, what would make more sense? I feel here are my options now.

A) RAMDISK
1. Should I create a RAMDISK which contains all the necessary files for the OS.
2. The 2nd stage loader, will read from the USB stick (current in Real mode, so using INT 0x13 routines) to RAM.
3. Switch to Protected mode and jump to `Kernel Loader`
4. The kernel loader will then unpack the RAMDISK in RAM and call kernel.

B) Write a USB Controller Driver (may be EHCI).
From what I understand, to get the base address, I will need some sort of PCI driver as well.

C) Some other alternate, that is less work than creating a full USB device driver
May be booting from a CD Image or something else. Sorry I really do not have much information on CD/DVD drives.

Please suggest.

[iansjack]

What a lot of people do, at least in the early stages of developing an OS, is to work with a virtual machine. Qemu is very popular. This provides good debugging facilities and presents a well-defined set of relatively simple hardware (e.g. PS/2 keyboard, IDE disk controller, E1000 NIC). You can fairly easily write drivers for such hardware and leave the more esoteric stuff, such as USB, till a later date.

This allows you to concentrate on the meat of the OS with a good range of hardware drivers to play with.

[Schol-R-LEA]

If you don't mind me asking, do you have an offsite repository (on a hosting site such as GitHub, Sourceforge, or Gitlab) where we could look over your code? Keeping a VCS repo is highly recommended anyway, but if we have any reason to help you with the code itself, it would save you and us a lot of trouble. I know I harp on this topic a lot, but it really does help.

Also, what language are you implementing your OS in? This is just curiosity on my part, though it may be relevant W.R.T. how practical implementing a USB driver will be (e.g., if you are writing everything in assembly, it would be possible but would be a massive effort).

Finally, could you tell us a bit about your test platform? I am specifically wondering how old the PC (particularly the motherboard and CPU) is.

With that out of the way, back to the topic at hand.

One more option for the short term, if you aren't already doing so, is to hold off on physical hardware for the time being and use an emulator or virtualizer (Bochs, QEMU, VirtualBox, etc.) which allows you to treat a file image on your system storage as if it were a physical disk, which you can define as needed (floppy drive, hard disk, CD-ROM, USB drive, etc.). You can even mount the USB drive live on most virtualizers, though you'd have the same issue you are facing now.

While this is only a short term solution, it would allow you to test your OS's kernel drivers, and get your file system code in order, before then trying to implement a USB stack.

I am guessing that this is what you've been doing already for developing your system to the point you have it at now, but I felt it would be remiss if I didn't at least bring this up. The file system code is (or usually should be) largely independent of the storage drivers (I hope Ben appreciates me recommending his books), so it makes sense to get the high-level file system debugged before trying to implement the more complex drive interfaces.

That having been said, I wouldn't recommend using a FAT12 file system on real hardware today, anyway, if only because you'd be limited to a very small maximum disk size (127 MiB using a standard sector size of 512). Even older USB drives are generally a great deal larger than that.

As for the other options:

• I would recommend against implementing V86 mode for the purpose of accessing the BIOS; it is almost as difficult to implement as the USB drivers you are hoping to avoid writing. In any case, as I explain below regarding UEFI, this isn't going to be feasible on newer systems anyway, as they don't have any BIOS to call.
• A RAMDisk would be a good deal simpler to work with, but it leads to a chicken and egg problem: you'd need a RAMdisk large enough to hold the entire disk image, but you'd need to be able to do this in real mode, before you'd have access to the full 32-bit memory space. Unless you happen to be using an 360KiB disk image, this simply isn't going to be possible.
• UEFI is where all (or at least most) real hardware is going, and if you are intending to work on live machines newer than 2019, you won't have the Legacy BIOS available to you (since the Compatibility Support Module - the UEFI legacy boot support - was phased out for new Intel and AMD firmware as of 2020). Sooner or later, you'll need to switch to UEFI booting, so it may be worth looking towards now - though in that case, you won't be using FAT12 anyway; my understanding is that UEFI GPT only works for partitionable drive types, and requires at least one 100MiB FAT32 partition for the GPT. The good news is, the UEFI Boot Services can start you directly in protected mode (or even long mode, for 64-bit systems), and at boot time will load any files you need to load for the OS itself; the bad news is that these services are only usable before you hand control over to the OS kernel.

In the end, I wouldn't see booting a FAT12 system on live hardware as a practical move, period. While it is good as an exercise (which is why I am doing it now myself), it is something best relegated to emulation, as it won't even be feasible on most modern computer hardware.

[arjob]

Firstly I really thank you for your comment. It helped me a lot.

If you don't mind me asking, do you have an offsite repository (on a hosting site such as GitHub, Sourceforge, or Gitlab) where we could look over your code? Keeping a VCS repo is highly recommended anyway, but if we have any reason to help you with the code itself, it would save you and us a lot of trouble. I know I harp on this topic a lot, but it really does help.

I have my code in Github Megha x86 OS.

Also, what language are you implementing your OS in? This is just curiosity on my part, though it may be relevant W.R.T. how practical implementing a USB driver will be (e.g., if you are writing everything in assembly, it would be possible but would be a massive effort).

Finally, could you tell us a bit about your test platform? I am specifically wondering how old the PC (particularly the motherboard and CPU) is.

The bootloaders are written in Assembly language and the Kernel in C. The `x86/kernel.c` just contains boilerplate code, so please excuse the quality. I tried to keep platform dependent parts of the OS separate from the independent parts.

I generally use qemu (to debug parts of kernel using GDB), and Bochs (to debug assembly language parts). I also use VirtualBox to test.

The FAT12 was an exercise to test my skills, but also I wanted to start from the simplest FAT (that is still supported).
You are right about the RAMDISK approach. Given the way I setup memory, there is about 500 KB of free addressable RAM at 2nd bootloader stage. It may become an issue in future, tough I am looking to keep the total kernel size as small as possible.
The UEFI is too new for me and have little idea. I do not want to jump to it before doing some reading on it. But I agree it is tempting.

One question tough, even if I use QEMU, I will still need to develop the floppy controller driver to get to the files in PMODE. But this driver will be of no use, on Physical hardware. How to you see this extra work?

Thank you again.

[iansjack]

Why use a floppy for the files? Why not use an IDE hard disk (as a virtual disk file), which is much simpler to access.

[bzt]

Why use a floppy for the files? Why not use an IDE hard disk (as a virtual disk file), which is much simpler to access.

Plus one on this. Furthermore FAT16 or FAT32 is easier to parse (no need to access half bytes in the FAT table).

Can you please explain how UEFI would be usable here?

Read this. Using POSIX-UEFI, you'll have the usual "fopen" interface (see code example).

Thanks, but how does other OS did it in the past. How does it talk to disk drives very early in Protected Mode

Read this. BTW reading count sectors from ATA hard disk starting at LBA start into buf can be as simple as:

Code: Select all

void readsec(int start, int count, unsigned char *buf)
{
    int i;
    for(i = 0; i < count; i++, buf += 512, start++) {
        while((inb(0x1F7) & 0xC0) != 0x40); // wait for ready
        outb(1, 0x1F2);                     // set the lba to read from
        outb(start, 0x1F3);
        outb(start >> 8, 0x1F4);
        outb(start >> 16, 0x1F5);
        outb((start >> 24) | 0xE0, 0x1F6);
        outb(0x20, 0x1F7);                  // issue read sectors command
        while((inb(0x1F7) & 0xC0) != 0x40); // wait for ready
        insl(0x1F0, buf, 512/4);            // read in data
    }
}

This code can be compiled in prot-mode because it does not use BIOS interrupts. But since this only provides sector level access, you'll have to interpret the file system yourself.

there is about 500 KB of free addressable RAM at 2nd bootloader stage. It may become an issue in future

That's doubtful. My 2nd stage is only about 13k, yet it interprets GPT, FAT16/32, loads RAMDISK in many formats (tar, cpio, sfs, James Molloy's etc.), uncompresses with gzip, parses PE+ and ELF kernels, sets up Symmetric Multi Processing, etc. etc. etc. All of that in 13k.
The UEFI implementation is much bigger, a bit over 100k, but that also supports AES encrpyted RAMDISKs. You can take a look at the source how I've used EFI_SIMPLE_FILE_SYSTEM_PROTOCOL to load files.

Cheers,
bzt

[arjob]

Why use a floppy for the files? Why not use an IDE hard disk (as a virtual disk file), which is much simpler to access.

Plus one on this. Furthermore FAT16 or FAT32 is easier to parse (no need to access half bytes in the FAT table).

I know that now, but when I began, I wanted to start at an earlier file system.

Can you please explain how UEFI would be usable here?

Read this. Using POSIX-UEFI, you'll have the usual "fopen" interface (see code example).

So yesterday, I started finding some resource on UEFI on the internet. I got the GNU-UEFI, compiled it and was able to run a `Hello World` type UEFI application in QEMU using the OVMF firmware. I was also able to copy this UEFI application to my pen drive (and rename it to /EFI/BOOT/BOOTX64.EFI) and boot on my laptop. The problem is that I am finding it difficult to find good resource on the internet, most just show the steps, but do not go deep. PHOENIX WIKI has some very good reference resource.

there is about 500 KB of free addressable RAM at 2nd bootloader stage. It may become an issue in future

That's doubtful. My 2nd stage is only about 13k, yet it interprets GPT, FAT16/32, loads RAMDISK in many formats (tar, cpio, sfs, James Molloy's etc.), uncompresses with gzip, parses PE+ and ELF kernels, sets up Symmetric Multi Processing, etc. etc. etc. All of that in 13k.
The UEFI implementation is much bigger, a bit over 100k, but that also supports AES encrpyted RAMDISKs. You can take a look at the source how I've used EFI_SIMPLE_FILE_SYSTEM_PROTOCOL to load files.

I think I will make my OS support both Legacy and UEFI BIOS. You are right, 500 KB is huge, it will be enough for the RAMDisk. But out of interest, I will also develop an UEFI bootloader, capable to handling larger RAMDisks.
I plan to use CPIO archives, because of their simplicity, and will attempt compressed formats later on.

What do you think? If possible can you please point me to some UEFI resource like reference material or some book. Reading the whole UEFI spec is not really feasible.
I would love to look at your source codes, but the URLs do not work.

[kzinti]

What do you think? If possible can you please point me to some UEFI resource like reference material or some book. Reading the whole UEFI spec is not really feasible.

One of the big problem with UEFI is that there is not a lot of info out there on how to use the APIs. There isn't a lot of good examples around the internet. The best you can do is look at them, look at the specifications and figure out how to do things.

Here is how I do it:

Main code:
https://github.com/kiznit/rainbow-os/bl ... fiboot.cpp

Display code:
https://github.com/kiznit/rainbow-os/bl ... isplay.cpp

File system code:
https://github.com/kiznit/rainbow-os/bl ... system.cpp

[bzt]

So yesterday, I started finding some resource on UEFI on the internet. I got the GNU-UEFI, compiled it and was able to run a `Hello World` type UEFI application in QEMU using the OVMF firmware. I was also able to copy this UEFI application to my pen drive (and rename it to /EFI/BOOT/BOOTX64.EFI) and boot on my laptop. The problem is that I am finding it difficult to find good resource on the internet, most just show the steps, but do not go deep. PHOENIX WIKI has some very good reference resource.

Give a shot to POSIX-UEFI, and also our wiki is one of the best places to find info on UEFI (still not very detailed, but has many great links to tutorials and working example code).

there is about 500 KB of free addressable RAM at 2nd bootloader stage. It may become an issue in future

That's doubtful. My 2nd stage is only about 13k, yet it interprets GPT, FAT16/32, loads RAMDISK in many formats (tar, cpio, sfs, James Molloy's etc.), uncompresses with gzip, parses PE+ and ELF kernels, sets up Symmetric Multi Processing, etc. etc. etc. All of that in 13k.
The UEFI implementation is much bigger, a bit over 100k, but that also supports AES encrpyted RAMDISKs. You can take a look at the source how I've used EFI_SIMPLE_FILE_SYSTEM_PROTOCOL to load files.

I think I will make my OS support both Legacy and UEFI BIOS. You are right, 500 KB is huge, it will be enough for the RAMDisk.

Not for that. For the 2nd stage loader. For the RAMDISK (or initrd how I like to call it, initial ram disk), surely will be at least a few MBs. My loader usually expects it to be around 8 - 16 MB and it is optimized for that. I can't tell where my loader loads the initrd, because it uses dynamic allocation consulting the firmware provided memory map.

Keep your focus on one thing only. Writing a multiplatform boot loader is not an easy task, it's more than enough if you can get your OS booted on one platform at first.

But out of interest, I will also develop an UEFI bootloader, capable to handling larger RAMDisks.
I plan to use CPIO archives, because of their simplicity, and will attempt compressed formats later on.

That sounds like a plan

What do you think? If possible can you please point me to some UEFI resource like reference material or some book. Reading the whole UEFI spec is not really feasible.
I would love to look at your source codes, but the URLs do not work.

No material that I know of, I've used the (very bad) specification, the C include files shipped with the EDK2, read through EDK2's and GNU-EFI's source many many times, and I've spent countless hours searching the net for examples and trying out things on my own (trial-and-error is slow, but guaranteed to work). You are in a big advantage over me because now tutorials exists and the wiki is a lot bigger than it was when I implemented by UEFI loader 5 years ago. Everything you want (UEFI, larger ramdisks, CPIO, decompression) I have already implemented, so you can just take a look at BOOTBOOT to see a working solution.

Cheers,
bzt