How Read Sectors from boot USB in Long Mode

[kemosparc]

Hi,

I have a problem reading sector from the USB device I boot from.

I have a hobby USB with my own boot loader. I boot from the USB and read a set of sectors using INT 13. I then proceed until I get into real mode where I map some memory enough to read data from
the USB I booted from (about 8M which I did not have enough memory in real mode to read it into).


The problem is that when I start my OS in QEMU with -usb I can detect the USB as my primary master IDE and this allows me to read the required sectors. The problem occurs when I boot from a physical real computer. I boot the USB using HDD emulation, but when I get to long mode and reach the code that scans the IDE/ATA I only can detect the internal drive and I cannot see the boot USB which I was able to detect in QEMU.

My question is that in this case where I need to load sectors from my boot device after switching to long mode for memory availability reasons, what is the best way to deal with my boot device as an IDE/ATA


Thanks,
Karim.

[glauxosdev]

As far as I know, there is no way to access a USB flash drive as an IDE drive. I suppose that QEMU emulated your USB disk as a HDD, I doubt there are other possibilities. A USB flash drive is not an IDE hard disk.

You need to write a proper USB stack. You can read the [ohci / uhci / ehci / xhci] specification, but also some source code to get an idea. I have implemented only USB 2.0, so for ehci:

• First you need to find the ehci controller on pci. Its address varies. Just use the bios to get it.
• Then you need to initialize it.
  1. Get base address, operational base address, extended capabilities offset.
  2. Request ownership from the bios.
  3. Reset controller
• Scan for devices (for now I recommend sticking with one device only).
• Reset and enable the port that has the device connected.
• Using control endpoint and address 0 get device-configuration-endpoint descriptors
• Using endpoints got from endpoint descriptors, set unique address on usb bus. From now use only this address.
• Similarly, set configuration.
• Send inquiry command.
• Test unit ready
• Read capacity
• Only then request Read10 / Read16 command

This is not an easy task, you will need to try for whole months filled with frustration to achieve USB read/write support. Good luck!

Regards,
glauxosdev

[kemosparc]

Thanks for the reply,

So you think that it might be a better idea to switch back and forth between long mode and real mode


Thanks,
Karim.

[glauxosdev]

Definitely no.

I said "Just use the BIOS", but only to get the ehci controller pci address, and only that. You can do this while you are still in real mode. I hope this is clear now.

Regards,
glauxosdev

[kemosparc]

Thanks for the reply

One question though crossed my mind. If I have 2 cores a BSP and an AP, and the BSP is at long mode and then it send and init IPI to the AP, so my question is that can the AP has access to INT 13h while it is in the trampoline? I mean that this way I can use it to read sectors from the USB. Is that possible?

Thanks
Karim.

[glauxosdev]

I don't know if this is possible, just do it the proper way.

Regards,
glauxosdev

[kemosparc]

Yes, you are right, I thihk I will do that.

Right now my long mode OS parses the PCI tables and can enumerate the devices (Vendor/Mode/Class/Subclass), IDT in place, and memory is mapped with page tables.

I know nothing about USB devices and how detect them and communicate with them.

Can you please point me to the documentation that would take me step by step from scratch.

Thanks a lot.
Karim.

[Brendan]

Hi,

So you think that it might be a better idea to switch back and forth between long mode and real mode

During boot, you mostly go through 4 distinct phases:

Phase I: Relying on firmware

During this phase, you can switch between real mode and protected mode or long mode as much as you like, you can use memory, and you can use any firmware/BIOS functions (while in real mode). However, during this phase the firmware "owns" the hardware - you can't mess with MTRRs, or reconfigure PICs, or setup IO APICs, or change anything in PCI configuration space, etc. The main goal is to prepare for "Phase II"; which includes (e.g.) using the BIOS to load everything you might want into memory before starting "Phase II".

Phase II: Taking control

When "Phase I" is finished you take control of all the hardware (so that you can start setting things up how you want them). Because you are now in control of the hardware you can not use the firmware anymore. During this phase you'd be initialising fundamental kernel functionality (memory management, scheduler, IPC, IRQ handling, etc); and detecting what hardware is present, then starting device drivers and services that depend on those device drivers (networking stack, file system, etc).

Phase III: Operational

This is where your kernel is initialised and working, and you're using your own device drivers. The OS might still not be "fully functional", but it has everything it needs. For example, there might be some non-essential device drivers (e.g. sound card) that still need to be loaded from disk and started, but the kernel can do that itself because it does have its own disk drivers and file system code working already.

Phase IV: Running

This is where everything is ready and (e.g.) the end user can login and start using applications.


As far as I can tell, you're still doing "Phase I" (and trying to load things like kernel and disk drivers into memory). You should use BIOS for that.

One question though crossed my mind. If I have 2 cores a BSP and an AP, and the BSP is at long mode and then it send and init IPI to the AP, so my question is that can the AP has access to INT 13h while it is in the trampoline? I mean that this way I can use it to read sectors from the USB. Is that possible?

Mostly; no it won't work.

What will happen is that the BIOS will wait for the disk controller or USB controller to send an IRQ, and the PIC will send that IRQ to the BSP. If you attempt to work-around that problem by setting up a special IDT for BSP and trying to forward the IRQ to the AP using an IPI; then it still won't work because the AP won't have any way to send an EOI to the local APIC (to acknowledge your IPI). Basically it'll work for one IRQ and then your IPI won't work anymore.

In theory it may be possible to work around the problems in some other way (potentially by using the IO APIC in the deprecated "symmetric mode" where both the PIC chip and the IO APIC are used at the same time); however it'd be like tap dancing through a mine field while someone is randomly changing the location of the land mines - getting it to work reliably on all computers is probably impossible (and just using the BIOS is far easier and a lot more reliable).


Cheers,

Brendan

[kemosparc]

Hi,

Thanks for the thorough reply. Although I think I am in phase 2 as I am in long mode and I have PIT, Keyboard, IDE(INT 7 and 15), memory mapped, e1000 working as well as context switching and user mode.

My problem is that the code I want to load to run in user mode is on the USB stick and it is large enough to load it while I am in real mode, basically I don't have enough memory. I am only able to load my ~ 100KB kernel, and I need to read the user mode code into memory after mapping some pages and make room for that.

When I use QEMU, I can do that because I can write everything to a disk image and boot from it and after switching to long mode I can read using LBA the user mode code.

But in physical environments (Real Box) when I try to detect my primary and secondary buses I only see the internal HDD which does not contain the stuf I need.

I hope that this clarified my situation

Thanks,
Karim.

[Octocontrabass]

Qemu does not provide access to the USB stick. It emulates an IDE disk drive, and uses the USB stick as the source for reads and destination for writes.

On a real machine, you need a USB driver to access a USB device.

My problem is that the code I want to load to run in user mode is on the USB stick and it is large enough to load it while I am in real mode, basically I don't have enough memory.

Unreal Mode is good for this. It can also be useful as your kernel grows larger.

[kemosparc]

Thanks a lot.

This is exactly what I was looking for. Thank you very very very much

Karim.

[Brendan]

Hi,

Thanks for the thorough reply. Although I think I am in phase 2 as I am in long mode and I have PIT, Keyboard, IDE(INT 7 and 15), memory mapped, e1000 working as well as context switching and user mode.

My problem is that the code I want to load to run in user mode is on the USB stick and it is large enough to load it while I am in real mode, basically I don't have enough memory. I am only able to load my ~ 100KB kernel, and I need to read the user mode code into memory after mapping some pages and make room for that.

Is it a micro-kernel (e.g. where the "user mode code" a driver driver)? If it is a micro-kernel, then the boot loader should/must load things like disk drivers, etc into memory (in addition to loading the kernel) using the BIOS. In this case, as the OS gets larger, you can expect everything the boot loader needs to load to grow to several MiB.

If it's a monolithic kernel where things like device drivers are implemented as separate "kernel modules", then it's mostly the same (boot loader must load kernel and the modules it needs). In this case, as the OS gets larger, you can expect everything the boot loader needs to load to grow to several MiB.

If it's a monolithic kernel where things like device drivers are built into the kernel itself, then boot loader might be able to just load the kernel. In this case, as the kernel gets larger, you can expect it to grow to several MiB.

Basically; for all of these cases, the boot loader has to switch between real mode and protected mode or long mode as it load everything into memory (before the OS takes control of the hardware).

If the problem is that you're doing this already, but your kernel doesn't have various device drivers (OHCI, UHCI, EHCI, XHCI, USB storage device driver, AHCI, SCSI, whatever) then you need to write device drivers and can't/shouldn't use the BIOS because you've already taken control of the hardware. However...

For micro-kernels and most (modular) monolithic kernels; because you're loading more than just a kernel during boot often it makes sense to load an initial RAM disk containing everything. In this case; it's possible for the initial RAM disk to contain more than necessary. For example, if you don't have USB drivers but want to load/test an application, then you could put that application into your initial RAM disk and load it during "Phase I", so that it's still in memory (and maybe mounted in the virtual file system somewhere, like at "/initrd") after boot (and can be used during "Phase II" and "Phase III", even though there is no USB drivers).


Cheers,

Brendan