Understanding a Good Generic FAT12 Floppy Boot Sector

[~]

Here I intend to explain the structure of a FAT12 floppy boot sector.

It needs to activate Unreal Mode, be compatible with DOS loading standards (loading image at 600h or 700h) so we can implement the skeleton of a DOS system with the intention to reuse code and known standards and load easily the second part of the system from there, while keeping a good deal of scalable capabilities at the DOS level for the booting stage.

With these simplifications, it will be very easy to start learning about the booting process from the simplest device (the floppy).

Being so tiny, this program can serve us as a model to pack concrete components that are directly related, and that aren't so apart that it would be difficult to describe them in a tutorial in a short time and space.


Look at how the program skeleton has a list of numbered comments that describe the exact order in which it must execute the different booting tasks. To complete it, you just need to include the equally-numbered pieces of actual code that will be presented in different posts containing explanations for these different specific tasks.
_________________________________________________________________________
_________________________________________________________________________
_________________________________________________________________________
_________________________________________________________________________
FAT12 Boot Sector Empty Skeleton

Code: Select all

BASEADDR     equ 0h   ;Change this to 0h or 7C00h for global start
BOOT_ADDR    equ 7C00h
BASEADDR_SEG equ BOOT_ADDR>>4
org BASEADDR

%define resbytes 510-(unused-_00h_jmp)

_00h_jmp: db 0EBh                             ;jmps 003E
          db 03Ch                             ;2  (2)
_02h_nop: nop                                 ;1  (3)

_03h_OEMid               db    "OEMIdent"      ;8  (11)
_0Bh_bytesPerSect        dw         0200h      ;2  (13)
_0Dh_sectsPerClust       db          001h      ;1  (14)
_0Eh_reservedSects       dw         0001h      ;2  (16)
_10h_numOfFATs           db          002h      ;1  (17)
_11h_numRootDirEntries   dw         00E0h      ;2  (19)
_13h_numSectors          dw         0B40h      ;2  (21)
_15h_mediaType           db          0F0h      ;1  (22)
_16h_numFATsectors       dw         0009h      ;2  (24)
_18h_sectorsPerTrack     dw         0012h      ;2  (26)
_1Ah_numHeads            dw         0002h      ;2  (28)
_1Ch_numHiddenSects      dd     00000000h      ;4  (32)
_20h_numSectorsHuge      dd     00000000h      ;4  (36)
_24h_driveNumber         db           00h      ;1  (37)
_25h_reserved            db           00h      ;1  (38)
_26h_signature           db           29h      ;1  (39)
_27h_volumeID            db        "????"      ;4  (43)
_28h_volumeLabel         db "VolumeLabel"      ;11 (54)
_36h_FSType              db    "FAT12   "      ;8  (62)




;INIT: 448 free bytes
;INIT: 448 free bytes
;INIT: 448 free bytes

 ;1. Code to configure CS:IP here:
 ;;


 ;2. Code to load GDT and enable Protected Mode here:
 ;;


 ;3. Load registers with GDT selectors, disable Protected Mode and
 ;   set up stack and Un/Real Mode segment registers:
 ;;


 ;4. Enable A20 Line here:
 ;;


 ;5. Reenable interrupts here:
 ;;


 ;6. Configure program disk parameter information here:
 ;;


 ;7. Search 8.3 file name in root directory:
 ;;


 ;8. Read file contents:
 ;;


 ;9. Jump to the 16-bit Real Mode bootup image
 ;   (it's intended to jump to 70h:0000h or 700h physical):
 ;;







 ;10. read_sectors BIOS-based function:
 ;;




 ;11. 8.3 bootup file name:
 ;;


 ;12. GDT table with 3 selectors (null, code, data) here:
 ;;


 ;13. Program variables here:
 ;;


  unused: times resbytes db 0x55
;END:  448 free bytes
;END:  448 free bytes
;END:  448 free bytes



_1F_55AA_signature      dw 0xAA55        ;2    (512)

_________________________________________________________________________
_________________________________________________________________________
_________________________________________________________________________
_________________________________________________________________________
Major Components

Once we know the structure of the boot sector skeleton, we need to perform the following tasks:

1. Enable line A20.
2. Enable Unreal Mode.
3. Finding the binary file to run, loading and searching the root directory 1 sector at a time.
4. Read the file directly to its final memory location, 1 cluster at a time (typically 1 sector per cluster).
5. Jump to the binary file to run it.

[b.zaar]

All of these things should be documented in the wiki pages but usually your major components list are done from a second stage boot loader, the boot sector is pretty small to fit all these functions.

[~]

All of these things should be documented in the wiki pages but usually your major components list are done from a second stage boot loader, the boot sector is pretty small to fit all these functions.

The first stage boot loader will also use a file here.

The idea is to have a boot sector capable to load a 16-bit Unreal Mode DOS-like kernel at physical address 0x700, and make it load a more formal 32 or 64-bit system (and make it configurable, even provide a simple command line shell), as if the final kernel was DOS program to run on top of it, but that completely replaces the DOS-like environmet, used only to detect the machine devices and capabilities.

[b.zaar]

I don't have the code online anymore to look over but the way I tried this was to create a standard FAT12 boot sector that loads the first data sector. This means your 2nd stage boot loader must be written as the first file, this was standard with all DOS version up to v6. The 2nd stage boot loader then sets the GDT and the selectors to work in unreal mode, there's no need to load the boot loader as a 16 bit pmode file because it mostly runs as a true 16 bit program. The code to safely initialize unreal mode takes about 512 bytes or more just to include code for detecting the cpu is 386+ and the hardcoded GDT data. With your current plan to include this in the boot sector you would also need to include the locate and read file functions and this is difficult if not impossible in 512 - sizeof(BPB) bytes.

[alexfru]

Check out my BootProg. FAT12 floppies, FAT32 USB sticks work. Loads standard .COMs/.EXEs into low memory just like you want. I've been meaning to put together a second stage bootloader demo, but haven't yet got around to doing it. The idea is to use my Smaller C compiler, which compiles for real mode too (in the huge memory model the C code needs not to deal with segmentation explicitly (the compiler translates pointers into segment and offset pairs behind the scenes), only calls to BIOS/DOS need special care). A bigger idea is to implement a small DOS-like OS, perhaps in real mode as well, and have it fully self-hosting together with Smaller C.

[mikegonta]

With these simplifications, it will be very easy to start learning about the booting process from the simplest device (the floppy).

Could it possibly be any simpler?

[b.zaar]

@alexfru: It's kinda cool you can load exe files straight from the bootsector and even search by filename, I think you've sacrificed something in the way of user friendly messages for normal users though.

[alexfru]

@alexfru: It's kinda cool you can load exe files straight from the bootsector and even search by filename, I think you've sacrificed something in the way of user friendly messages for normal users though.

There isn't much to message about. There are messages for when the file isn't found or can't be read. That file (2nd stage bootloader) can print "relax, foobaros is loading" for the user's amusement.

[~]

Check out my BootProg. FAT12 floppies, FAT32 USB sticks work. Loads standard .COMs/.EXEs into low memory just like you want. I've been meaning to put together a second stage bootloader demo, but haven't yet got around to doing it. The idea is to use my Smaller C compiler, which compiles for real mode too (in the huge memory model the C code needs not to deal with segmentation explicitly (the compiler translates pointers into segment and offset pairs behind the scenes), only calls to BIOS/DOS need special care). A bigger idea is to implement a small DOS-like OS, perhaps in real mode as well, and have it fully self-hosting together with Smaller C.

I learned about boot sectors mostly with bootf02.zip and bootr01.zip from John Fine.

OSDever.net - OS Development Downloads

Loading binaries with a file format will be very useful later soon.

I was thinking about implementing task containing a 386 emulator in 16-bit Unreal Mode in the kernel to run BIOS services or U/EFI services later in time. It would be very important but the system wouldn't rely on it so if it fails, it could be restarted for being able to access low-level functions in case they are needed. I suspect that Windows still keeps an internal emulator to run the BIOS for some tasks, mostly display and other basics, and that's why it still gets unstable slowly - maybe when the emulator fails it can adjust less things at that level or some runtime configuration is broken until next reboot.

I was also thinking about implementing driver interfaces to handle Linux and Windows 9x and/or Windows NT drivers (and find out how to use the different kinds and classes of devices in a standard way). It will be necessary to load ELF and PE files.

Then I will add an interface to make use of the same virtual addresses and virtual I/O ports for the same devices always no matter the vendor or model of the actual peripheral. In this way I can get to use many devices using the drivers they are shipped with and assign the same standarized hardware interface program, which should be really simple to use yet complete and scalable, but capable of keeping the fundamental hardware interfaces unchanged permanently no matter how much time passes. I just need to understand the different executable file formats and how they are loaded through the OS, and how to use the services they provide.

With these simplifications, it will be very easy to start learning about the booting process from the simplest device (the floppy).

Could it possibly be any simpler?

It's really good to have FAT32 in a floppy. With that now we can practice accessing FAT32 in practical-sized floppy images. I just hope it doesn't disappear eventually from the Internet like most extremely valuable programming content from classic 90's, at least enough to back it up massively.

Probably many of the people who did those example programs and code got their storage media damaged and maybe that's why it's now scarce. But that's the most serious case possible of content disappearance.

[~]

1. Normalizing the Boot Code Address

It looks like there are machines that boot from 7C0h:0000h and others that boot from 000:7C00h, and that could prevent booting properly, specially if we involve elements that require absolute physical memory addresses, such as the GDT (and paging structures for the loaded boot manager/booting kernel).

So I have made sure that this boot sector will always run from 7C0h:0000h, so all addresses inside the boot code will be relative to base address 0 unless we modify the segment registers, and CS and DS will have base address of 7C0h so that we can access local data normally with such base address 0.

The following code will:
- Set a regular CS:IP across machines by jumping to 7C0h:start_address.
- Disable interrupts.
- Set DS to the same value as CS (7C0h).
- Save the boot drive number (in DL) at physical address 7C00h. Used for sector-reading BIOS 13h service AH=2.

Code: Select all

 ;1. Code to configure CS:IP here:
 ;;

 ;>   cs = 0
 ;>>  dl = drive we were booted from
 ;INIT: Normalize the Boot Code Segment:Offset address
 ;INIT: Normalize the Boot Code Segment:Offset address
 ;INIT: Normalize the Boot Code Segment:Offset address

  ;Force CS to be 7C00h and start program at start_address:
  ;;
   jmp BASEADDR_SEG:start    ;jmp 7C0h:start_address

   start:
   xor eax,eax   ;Get clean divisions by making sure 32 bits of EAX are 0
   cli           ;Make safe transition to Protected Mode
   push cs       ;Get value of Code Segment
   pop ds        ;Make DS == CS



  ;Save drive number we came from. Since
  ;we won't use again the first byte of code from
  ;the boot, the byte of *jmp 03Eh* instruction,
  ;we will use that byte as the variable space for
  ;the disk number. We must recognize that this is
  ;pretty clever.
  ;;
   mov [_00h_jmp],dl


 ;END:  Normalize the Boot Code Segment:Offset address
 ;END:  Normalize the Boot Code Segment:Offset address
 ;END:  Normalize the Boot Code Segment:Offset address

2. Enabling Simple Protected Mode

Here we will load Protected Mode. For that we need to keep a GDT table that contains a null selector, a 4GB code selector, and a 4GB data selector, each of them 8 bytes in size. So this table will be 24 bytes in size.

The GDTR is inside the null selector to save space (anyway the null selector is never used so it's ideal to hold the GDTR).
The following GDT should be put inside but at the very end of the "448 free bytes" area, since we need to put all data out of the way of the boot code (it's as if we were storing the GDT near frome where the partition tables woud be):

Code: Select all

 ;12. GDT table with 3 selectors (null, code, data) here:
 ;;

 ;INIT: GDT
 ;INIT: GDT
 ;INIT: GDT
    GDT:
   ;WARNING: This selector, besides being the pointer to the GDT,
   ;         is the null selector.
   ;;
    SELNull equ 0
         GDT_size:
          dw GDTsize
         GDT_actualptr:
          dd 7C00h+GDT
          dw 0x0000

   SELCod32 equ 8
     dw 0FFFFh        ; bits 0-15 length
     dw 00000h        ; bits 0-15 base address
     db 0             ; bits 16-23 base address
     db 10011010b     ; bits P, DPL, DT and type
     db 11001111b     ; bits G, D and bits 16-19 length
     db 0             ; bits 24-31 base address

   SELDat32 equ 16    ; This is the "flat data selector"
     dw 0FFFFh        ; bits 0-15 length
     dw 00000h        ; bits 0-15 base address
     db 0             ; bits 16-23 base address
     db 10010010b     ; bits P, DPL, DT and type
     db 11001111b     ; bits G, D and bits 16-19 length
     db 0             ; bits 24-31 base address
   GDT_end:

   GDTsize equ (GDT_end-GDT)-1
 ;END:  GDT
 ;END:  GDT
 ;END:  GDT

Now we need to load the GDT and activate the Protection bit (bit 0) in CR0.
The following code should be placed after the code that normalizes the booting CS:IP to 7C0h:0000h:

Code: Select all

 ;2. Code to load GDT and enable Protected Mode here:
 ;;

 ;INIT: Enable protected mode to load GDT and keep it enabled
 ;INIT: Enable protected mode to load GDT and keep it enabled
 ;INIT: Enable protected mode to load GDT and keep it enabled

   lgdt[GDT]     ;Load GDT. This is how we would
                 ;exactly access the GDT in
                 ;Real Mode

   mov ecx, cr0  ;Switch to Protected Mode...
   inc cx        ;Set PE bit
   mov cr0, ecx  ;{5} Here we activate Protected Mode


 ;END:  Enable protected mode to load GDT and keep it enabled
 ;END:  Enable protected mode to load GDT and keep it enabled
 ;END:  Enable protected mode to load GDT and keep it enabled

Unused Extras

This is an extra that could be useful for some case where we need to place or move around the GDT dynamically.

Here we just take the final correct Real Mode code segment (or another data segment that contains the right base address).
Now we shift multiply it 4 positions left (to multiply by 16). Now segment is an absolute physical address.
Now we add the zero-based address for the "GDT" label, and that's the addres that the GDTR should contain.

Code: Select all

GDTR:
dw 0 ;Absolute GDT table size minus 1
dd 0 ;Absolute physical base address where the GDT table resides

Now, what is frequently forgotten is how to use the LGDT instruction.
In short, we just need to keep the right base address in DS, and the "GDT" label can be relative to the current segment.
What's important is that the actual GDTR that the passed label points to, actually contains a physical final address in turn.

To calculate dynamically that GDT table base address into the GDTR, we can use something like this:

Code: Select all

push cs
pop ax
shl ax,4      ;Convert segment value to physical value
mov si,GDT    ;Get the GDTR structure to load
add ax,si     ;Calculate full physical address of GDT table
mov [si+2],ax ;Save it into GDTR
lgdt[si]      ;Load GDT. This is how we would
              ;exactly access the GDT in
              ;Real Mode

[Octocontrabass]

I suspect that Windows still keeps an internal emulator to run the BIOS for some tasks, mostly display and other basics, and that's why it still gets unstable slowly - maybe when the emulator fails it can adjust less things at that level or some runtime configuration is broken until next reboot.

There is an emulator in 64-bit versions of Windows, but it's only used for VESA VBE functions when no better video driver is available. Recent versions of Windows don't use it at all; they use a generic VGA driver instead of attempting to access the BIOS.

I was also thinking about implementing driver interfaces to handle Linux and Windows 9x and/or Windows NT drivers (and find out how to use the different kinds and classes of devices in a standard way).

This is a lot more ambitious than you think.

[mikegonta]

With these simplifications, it will be very easy to start learning about the booting process from the simplest device (the floppy).

Could it possibly be any simpler?

It's really good to have FAT32 in a floppy. With that now we can practice accessing FAT32 in practical-sized floppy images.

And by accessing the BIOS from 32 bit protected mode we have the simplicity and ease of that old time programming style without
any of the 16 bit real mode pain.

[~]

And by accessing the BIOS from 32 bit protected mode we have the simplicity and ease of that old time programming style without
any of the 16 bit real mode pain.

The things that result more painful are trying to access nonstandard hardware (like that after the SoundBlaster, a few Super VGA cards), and also a programming environment that doesn't keep a standard and backwards-compatible structure for simply recompiling without significant and easy-to-find, trivial changes.

[alexfru]

I was also thinking about implementing driver interfaces to handle Linux and Windows 9x and/or Windows NT drivers (and find out how to use the different kinds and classes of devices in a standard way).

This is a lot more ambitious than you think.

Yep. One would need to emulate the entire Windows Driver Model with underlying APIs and behaviors (including bugs). It's impractical and nearly impossible to do. The existing documentation is of limited help (speaking from experience). Further, by the time some WDM is deemed implemented reasonably well, there won't be much existing hardware with drivers for that WDM.

[~]

3. Getting 4GB Address Space for 16 Bits and Specify Stack Pointer

The PUSH instruction decreases the stack pointer and then writes a value at the end of the stack.
The POP instruction reads a value from the stack and then increases the stack pointer.

We can remember those facts when we want to align the stack and prevent the fear of pointing the end of the stack 1 byte after the intended free stack memory range from confusing us and driving us away from this correct configuration concept.

It means that we can have a 512-byte stack right before the last loaded image, at least during boot.
We will use the area 0x500-0x700 to hold the stack.
We will set SS to 0 and SP to 0x700.
We set SP to a WORD/DWORD/QWORD aligned address such as 700h even though the last free address at this region is 6FFh (1 less byte). Then there's the loaded binary.
Knowing how the stack works, we know that as soon as we want to push something into the stack, the pointer will first be decreased and then a value written, so it will always effectively only work within the actually free 500h-700h range.

Code: Select all

;3. Load registers with GDT selectors, disable Protected Mode and
;   set up stack and Un/Real Mode segment registers:
;;

 ;;INIT: Configuration of Unreal segments and also stack
 ;;INIT: Configuration of Unreal segments and also stack
 ;;INIT: Configuration of Unreal segments and also stack
  ;Registers that are modified in this INIT-END portion:
   ;AX (EAX) 0
   ;DS 0
   ;ES 0x800
   ;SS 0
   ;SP 0x800

     ;Registers with values to reuse in the next INIT-END portion:
      ;Apparently ALL of the previous ones.

 ;>   ah = 0
 ;>   dl = drive we were booted from
  mov ax,SELDat32 ;Selector for 4GB data seg. What we do here
                  ;is to take the address of the second selector
                  ;of the GDT.
                  ;We are using the 8-bit register AL to hold the
                  ;16-bit data selector number because AH is
                  ;already set to 0, and with this we save 1 byte
                  ;or so of boot space.

  mov ds,ax       ;{2} Extend limit for DS. Typically
                  ;would get the value 0010h. (16)

  mov es,ax       ;Extend limit for ES. ES would also
                  ;be set to 16.

  dec cx          ;Switch back into UnReal Mode. From the start, ECX
                  ;contained the value of CR0 with PE
                  ;bit set. Here we disable the PE bit again,
                  ;and we put it into CR0 in the next
                  ;instruction.

 ;Disable Protected Mode but keeping extended limits
 ;for segments so we can get into Unreal Mode
 ;;
  mov cr0,ecx


 ;Here we are in UnReal Mode:
 ;Here we are in UnReal Mode:
 ;Here we are in UnReal Mode:
 ;;

 ;Set end of stack to 700h.
 ;From here we have 200h (512) free stack bytes.
 ;The first PUSH will result at 6FCh and if we POP again
 ;we will return to the empty limit of 700h:
 ;;
  xor ax,ax
  mov ss,ax            ;Segment 0.
  mov sp,0x700         ;{1B}. Configure the end of the stack at 0x700
                       ;the very end of the free memory area
                       ;from 500h-700h


 ;Now that we are back in Unreal Mode,
 ;we can set the main data segment register, DS,
 ;to its intended value throughout the program,
 ;since we loaded a GDT selector value here, but now
 ;we need a Real-Mode-styled memory segment value again:
 ;;
  push cs
  pop ds

 ;;END:  Configuration of Unreal segments and also stack
 ;;END:  Configuration of Unreal segments and also stack
 ;;END:  Configuration of Unreal segments and also stack

___________________________________________________________________
___________________________________________________________________
___________________________________________________________________
___________________________________________________________________
4. Enabling Line A20 (ISA - PS/2 Keyboard Controller Method)

Code: Select all

;4. Enable A20 Line here:
;;

 ;;INIT: Enable A20
 ;;INIT: Enable A20
 ;;INIT: Enable A20
  ;Registers modified in this INIT-END portion:
   ;AX (EAX)
    ;Registers with values to reuse in the next INIT--END:
     ;AH (cleared to 0)

 ;To see if the A20 line is already enabled,
 ;look at address (7C00h+510) from 2 different
 ;addresses, one in the first Megabyte and the second
 ;in the second Megabyte.
 ;
 ;Compare the 16-bit word from 0000:7DFEh (7DFEh physical)
 ;and from FFFFh:7E0Eh (107DFEh physical).
 ;
 ;They must be different. If they aren't, then the A20 line
 ;is disabled and we will have to enable it:
 ;;
 mov ax,[510]
 push word 0xFFFF
 pop es
 cmp word[es:7E0Eh],ax
 jne short .A20alreadyEnabled

 .5:
  in al,0x64 ;Enable A20 {4A}. Port 0x64 is the
             ;KBC port at the motherboard.

  test al,2  ;See if bit 2 at this port is set to 1, which means
             ;that the KBC is not ready.

  jnz .5     ;Repeat until bit 2 of port 0x64 is 0, which means
             ;that the KBC is ready for commands.

  mov al,0xD1 ;This command is to write the status byte.
              ;This is the so-called *WRITE OUTPUT PORT*.

  out 0x64,al ;Here we send it and it gets executed.

 .6:
  in al,0x64    ;Read the byte at port 0x64.
  and ax,byte 2 ;See if this bit is 0.
                ;NOTE: This will leave AL to 0 at once, which
                ;      will be used in the next
                ;      INIT-END block.

  jnz short .6  ;Repeat until the KBC is ready
                ;(bit 2 to 0)

  mov al,0xDF   ;Set the configuration bits to send.

  out 0x60,al   ;Send this parameter to the data port
                ;of the KBC. At this point is where the
                ;A20 line is enabled.

 .A20alreadyEnabled:

 ;;END:  Enable A20
 ;;END:  Enable A20
 ;;END:  Enable A20

___________________________________________________________________
___________________________________________________________________
___________________________________________________________________
___________________________________________________________________
5. Reenabling Interrupts in Real Mode

Once we have Unreal Mode set up along with the A20 line enabled and a good stack, we simply reenable interrupts to continue with the actual loading of a bootup program:

Code: Select all

;5. Reenable interrupts here:
;;

sti  ;Reenable interrupts for normal BIOS functioning