OSDev.org

Hi, this is my first post, as I'm starting to make my own operating system. The Wiki part of this site already helped me a lot !
I'm still at the beginning of the project, stuck at the bootloader level.
This bootloader is very basic ; its final version will read sector 2 of the initial disk (a USB drive), place it in the RAM, go into 32-bit protected mode, and far jump to the memory where the 2nd sector was loaded.
The problem I'm facing only occurs on my real-life Toshiba Satellite Pro NB10-A, and I don't have any other real-life ressource to try my bootloader on.
Here is my current bootloader : It only loads sector 2 and print its first byte as an ascii character, without jumpping to it. Here is the source for gdt.asm (Global Descriptor Table) Here is a description of the problem:
A. Loading in qemu-system-i386 :
This code works flawlessly : the "X" (black on white) is displayed in the top-left corner.

B. Loading in my Toshiba :
The top-left corner shows an empty white box (null byte interpreted as ascii, shown as whitespace).
While debugging with a custom print function displaying hexadecimal, I realised that when reading sector 1, it works well (a "3" is displayed at the top left, ascii 51), but when reading sector 2, the buffer is full of zeros. The carry register indicates normal behaviour, and the number in "al", the number of sectors actually read, is 1 (correct). But, the buffer is actually full of zeros. I tried different addresses for the buffer, I checked a lot of registers, tried some solutions I found in the web. None solved the problem, but what's worse is that I don't understand where it could come from : I specified correct parameters of the interrupt 0x13,2, so why does the real environment won't let me access this sector ? Grub loads flawlessly on this device, so I'm the one missing something.
Also, This USB drive does not contain any partitionning system at this state of the project.
To compile my project :
real-life (z my USB drive) : ..\..\fasm\FASM.EXE ath_os.asm ath_os.bin && dd if=ath_os.bin of=\\.\z: bs=512 count=2
qemu : ..\..\fasm\FASM.EXE ath_os.asm ath_os.bin && "C:\Program Files (x86)\qemu\qemu-system-i386w.exe" -hda ath_os.bin
I also tried putting the .bin to my USB drive with rufus and with dd on linux.

I'm attaching the full code including disk_load, print, and debug functions.
Also I'm a beginner in assembly but I'll do my best to understand complex answers.

The simplest program reproducing the error : Same problem, shorter & simpler code.

After a quick look-over, I didn't see anything in error, however, being you are reading from a USB thumb drive using the firmware's emulation, this gives me the idea that this is the problem.

Depending on the firmware's emulation, CHS address of 2 (C = 0, H = 0, S = 2) may not be LBA 1 as you might think it is. Some BIOS' start at a given partition address range, others start at LBA 0.

Are you sure that LBA 1 (the second physical sector of the device) actually has something other than zeros in it? I am not familiar with DD so I can't tell you if your writing to the USB drive is correct.

However, I would be looking at how you are writing the second sector to the USB device to make sure it actually is the second sector.

There are many problems with BIOSes and their emulation of USB boot devices... Some will only emulate a hard drive, some will only emulate partitions on that device as hard drive(s), some will think it is a floppy and emulate as a floppy, some with think it is a hard drive even though you have specified floppy. Others will read the first sector and expect a valid (FAT) BPB and use that as the parameters.

For example, if you do not have a BPB at LBA 0, but the firmware read it as one anyway, it might have read that there is only one sector per track and has now moved to the second track of the device (i.e.: move so many sectors from LBA 0) to read that second sector. i.e.: Let's say that the read data from the first part of the first sector has the follow: even though you don't have a valid BPB there, it might have read that as so. Now, since it thinks that there are 14 heads, but only 1 sector per track, there actually isn't a CHS of address of 0/0/2, since there is only 1 sector per track. It will also think that the very next sector (LBA 1) is actually 14 sectors away, depending on whether you are the 'writer' or the 'reader'.

If you continue to use a USB device as your booting (test) device, I strongly suggest using a valid (FAT) BPB whether you have a FAT formatted device or not.

In my research, I have found that to be sure your USB device will boot on the most firmware emulation possible, have a small amount of sectors reserved at the beginning of your USB device for a valid BPB and MBR that simply loads another sector (starting at LBA 1 if you want) and jumps to it. Then place your boot code at that LBA. The BPB at LBA 0 specifies the type of disk you want emulated.

Ben
- http://www.fysnet.net/osdesign_book_series.htm

Thanks a lot for your time, I will try a valid BPB tomorrow

Ben these days, the lack of a BPB is a problem on a lot of firmware booting USB as FDD (and specifically FDD) because many BIOSes blindly write the USB's drive geometry into the BPB area (after reading the boot sector to 0x07c00) prior to transferring control to physical address 0x07c00. If you don't at least have the space reserved for the BPB you end up having these BIOSes overwrite code and data near the beginning of the bootloader. This has actually been a recurring theme in Stackoverflow questions for nearly 2 years. I even wrote about the problem in this Stackoverflow Answer (which has more or less become the canonical response on SO)

Without sounding rude, that's pretty much what I said, but in a different way :-)

Anyway, for the OP, it most likely is not your code, but more likely the emulated disk read service. Place a valid BPB at LBA 0 and you should be fine.

If you don't want a FAT file system, that's fine, start at LBA 1 with whatever file system you want and let LBA 0 jump to it via a standard MBR.

Ben
- http://www.fysnet.net/osdesign_book_series.htm

Your comment was more about drive geometry values in the BPB (for a case where none exists) and having subsequent disk reads fail. What I was suggesting can affect code whether it is related to disk reads or something else -as long as that code falls in the area that a BPB may occupy and especially where the drive geometry related fields sit. You could have your bootloader simply using BIOS routines to print a string to the display and have that fail when run as USB with FDD emulation.

MichaelPetch wrote:Your comment was more about drive geometry values in the BPB (for a case where none exists) and having subsequent disk reads fail. What I was suggesting can affect code whether it is related to disk reads or something else -as long as that code falls in the area that a BPB may occupy and especially where the drive geometry related fields sit. You could have your bootloader simply using BIOS routines to print a string to the display and have that fail when run as USB with FDD emulation.

Absolutely, you are correct. I did fail to mention that.

Either way, I am guessing this will fix the problem the OP is having.

Thanks,
Ben
- http://www.fysnet.net/osdesign_book_series.htm

Hello everyone,
I found various examples of BPBs online and I ended up uploading this to my USB drive: I see no change... Before I try to put an MBR, I would like to understand, how is an MBR different from this apart from the partition table ? If the same code is executed, why does it matter ?
Also, Yes I'm sure the sector 2 is written correctly to the USB drive because I dumped it with dd and inspected it : the "X" is there as expected.
One thing I'm starting to imagine is that the drive label specified to DD might be a... partition ? even if Windows tells me that it's taking the space of the whole USB drive, it might be wrong... I'll try uploading the code on linux again, with the BPB this time.

As for DD and Windows, again, I have not used DD so cannot be sure, but Windows usually does deal with partitions than with devices, unless you develop DD to use devices.

As for why a BPB and a MBR? In theory, you can do whatever you wish. Have your code just as you show with a "dummy" BPB and go from there. However, a MBR is the traditional first sector of a media device and you have now noticed what happens when you don't use traditional techniques, i.e.: the traditional BPB.

It is traditional to use a MBR at LBA 0 that has a single partition entry that points to either an extended partition table or a single partition at LBA 63.

However, to bypass the issues I and MichealPetch (more clearly) explained, you need to have a BPB at LBA 0 on a USB bootable device. Some file systems will allow this by simply skipping the number of bytes needed for the BPB. Some will not. For example, the Ext2 filesystem only allows a single block (usually 1024 bytes) before the super block. This 1024-byte block is valuable real estate when a bootable Ext2 is needed (not mentioning GRUB or other boot techniques).

Personally, I don't want to "have to" put the BPB in my boot code just because someone else was too lazy to create a decent emulation of the device in the first place. Therefore, I use a specific 512-byte un-traditional MBR with a BPB all pointing to LBA 1 for the start of the bootable partition. This 512-byte MBR/BPB can be exactly the same for any USB bootable device and installed file system. Simply binary write it to LBA 0 and then my boot partition to LBA 1. Period.

Just my opinion,

Ben
- http://www.fysnet.net/osdesign_book_series.htm

On Windows your DD is likely not writing to the first sector of the drive when you use the option of=\\.\z:. This is likely writing to the first sector of the partition Z: resides on. Get the latest version of Chrysocome DD for Windows and use the od= opttion where you simply specify the drive letter for the USB (ie od=z:).You must have administrator privileges. I have written more about the issue of Windows and using DD in this Stackoverflow answer (see the second section of the answer).If using DD with administrator privileges you keep getting an access denied error you will want to clean the volume using these instructions.

I just checked, and this is still not solved by uploading through Linux -> dd.
I'm using Chrysocome's DD already
I also tried on another laptop, an Asus EEE-PC 900A, and it produces the same behaviour...
An MBR would try to read the disk the same way, I guess just adding a partition table from byte 446 wouldn't solve the problem...
Current things tried:
- Sectors successfully uploaded to the 1st sectors of the USB drive [Done]
- Putting a BPB at the beginning (byte 3 -> ...) [Done]
- Test on another laptop [Done, same result]
- MBR [don't see why it wouldn't produce the same result]

But now that I use DD with the "od" option, my toshiba laptop seems to ignore the drive, as if it was not bootable xD

AlfredTH wrote:But now that I use DD with the "od" option, my toshiba laptop seems to ignore the drive, as if it was not bootable xD

One UEFI CSM (InsydeH2O?) is particularly bad and will only boot USB drives with a valid MBR and partition table. Sounds like whatever you're using now isn't valid.

I try to get this out to all new members, to get everyone started on the same footing. I hope this helps. I usually send it by PM, but I try to post it to a thread periodically so that anyone who doesn't check their PMs might see it anyway.
----------------------------------

The first thing I want to say is this: if you aren't already using version control for all software projects you are working on, drop everything and start to do that now. Set up a VCS such as Git, Subversion, Mercurial, Bazaar, or what have you - which you use is less important than the fact that you need to use it. Similarly, setting up your repos on an offsite host such as Gitlab, Github, Sourceforge, CloudForge, or BitBucket should be the very first thing you do whenever you start a new project, no matter how large or small it is.

If nothing else, it makes it easy to share your code with us on the forum, as you can just post a link, rather than pasting oodles and oodles of code into a post.

Once you have that out of the way (if you didn't already), you can start to consider the OS specific issues.

If you haven't already, I would strongly advise you to read the introductory material in the wiki:

• Getting Started
• How To Ask Questions
• FAQ
• Required Knowledge
• Beginner Mistakes

After this, go through the material on the practical aspects of
running an OS-dev project:

• What order should I make things in
• Code Management
• How kernel, compiler, and C library work together
• Using Programming Languages other than C
• Assembly Language

I strongly suggest that you read through these pages in detail, along with the appropriate ones to follow, before doing any actual development. These pages should ensure that you have at least the basic groundwork for learning about OS dev covered.

This brings you to your first big decision: which platform, or platforms, to target. Commonly options include:

• x86 - the CPU architecture of the stock PC desktops and laptops, and the system which remains the 'default' for OS dev on this group. However, it is notoriously quirky, especially regarding Memory Segmentation, and the sharp divisions between 16-bit Real Mode, 16-bit and 32-bit Protected Modes, and 64-bit Long Mode.
• ARM - a RISC architecture widely used on mobile devices and for 'Internet of Things' and 'Maker' equipment, including the popular Raspberry Pi and Beagleboard single board computers. While it is generally seen as easier to work with that x86, most notably in the much less severe differences in between the 32-bit and 64-bit modes and the lack of memory segmentation, the wiki and other resources don't cover it nearly as well (though this is changing over time as it becomes more commonly targeted).
• MIPS, another RISC design which is slightly older than ARM. It is one of the first RISC design to come out, being part of the reason the idea caught on, and is even simpler than ARM in terms of programming, though a bit tedious when it comes to assembly programming. While it was widely used in workstations and game consoles in the 1990s, it has declined significantly due to mismanagement by the owners of the design, and is mostly seen in devices such as routers. There are a handful of System on Chip single-board computers that use it, such as the Creator Board and the Onion Omega2, and manufacturers in both China and Russia have licensed the ISA with the idea of breaking their dependence on Intel. Finding good information on the instruction set is easy, as it is widely used in courses on assembly language and computer architecture and there are several emulators that run MIPS code, but finding usable information on the actual hardware systems using it is often difficult at best.
• RISC-V is an up and coming open source hardware ISA, but so far is Not Ready For Prime Time. This may change in the next few years, though.

You then need to decide which Language to use for the kernel. For most OS-Developers this means knowing and using C; while other languages can be used, it is important to know how to read C code, even if you don't use C, as most OS examples are written in it. You will also need to know at least some assembly language for the target platform, as there are always parts of the kernel and the device drivers which cannot be done in high-level languages.

You further need to choose the compiler, assembler, linker, build tool, and support utilities to use - what is called the 'toolchain' for your OS. For most platforms, there aren't many to choose from, and the obvious choice would be GCC and the Binutils toolchain due to their ubiquity. However, on the Intel x86 platform, it isn't as simple, as there are several other toolchains which are in widespread use for it, the most notable being the Microsoft one - a very familiar one to Windows programmers, but one which presents problems in OSDev. The biggest issue with Visual Studio, and with proprietary toolchains in general, is that using it rules out the possibility of your OS being "self-hosting" - that is to say, being able to develop your OS in the OS itself, something most OSdevs do want to eventually be able to do. The fact that Porting GCC to your OS is feasible, whereas porting proprietary x86 toolchains isn't, is a big factor in the use Binutils and GCC, as it their deep connection to Linux and other Unix derivatives.

Regardless of the high-level language you use for OS dev (if any), you will still need to use assembly language, which means choosing an assembler. If you are using Binutils and GCC, the obvious choice would be GAS, but for x86 especially, there are other assemblers which many OSdevs prefer, such as Netwide Assembler (NASM) and Flat Assembler (FASM).

The important thing here is that assembly language syntax varies more among the x86 assemblers than it does for most other platforms, with the biggest difference being that between the Intel syntax used in the majority of x86 assemblers, and the AT&T syntax used in GAS. You can see an overview of the differences on the somewhat misnamed wiki page Opcode syntax. While it is possible to coax GAS to use the Intel syntax using the .intel_syntax noprefix directive, the opposite is generally not true for Intel-based assemblers such as NASM, and even with that directive, GAS is still quite different from other x86 assemblers in other regards.

It is still important to understand that the various Intel syntax assemblers - NASM, FASM, and YASM among others - have differences in how they handle indexing, in the directives they use, and in their support for features such as macros and defining data structures. While most of these follow the general syntax of Microsoft Assembler (MASM), they all diverge from it in various ways.

Once you know which platform you are targeting, and the toolchain you want to use, you need to understand them. You should read up on the core technologies for the platform. Assuming that you are targeting the PC architecture, this would include:

• Real Mode, especially the section on memory addressing, and Segmentation
• Protected mode
• Long Mode
• Learning 80x86 Assembly, especially real mode assembly
• Memory Map, Detecting Memory and A20 Line
• Interrupts
• BIOS, and Boot Sequence

This leads to the next big decision: which Bootloader to use. There are a number of different standard bootloaders for x86, with the most prominent being GRUB. We strong recommend against Rolling Your Own Bootloader, but it is an option as well.

You need to consider what kind of File System to use. Common ones used when starting out in OS dev include:

• FAT
• SFS
• Ext2fs

We generally don't recommend designing your own, but as with boot loaders, it is a possibility as well.

While this is a lot of reading, it simply reflects the due diligence that any OS-devver needs to go through in order to get anywhere. OS development, even as a simple project, is not amenable to the Stack Overflow cut-and-paste model of software development; you really need to understand a fair amount of the concepts and principles before writing any code, and the examples given in tutorials and forum posts generally are exactly that. Copying an existing code snippet without at least a basic idea of what it is doing simply won't do. While learning itself is an iterative process - you learn one thing, try it out, see what worked and what didn't, read some more, etc. - in this case a basic foundation is needed at the start. Without a solid understanding of at least some of the core ideas before starting, you simply can't get very far in OS dev.

Hopefully, this won't scare you off; it isn't nearly as bad as it sounds. It just takes a lot of patience and a bit of effort, a little at a time.

It all depends on how your BIOS is booting USB media.If it is using HDD emulation then you'd almost certainly need a partition table. I'd be curious what happens if you take this exact file and assemble it and use DD with the od=z: to write it to the first sector. This is a simple bootloader with a partition table but the active partition is one that starts at the first sector of the drive (LBA=0, CHS=0,0,1). I'm curious if prints 'Hello,world!' or not?