Alignment problem ?

[Brendan]

Hi,

I have attached a new version here:

Let's create a list of problems:

• There's no space for a BPB (needed for floppy) and no space for a partition table (needed for hard disk); and if it's not for booting from floppy and not for booting from hard disk (and not network or CD-ROM) then it's not useful for booting from anything.
• It's using 32-bit instructions (for no reason) and C calling conventions (for no reason), and the instruction selection is mostly bad (I'm guessing GCC generated without optimisations). All of this means that the code is probably taking up twice as much space as necessary (which is a severe problem when space is limited to less than 512 bytes).
• There's little error checking and no sane error messages:
  • When enabling A20 you need to check if A20 was actually enabled (or if the BIOS doesn't support the function and/or is buggy) and display an "ERROR: Failed to enable A20" error message.
  • Before enabling long mode you need to check if the CPU supports long mode and display an "ERROR: CPU is too old" error message if long mode isn't supported.
  • When loading data from disk you should have about 10 different error messages so the user can know what went wrong (and not just that something went wrong - e.g. was it a read error, was the disk removed, was the function not supported, were the parameters wrong, etc).
• Before you display anything you should make sure something can be displayed (e.g. make sure the video mode is 80*25 text mode and hasn't been set to something else by anything else). Note: if/when something is wrong and you can't boot, it's also a good idea to make PC speaker beep (in addition to a good descriptive error message) in case the system has no video card and/or no monitor.
• When your code is first started, DL contains the correct device number that you need to use. You ignore this and assume the boot device is device 0x80. Given that almost all boot managers support booting from other disk drives this is a mistake (e.g. your OS may be installed on a partition on device 0x81).
• The code to get a memory map looks very fragile to me. For example:
  • It's "hard-coded" for 24-byte entries (even though recent ACPI specs have extended it to 28 bytes)
  • Some BIOSs set EBX to zero to indicate that the entry is the last entry; and you don't check for that (which may result in an endless loop that fills up all RAM until SI (not ESI) overflows and wraps around, resulting in your code being overwritten and crashing)
  • You don't check any of the returned entries (e.g. one of the many common problems is that some BIOSs return entries with "area size = 0" that can/should be ignored).
• The "extended disk read" function is limited to 127 sectors on some systems. This means that if/when the kernel you're loading grows larger than 63.5 KiB you will be unable to load all of it safely. Also note that your kernel will probably grow to be several MiB, and therefore will not fit in the (less than 640 KiB) RAM that can be accessed while you're in real mode. Your boot code needs to load part of the kernel (e.g. 63.5 KiB) and switch to protected/long mode and copy that part elsewhere and switch back to real mode; then load/copy the next part; and so on until the entire kernel is loaded. It can't load the entire thing in one go.
• Your "mov $0x7bff,%esp" will cause every stack push/pop to be misaligned (it should be "mov $0x7C00,%esp")
• Your code to enable long mode assumes there's a valid PML4 at 0x2000. I can't see any code to create a valid PML4 at 0x2000 (or any of the tables it will require - PDPT, PD, PT, etc). Note that the kernel seems to be loaded at 0x8000 (and not 0x2000) so the necessary paging structures can't be hard-coded into the kernel either.

In addition to the problems above; there are things that are technically optional that' I'd want to consider:

• It's a good idea to check if the kernel is sane (and not corrupt/trashed) before starting it. For example, I typically include a CRC32 check (and an "ERROR: Kernel is corrupt" error message). I also tend to use compression (where boot code loads a compressed file and then decompresses it) as this improves boot times (e.g. half as much slow disk IO involved).
• Text modes suck (should've been banned/deprecated in 1990). You may want to consider setting a graphical video mode (e.g. using VBE) before passing control to the kernel (e.g. while you're in real mode and still able to use BIOS/VBE functions) so that your OS is able to use a graphical video mode during boot (and possibly after boot).
• UEFI is something that should be on your mind. Ideally, the boot code for BIOS and the boot code for UEFI would both start the same kernel and hide the differences between BIOS and UEFI from that kernel. This may include providing all information needed by the kernel (e.g. memory map, information describing video framebuffer, etc) in the same way. For example; it'd be nice to parse/sanitise the memory map given to you by the BIOS and transform it into a "standardised for your OS" format (where that "standardised for your OS" format is a super-set of the information that could be provided by both BIOS and UEFI, and not a sub-set).

Finally; if you do just some of the things I've mentioned above properly you're not going to be able to make it fit in 512 bytes - essentially; if you want boot code that's be better than "very bad" then your boot code must use more than 512 bytes. Fortunately there's no reason to limit the boot code to 512 bytes in the first place. For example, your boot code can be 50 KiB of code (where code in the first 512 bytes loads the remaining 49.5 KiB of itself).

Now; you'll notice I haven't found the cause of your symptoms (and all of the above may seem "off topic"). What I'm saying is that all of your code needs to be redesigned and then rewritten. Mostly, you have a choice: you can fix the current bug, and then redesign and rewrite everything; or you can forget about the current bug and then redesign and rewrite everything. Obviously, the latter is more efficient. Basically, there's no point finding or fixing the bug.


Cheers,

Brendan

[staringlizard]

Hi Brendan,

Thanks for all your comments !

First I want to say that I am actually just trying to see if it would be possible to make a bootloader using .codegcc16 hack that would work on most newer computers and that is loading a x86-64 kernel. I am really enjoying this and trying to get something up and running.

There's no space for a BPB (needed for floppy) and no space for a partition table (needed for hard disk); and if it's not for booting from floppy and not for booting from hard disk (and not network or CD-ROM) then it's not useful for booting from anything.

I do not need these things. I have no such constraint or "must have" that you are mentioning. I can boot from primary hard drive or usb drive which is actually the only thing I need. When did you boot from floppy or CD-ROM last time on your PC?

[*]It's using 32-bit instructions (for no reason) and C calling conventions (for no reason), and the instruction selection is mostly bad (I'm guessing GCC generated without optimisations). All of this means that the code is probably taking up twice as much space as necessary (which is a severe problem when space is limited to less than 512 bytes).

Code is using .codegcc16 hack with optimization -Os. I am amazed that I can get so much code into the bootloader this way and I enjoy trying this out. This is of course NOT the correct way to make a bootloader.

[*]There's little error checking and no sane error messages:

• When enabling A20 you need to check if A20 was actually enabled (or if the BIOS doesn't support the function and/or is buggy) and display an "ERROR: Failed to enable A20" error message.

It is not mandatory to have the error messages that you mention. A number will also work. I am checking if A20 is enabled, and if it is not, I call the BIOS function to enable it. If it does not work on some computer, then too bad. The bootloader is meant to work with new computers. If you have any tip what best will work with newer computers I would appreciate it. For now I stick to the BIOS function since I get the feeling that this is the best thing to use.

[*]Before enabling long mode you need to check if the CPU supports long mode and display an "ERROR: CPU is too old" error message if long mode isn't supported.

As I said. This bootloader is only meant to work with new computers. The 32-bit computer era is over I think, time to leave it behind.

[*]When loading data from disk you should have about 10 different error messages so the user can know what went wrong (and not just that something went wrong - e.g. was it a read error, was the disk removed, was the function not supported, were the parameters wrong, etc).

I somewhat agree with this. But it is not mandatory. My bootloader I posted here is very simple, it does not do a whole lot, just the bear minimum.

[*]Before you display anything you should make sure something can be displayed (e.g. make sure the video mode is 80*25 text mode and hasn't been set to something else by anything else). Note: if/when something is wrong and you can't boot, it's also a good idea to make PC speaker beep (in addition to a good descriptive error message) in case the system has no video card and/or no monitor.

Yes, I agree. But I do think that majority of today's modern computer it is not necessary to do this check ?!

[*]When your code is first started, DL contains the correct device number that you need to use. You ignore this and assume the boot device is device 0x80. Given that almost all boot managers support booting from other disk drives this is a mistake (e.g. your OS may be installed on a partition on device 0x81).

As I understand it, the boot device is always 0x80. And if it happens to be something else for a minuscule part of the computers out there, then I am still happy tbh.

[*]The code to get a memory map looks very fragile to me. For example:

• It's "hard-coded" for 24-byte entries (even though recent ACPI specs have extended it to 28 bytes)

That might be so, and I'm sure you know more than me in this field. I followed this OSDev page:
http://wiki.osdev.org/Detecting_Memory_(x86)

It is probably best to always store the list entries as 24 byte quantities

[*]Some BIOSs set EBX to zero to indicate that the entry is the last entry; and you don't check for that (which may result in an endless loop that fills up all RAM until SI (not ESI) overflows and wraps around, resulting in your code being overwritten and crashing)

I do check for this.

Code: Select all

    7d83:	cd 15                	int    $0x15
    7d85:	72 18                	jb     7d9f <memory_discover_exit>
    7d87:	66 89 da             	mov    %ebx,%edx
    7d8a:	66 a1 f4 7d          	mov    0x7df4,%eax
    7d8e:	66 83 c6 18          	add    $0x18,%esi
    7d92:	66 89 36 f0 7d       	mov    %esi,0x7df0
    7d97:	67 fe 00             	incb   (%eax)
    7d9a:	66 85 d2             	test   %edx,%edx
    7d9d:	75 ca                	jne    7d69 <memory_discover+0x19>

[*]You don't check any of the returned entries (e.g. one of the many common problems is that some BIOSs return entries with "area size = 0" that can/should be ignored).[/list]

I don't see how this is a problem. I can easily handle these kind of entries in kernel. Could you elaborate?

[*]The "extended disk read" function is limited to 127 sectors on some systems. This means that if/when the kernel you're loading grows larger than 63.5 KiB you will be unable to load all of it safely. Also note that your kernel will probably grow to be several MiB, and therefore will not fit in the (less than 640 KiB) RAM that can be accessed while you're in real mode. Your boot code needs to load part of the kernel (e.g. 63.5 KiB) and switch to protected/long mode and copy that part elsewhere and switch back to real mode; then load/copy the next part; and so on until the entire kernel is loaded. It can't load the entire thing in one go.

Yes, I am aware of this. But as you write above, there is a solution that would probably work well.

[*]Your "mov $0x7bff,%esp" will cause every stack push/pop to be misaligned (it should be "mov $0x7C00,%esp")

Yes, you are correct! That is much better, thanks for the tip.

[*]Your code to enable long mode assumes there's a valid PML4 at 0x2000. I can't see any code to create a valid PML4 at 0x2000 (or any of the tables it will require - PDPT, PD, PT, etc). Note that the kernel seems to be loaded at 0x8000 (and not 0x2000) so the necessary paging structures can't be hard-coded into the kernel either.[/list]

If I did not have this code then it would obviously not work....

Code: Select all

    7c32:	66 b8 00 20 00 00    	mov    $0x2000,%eax
    7c38:	0f 22 d8             	mov    %eax,%cr3
    7c3b:	67 66 c7 00 03 30 00 	movl   $0x3003,(%eax)
    7c42:	00 
    7c43:	b8 00 30             	mov    $0x3000,%ax
    7c46:	67 66 c7 00 03 40 00 	movl   $0x4003,(%eax)
    7c4d:	00 
    7c4e:	b8 00 40             	mov    $0x4000,%ax
    7c51:	67 66 c7 00 03 50 00 	movl   $0x5003,(%eax)
    7c58:	00 
    7c59:	b8 00 50             	mov    $0x5000,%ax
    7c5c:	66 ba 03 00 00 00    	mov    $0x3,%edx

PML4 is on address 0x2000.
PDPT is on address 0x3000.
PDT is on address 0x4000.

The page table then starts at 0x5000 and is filled in like so:

Code: Select all

    7c62:	67 66 89 10          	mov    %edx,(%eax)
    7c66:	83 c0 08             	add    $0x8,%ax
    7c69:	66 81 c2 00 10 00 00 	add    $0x1000,%edx
    7c70:	66 81 fa 03 00 20 00 	cmp    $0x200003,%edx
    7c77:	75 e9                	jne    7c62 <sboot_switch_to_64bit+0x36>

In addition to the problems above; there are things that are technically optional that' I'd want to consider:

• It's a good idea to check if the kernel is sane (and not corrupt/trashed) before starting it. For example, I typically include a CRC32 check (and an "ERROR: Kernel is corrupt" error message). I also tend to use compression (where boot code loads a compressed file and then decompresses it) as this improves boot times (e.g. half as much slow disk IO involved).

Yes, I agree. When I do something real serious I will definitely consider this.

[*]Text modes suck (should've been banned/deprecated in 1990). You may want to consider setting a graphical video mode (e.g. using VBE) before passing control to the kernel (e.g. while you're in real mode and still able to use BIOS/VBE functions) so that your OS is able to use a graphical video mode during boot (and possibly after boot).

Yes, that would be neat. But I like to keep things simple.

[*]UEFI is something that should be on your mind. Ideally, the boot code for BIOS and the boot code for UEFI would both start the same kernel and hide the differences between BIOS and UEFI from that kernel. This may include providing all information needed by the kernel (e.g. memory map, information describing video framebuffer, etc) in the same way. For example; it'd be nice to parse/sanitise the memory map given to you by the BIOS and transform it into a "standardised for your OS" format (where that "standardised for your OS" format is a super-set of the information that could be provided by both BIOS and UEFI, and not a sub-set).[/list]

Normally with my spare time projects I dismiss these kinds of complex overworked specification that exceeds 1000 pages. But I will take a look.

Finally; if you do just some of the things I've mentioned above properly you're not going to be able to make it fit in 512 bytes - essentially; if you want boot code that's be better than "very bad" then your boot code must use more than 512 bytes. Fortunately there's no reason to limit the boot code to 512 bytes in the first place. For example, your boot code can be 50 KiB of code (where code in the first 512 bytes loads the remaining 49.5 KiB of itself).

I totally agree with you on this. As I said earlier I am experimenting a bit for my own pleasure.

Now; you'll notice I haven't found the cause of your symptoms (and all of the above may seem "off topic"). What I'm saying is that all of your code needs to be redesigned and then rewritten. Mostly, you have a choice: you can fix the current bug, and then redesign and rewrite everything; or you can forget about the current bug and then redesign and rewrite everything. Obviously, the latter is more efficient. Basically, there's no point finding or fixing the bug.

No, I disagree. The bootloader I have written is very small and pretty simple. That is the big benefit of not including all the things that you have commented on and that you feel is essential. I for one, like to find these kinds of bugs and understand them. I also think at least some of the people on this forum is like me. That is the reason for why I posted this here.

Again, thanks for all your comments, I appreciate it.

Regards,
StaringL

[staringlizard]

Then a code entry:
00 00 00 00 00 98 20 00

Then the data entry:
00 00 00 00 00 90 00 00

These two descriptors define segments with 0 limit. Also, I don't see you setting the D bit on code segment: is this intentional, i.e. are you going to execute 16-bit protected code afterwards?

Hi Icee,
Thanks for your post.

I followed these tutorials:
http://wiki.osdev.org/User:Stephanvansc ... _Long_Mode
http://wiki.osdev.org/Entering_Long_Mode_Directly

I think the descriptors are pretty much right for x86-64.

Regards,
StaringL

[Combuster]

Code is using .codegcc16 hack with optimization -Os. I am amazed that I can get so much code into the bootloader this way and I enjoy trying this out.

I'm missing -fomit-frame-pointer

[Icee]

I think the descriptors are pretty much right for x86-64.

Sorry, my bad. The values are fine for x86-64, indeed.

[staringlizard]

Code is using .codegcc16 hack with optimization -Os. I am amazed that I can get so much code into the bootloader this way and I enjoy trying this out.

I'm missing -fomit-frame-pointer

Holy crap, you just saved me 64 bytes!!

[Brendan]

Hi,

First I want to say that I am actually just trying to see if it would be possible to make a bootloader using .codegcc16 hack that would work on most newer computers and that is loading a x86-64 kernel. I am really enjoying this and trying to get something up and running.

Ah, OK - that explains some things.

There's no space for a BPB (needed for floppy) and no space for a partition table (needed for hard disk); and if it's not for booting from floppy and not for booting from hard disk (and not network or CD-ROM) then it's not useful for booting from anything.

I do not need these things. I have no such constraint or "must have" that you are mentioning. I can boot from primary hard drive or usb drive which is actually the only thing I need. When did you boot from floppy or CD-ROM last time on your PC?

Sure, you can boot from the primary hard drive's MBR (and basically treat the entire hard drive like one huge unpartitioned floppy drive). However; all sane OSs will see your "lack of partition table" and assume the disk is unformatted and/or contains nothing more than garbage and/or misinterpret trash and think partitions exist where they don't.

Also, don't be too surprised if the firmware in your computer checks the hard disk's partition tables (to determine if it should boot it as UEFI or boot it as BIOS), where not having a partition table causes undefined behaviour (not necessarily excluding the possibility of crashing when certain data is where the partition table should be, because the firmware thought it was GPT/UEFI and not MBR/BIOS and got confused trying to find the EFI system partition).

[*]There's little error checking and no sane error messages:

• When enabling A20 you need to check if A20 was actually enabled (or if the BIOS doesn't support the function and/or is buggy) and display an "ERROR: Failed to enable A20" error message.

It is not mandatory to have the error messages that you mention. A number will also work. I am checking if A20 is enabled, and if it is not, I call the BIOS function to enable it. If it does not work on some computer, then too bad. The bootloader is meant to work with new computers. If you have any tip what best will work with newer computers I would appreciate it. For now I stick to the BIOS function since I get the feeling that this is the best thing to use.

Sadly, you're right - there's no mandatory requirement for your code to be usable or useful. A number will "work", if you're willing to assume that "work" means "frustrate and confuse the unfortunate end user at the worst possible time" (e.g. when the computer they'd use to find out what your inadequate error code actually means can't be used because it won't boot).

[*]Before enabling long mode you need to check if the CPU supports long mode and display an "ERROR: CPU is too old" error message if long mode isn't supported.

As I said. This bootloader is only meant to work with new computers. The 32-bit computer era is over I think, time to leave it behind.

32-bit systems aren't dead - they're being used in small/mobile systems (everything from phones and touchpads to wearable computers). Intel will release their next 32-bit CPU SoC soon. However, this is beside the point.

The point is that people don't go and buy themselves a fancy new computer just to try out an alternative "work in progress" OS like yours. They grab an old computer out of their closet (that they replaced 10 years ago with the computer they're currently using) and wonder why your OS won't boot. Of course then they just assume you're incompetent because your error message was worthless; and the fact that the problem was caused by the user not reading the OS's instructions won't prevent this in the slightest (because let's be honest, no user ever reads any instructions and it's foolish for developers to assume they might).

[*]Before you display anything you should make sure something can be displayed (e.g. make sure the video mode is 80*25 text mode and hasn't been set to something else by anything else). Note: if/when something is wrong and you can't boot, it's also a good idea to make PC speaker beep (in addition to a good descriptive error message) in case the system has no video card and/or no monitor.

Yes, I agree. But I do think that majority of today's modern computer it is not necessary to do this check ?!

An OS is complex. If there are 100 different things that all "should work 95% of the time", then the chance of all 100 things working at the same time is "0.95 * 0.95 * 0.95 * 0.95 * .... * 0.95" (or 0.95^100); which works out to approximately zero chance of the OS working.

Basically; if it is not necessary to do this check on majority of today's modern computers; then it's definitely necessary to do the check.

[*]When your code is first started, DL contains the correct device number that you need to use. You ignore this and assume the boot device is device 0x80. Given that almost all boot managers support booting from other disk drives this is a mistake (e.g. your OS may be installed on a partition on device 0x81).

As I understand it, the boot device is always 0x80. And if it happens to be something else for a minuscule part of the computers out there, then I am still happy tbh.

Someone will install your OS on a USB flash stick and try to boot it. In this case I doubt that the boot device will be 0x80 on any computer they try.

[*]The code to get a memory map looks very fragile to me. For example:

• It's "hard-coded" for 24-byte entries (even though recent ACPI specs have extended it to 28 bytes)

That might be so, and I'm sure you know more than me in this field. I followed this OSDev page:
http://wiki.osdev.org/Detecting_Memory_(x86)

No, I was wrong. It was 20 bytes and got extended to 24 bytes (not 24 bytes and extended to 28 bytes). What this means is that if the BIOS only returns 20 bytes (which is still quite common) it won't set the last 4 bytes to anything at all, and your kernel won't know those last 4 bytes aren't meaningful.

[*]You don't check any of the returned entries (e.g. one of the many common problems is that some BIOSs return entries with "area size = 0" that can/should be ignored).[/list]

I don't see how this is a problem. I can easily handle these kind of entries in kernel. Could you elaborate?

I'd just prefer to give the kernel a nice clean/correct/complete memory map (rather than potentially unusable/unchecked garbage), and various checks (like skipping entries with "area size = 0") are so easy to do that it's sad when they aren't done.

[*]Your code to enable long mode assumes there's a valid PML4 at 0x2000. I can't see any code to create a valid PML4 at 0x2000 (or any of the tables it will require - PDPT, PD, PT, etc). Note that the kernel seems to be loaded at 0x8000 (and not 0x2000) so the necessary paging structures can't be hard-coded into the kernel either.[/list]

If I did not have this code then it would obviously not work....

Heh - I didn't assume it worked, and was looking for a "create_virtual_address_space" function (rather than a few writes in one place, and code to pre-fill the area with zeros in another place).


Cheers,

Brendan

[staringlizard]

Hi again Brendan,

Sure, you can boot from the primary hard drive's MBR (and basically treat the entire hard drive like one huge unpartitioned floppy drive). However; all sane OSs will see your "lack of partition table" and assume the disk is unformatted and/or contains nothing more than garbage and/or misinterpret trash and think partitions exist where they don't.

Also, don't be too surprised if the firmware in your computer checks the hard disk's partition tables (to determine if it should boot it as UEFI or boot it as BIOS), where not having a partition table causes undefined behaviour (not necessarily excluding the possibility of crashing when certain data is where the partition table should be, because the firmware thought it was GPT/UEFI and not MBR/BIOS and got confused trying to find the EFI system partition).

Yes I would not be surprised about this. In my hack bootloader I will consider anything other than loading the mbr to ram and execute it as a bios/firmware error.

Sadly, you're right - there's no mandatory requirement for your code to be usable or useful. A number will "work", if you're willing to assume that "work" means "frustrate and confuse the unfortunate end user at the worst possible time" (e.g. when the computer they'd use to find out what your inadequate error code actually means can't be used because it won't boot).

I cannot agree with you more. If I would have a bunch of people using my bootloader I would never do stupid error messages like this.

32-bit systems aren't dead - they're being used in small/mobile systems (everything from phones and touchpads to wearable computers). Intel will release their next 32-bit CPU SoC soon. However, this is beside the point.

The point is that people don't go and buy themselves a fancy new computer just to try out an alternative "work in progress" OS like yours. They grab an old computer out of their closet (that they replaced 10 years ago with the computer they're currently using) and wonder why your OS won't boot. Of course then they just assume you're incompetent because your error message was worthless; and the fact that the problem was caused by the user not reading the OS's instructions won't prevent this in the slightest (because let's be honest, no user ever reads any instructions and it's foolish for developers to assume they might).

As I said, computers (e.g PC) are 64 bit today (e.g. 64 bit CPU is a requirement for the bootloader and the kernel). I'm am not developing a bootloader for a mobile phone.

An OS is complex. If there are 100 different things that all "should work 95% of the time", then the chance of all 100 things working at the same time is "0.95 * 0.95 * 0.95 * 0.95 * .... * 0.95" (or 0.95^100); which works out to approximately zero chance of the OS working.

Basically; if it is not necessary to do this check on majority of today's modern computers; then it's definitely necessary to do the check.

You are too negative, it will work, trust me

Someone will install your OS on a USB flash stick and try to boot it. In this case I doubt that the boot device will be 0x80 on any computer they try.

Sure, I can guarantee it. But as I already wrote in my previous post...

Thanks for the comments Brendan !

Regards,
StaringL