Need assist with multi-stage bootloader...

[ATC]

omg can sum1 PLZ give me codes to mak new verzin of wind0ws!!!111!!one!!!1!

No... lol, this isn't one of those posts... I swear! Well,

Very sorry to pester anyone with might seem like a totally trivial thing. But I've gone through absolute hell trying to find even tiny scraps of information. I've been an application programmer for years, but system programming is relatively foreign to me. All I've been able to succeed with so far is getting a simple bootloader to boot in real mode and write to the screen. Why is this so difficult? I program in Visual Studio Pro 08 on Windows. Just searching on the internet returns myriads of results for Linux; and nothing for Windows/VS. I FINALLY, after 3 days of no sleep, found a website with some solid tutorials for using VS. All was well, until I hit the "deadly" snag: Floppy disks.

Not only do I NOT want to use floppy disks, I don't have any, don't have a drive and lack any facilities to use them even if I wanted to. So using ISO images or CD/DVDs is a requirement. That might sound strange, but there is a reason for it all. This project is some early "academic" research I'm doing to plan a future project properly (and no, it won't be a desktop system). I'm new to ASM and being in ring 0 altogether, really. The concepts make sense, and I understand the general scheme of things. But I'm totally snagged when it comes to transferring control out of that little 512b bootloader and trying to go beyond.

When I make an ISO image, this is what I can do:

1) Put the boot.bin into the bootsector (this works on its own just fine)
2) Add other files to the main file list

I have no idea how memory is partitioned/segmented in an ISO, and how the heck I get to it to invoke the entry point of another executable. All examples I come across are using floppies, which I cannot use. I guess this really throws everything I've learned so far out the Window. I don't think this is FAT-12, and I believe the ISO memory is contiguous as well. I've searched endlessly, and it's been totally fruitless.

I'm not trying to beg people to do all the work for me, but I don't think searching anymore is going to do any good. I'm exhausted and frustrated, and haven't slept in the last 26+ hours. It's wearing me down quickly. I hope someone can point me in the right direction or pull me out of this rut! I want to focus on the implementation of the system, and not this.

[Selenic]

Just searching on the internet returns myriads of results for Linux; and nothing for Windows/VS

That's what Cygwin is for

Not only do I NOT want to use floppy disks, I don't have any, don't have a drive and lack any facilities to use them even if I wanted to. So using ISO images or CD/DVDs is a requirement.

VMs like Bochs support using images instead of real floppy drives, so your best bet is to use that until you at least have a working CD driver and ISO9660 support. Or you can start with a CD image (like this). Either way, a VM is much better than real hardware - you might need a hundred rebuilds before you get a finished version, and you don't want to waste a hundred CDs!

I have no idea how memory is partitioned/segmented in an ISO,

It's not memory, but disk space.
Anyway, see the ISO9660 filesystem - that's the one that's used for CDs (dunno about bootable DVDs - do they have two filesystems, or do modern BIOSes support UDF booting?)

and how the heck I get to it to invoke the entry point of another executable.

That can be quite a complicated one; you can avoid it for the kernel proper by using GRUB, and leave apps for a while.
I think part of it is that you have to learn to treat code like data - see relocation, for example (in a linker). You have to go in and edit bits of the code to make it point to the correct other code or data.

[matematik]

If I understood correctly you want to read files from CD to memory.
You can read CD using int 13h extended read:
http://www.ctyme.com/intr/rb-0708.htm
CDs usually have ISO 9660 filesystem. You can get basic info about ISO 9660 on wikipedia (http://en.wikipedia.org/wiki/ISO_9660), but you will probably need to find other spec to be able to parse it.

[ATC]

Thank you for the brief and informative answer.

I'm using two emulators: Virtual Box and Virtual PC. No way I'm trying to boot off my real hardware and screw up my working OS! And also no way I'm going through a billion shutdowns either. The best thing about the emulators though is that even if I destroy something critical, I can delete the hard drive and try again.

But yes, I'm sure the emulator is "pretending" the ISO image it captures is a real CD/DVD. I just can't find the proper info on parsing it and getting to stage 2. If I can get past this and get a basic kernel going, I'll be ok. At least I'm comfortable with C syntax and I know how to implement things properly. :S

EDIT: Also, if anyone can link me to some proper assembly language books for modern x86/64 processors, I will GREATLY appreciate it. I'm scared to spend hundreds of bucks and end up with outdated material (and most books look ancient). Any other books worth reading would be great too. I'm getting far too bogged down with ASM, and I guess I need to bite the bullet and get after it.

[Selenic]

But yes, I'm sure the emulator is "pretending" the ISO image it captures is a real CD/DVD. I just can't find the proper info on parsing it and getting to stage 2.

Yeah, looking at the article on the wiki about ISO9660, it's not too clear on how exactly it is structured. Best bet is to cross-reference it with some real code (eg, from cdrtools) which deals with images to get an idea of what to look for. (edit: that or try to dig up the actual specification(s) and see if that's any help)

Also, if anyone can link me to some proper assembly language books for modern x86/64 processors, I will GREATLY appreciate it.

The Intel and AMD manuals are pretty good; I believe you can get both from their websites. They cover everything - paging, CPUID, opcodes (the AMD manuals are better for this, as they split it into different sections for regular, MMX, SSE and so on, while the Intel ones are everything mixed together) and so on.

I don't know about anything else. Try the O'Reilly website - they have some pretty good books on most computer-related things.

[ATC]

Thanks, just still struggling with this. If there is any way I can accomplish this with GRUB, I will happily do so. There's really no need to reinvent the wheel I suppose. Same problem with GRUB though. Can't find anything about how to put it on the bootsector of an ISO which contains the kernel. Furthermore, I don't think it supports PE format, right? All I can find about it pertains to *Nix systems.

I have a thread at BrokenThron trying to resolve the same issue. I just posted an .ASM file I stumbled across in Microsoft's Singularity RDK. No, I'm NOT trying to make a managed OS/kernel, lol, but it looks like that bit of code can load the kernel from ISO. The syntax looks funny though, and it can't be compiled with nasm. I just don't know enough about all of this to sort it out. Spent all day searching too, and having the same frustration. I love Linux, but goodness! You guys hog the whole OS dev scene!

http://www.brokenthorn.com/forums/viewt ... =15&t=1305

There's my similar n00bish thread, lol. Honestly, if this is really common n00b stuff and annoying anyone, please let me know. I don't mean to pester anyone. Just understand, it's nearly impossible to find good information for this. So far absolutely 0 results, after 2 days of searching, about how to load files from ISO/CD/DVD.

[Combuster]

All I can find about it pertains to *Nix systems.

Wrong conclusion. Most, if not all of the information you see around here pertains to GNU toolchains, which work happily under windows as well. The absence of a fancy GUI to call the compiler for you doesn't imply some unix is running - see Cygwin

So far absolutely 0 results, after 2 days of searching, about how to load files from ISO/CD/DVD.

All the good pointers have already been given.
1. Get yourself a sane toolchain: GCC Cross-Compiler
2. Create and build a kernel: Bare Bones
3. Put the kernel and bootloader on CD: Bootable El-Torito CD with GRUB Legacy
4. Try it out.


On another note, here's a tip that solves some potential TL;DR issues:

There's my similar n00bish thread, lol.

Some people who get that they shouldn't behave rudely or arrogantly, demanding an answer, retreat to the opposite extreme of grovelling. “I know I'm just a pathetic newbie loser, but...”. This is distracting and unhelpful. It's especially annoying when it's coupled with vagueness about the actual problem.
Don't waste your time, or ours, on crude primate politics. Instead, present the background facts and your question as clearly as you can. That is a better way to position yourself than by grovelling.

[ATC]

I hope you're not implying that I'm "grovelling". Just admitting I'm new to this and sincerely hope I'm not being a pest. The word "noob" is not something I take seriously or take offense to, lol.

The absence of a fancy GUI to call the compiler for you doesn't imply some unix is running

I know that. Design requirement again: PE Format. And I've already got all the tools I need, and they work fine; save maybe a better ISO/CD burning program. And yeah, I went to get Mkisofs yesterday, but ended up failing, lol. I was too tired, which is my fault. But as far as my environment, compilers, linkers etc are concerned, I'm fine.

My problem is conceptual, and due to my inexperience with ASM. Everything works alright on a floppy or floppy emulation on VPC/VB. The real target isn't going to jive with floppies though, so this is what I've got to sort out. I suppose I've got to learn the hard way; as I've always done. Amazon, here I come! *shudder*

I've got a few ideas to try in the mean time. I'm going to end up writing a book of my own before it's done, heheh. I'll experiment with Grub and see if I can do anything with it. Maybe if I meet the MBS it can invoke a PE.

[neon]

Hello,

This is a reason I recommend floppy's over natively supporting CD's for beginners: They are easier to work with and their are more information to work with. You can use a floppy image on a CD and boot from CD if you like. If you want to natively support CD booting instead of CD floppy emulation, you will be required to go through the documents on the Wiki and the specification: El-Torito with ISO 9660 (CDFS) filesystem supporting.

It can work with a modified series bootloader though with CD support; Its been done before. This particular issue has nothing to do with the toolchain in use.

[Combuster]

Seriously, if you are expert enough and know you can deal with the stream of problems you'll get, feel free to use PE. But please don't complain if you do. If you want things to just work, use ELF. There's a reason why people use it over PE.

Also, it is possible to use the visual studio IDE with GCC rather than microsoft's compiler, so that can't be an argument either.

[Selenic]

I just posted an .ASM file (...) The syntax looks funny though, and it can't be compiled with nasm.

That's because there're two different formats for x86 assembly. GCC, gas, etc. use AT&T syntax, NASM uses Intel syntax (I think that's the right way around) and I believe YASM can handle either.

Honestly, if this is really common n00b stuff and annoying anyone, please let me know. I don't mean to pester anyone.

Just to expand on what's already been said, the fact that you're asking if it's noobish stuff is generally a sign that it isn't. This article pretty much explains why. Or, to see it in action, this bash.org entry is amusing.

Design requirement again: PE Format.

Why is that a design requirement? Besides, even if the apps are PE format, the kernel itself could be a (multiboot) ELF file...

[DavidBG]

I did this once before but it has been a long time.
First of all, for the BIOS to load your bootloader from a CD, you'll have to format your CD according to the "El Torito" standard.

After that you make the BIOS emulate a floppy drive by reading through the INT 13h calls with device 00.

Check out this page for more info.

You also may want to check this out. It is the Linux source for a CD bootloader.

I hope this helps, contact me for more info if you get stuck. If this info is in any way inaccurate, feel free to correct me.

David

[Lithorien]

Thanks, just still struggling with this. If there is any way I can accomplish this with GRUB, I will happily do so. There's really no need to reinvent the wheel I suppose. Same problem with GRUB though. Can't find anything about how to put it on the bootsector of an ISO which contains the kernel. Furthermore, I don't think it supports PE format, right? All I can find about it pertains to *Nix systems.

GRUB can happily load PE files: http://forum.osdev.org/viewtopic.php?f=1&t=21202

It requires you to have a little 'glue' (in the form of your multiboot loader - you need the AOUT kludge), and a little bit of digging. I started with a PE kernel, actually - but it's going to be a path that has very little documentation and assistance for you. I'd reccomend just using the ELF format to start and then, once you get the hang of things, work with PE... but that's my 2c on it.

[DavidBG]

Seriously, if you are expert enough

This is a little off-topic, but why does your quiz there say a heapsort is the best suited sorting algorithm for embedded realtime devices over a quicksort?

I always used quicksort, should I change to heapsort?

David

[ATC]

This particular issue has nothing to do with the toolchain in use.

Bingo. It's just the first time I've ever tried to use it for this, and the NT/VS community doesn't do much OS dev afaik. So information is very sparse, unfortunately. I pay good money for the tools I have because they're fast, comfortable, powerful and productive. That's why I'm not interested in suggestions to switch toolsets. But Combuster is correct; you can use external tools to build in VS, and I've done it a lot (and still do). I've used MASM on occasion to compile ASM given to me straight from VS. A beautiful thing; less typing commands and BATs and more typing code.

@ Lithorien: Yup, I figured that part out. Had to adjust the base address, but it's ok.
@ Combuster: Don't worry. I won't complain here because whatever goes wrong is my own fault!

Anyway, I think I'm getting on the right track now. I've been pouring over tech docs and brushing up my asm skills. Hey, if the Saints can make it to the Super Bowl, then I can do this, surely! Lol! Like I said, everything worked fine my first go'round with the floppies. But, I was using borrowed code and was pretty clueless about the whole thing. Now I'm writing my own ground up and peeping at some good examples. So I should get it sooner or later.

I know these "mysterious" requirements I've mentioned might sound outlandish to you guys, but there's a reason behind it all. Trust me, I would spare myself the trouble if there wasn't! I'm making the next gen Wii! No... Anyway, I'll post a video when I have something to show for myself. Surely I'll need several months minimum though. Long road ahead! Fortunately, this isn't a desktop OS so I don't have to fret about any graphics. If I have any more questions, I'll give it some more effort before slapping a thread up pre-maturely. Thankfully, I've got a bit of guidance now too. But thanks for all of the helpful answers thus far, folks, and I'll still be listening if you can offer more!

P.S.- Still open to suggestions on good books to buy concerning asm/OS development.