Blind-sided by multi-boot/PE problems...

[ATC]

Perhaps I celebrated too soon about my PE kernel and being "multiboot compliant".

I thought everything was ok, and as far as I could originally tell it was. Grub booted the kernel with ease, and my kernel obviously had control of the system. So, I played around a bit and got it to clear the screen, change colors, print a couple chars to the screen, etc, and figured it was time to move on to some real implementations of the groundwork. Wrong. I soon discovered my basic I/O implementation was giving horrible results. If, let's say, I tried to print char* teststring = "Hello from the kernel!"; I would get some random junk like:

sdf9087=!@^y654%$P .... (there was a LOT more, covering several lines, lol)

So I wonder what the heck is wrong. I check and recheck the implementations, and can find no problems with the code. I even copy and paste them into a Win32 console application to try out, and it worked fine; as if I was using a regular standard library. THEN I become aware that only local variables actually exist at runtime. If I have a global variable and try to print it via VGA, Virtual PC again prints random junk. Mind you, at the local level inside a function printing a single character at a time works perfectly somehow. On Sun's Virtual Box, nothing is printed at all if I try to print a global char*/string. It's like they disappear into an unknown (void *).

Then I got to thinking that maybe the way I've implemented the multi-boot code was bad, and causing data/resources to get chopped off during loading. So, I went to the only two online tutorials for this that I know exist:

http://ksrenevasan.blogspot.com/
http://forum.osdev.org/viewtopic.php?f=1&t=21260

Following those and all their settings to the letter, I'm still plagued with the same problem. Random junk is written to the video buffer when you try to print a string, and global variables (anything outside the functions) doesn't exist at all -- and may result in more random junk. For ksrenevasan's PE/multi-boot tutorial, I discovered that changing:

Code: Select all

...
                dd(0x0010200F)               ; load_end_addr
                dd(0x0010200F)               ; bss_end_addr
...

to:
                dd(0x001020FF)               ; load_end_addr
                dd(0x001020FF)               ; bss_end_addr

...Eliminates the issue with being totally unable to print an uncorrupted string. However, the same problem is still there in respect to only local variables working.

I'm using Grub 0.97 and the stage2_eltorito loader, if it is of any significance. I'm going to be trying to solve this, and thoroughly (re-)reading all of the information available for both the PE format and multi-boot specifications. But I'm hoping someone at least knows what could be the issue with this or slap a bit of sense into me. My guess is that there is just something obviously wrong with how my kernel gets loaded, and it's likely within that multiboot header somehow (it suggests to me that data/resources are getting chopped?). Alternatively, the way I've rigged VS to output the kernel could be flawed, but I really doubt this. The weird thing is the tutorial writers who I copied for validation suggest it works fine, and I doubt they would have posted it if it didn't. And yes, I've more than tripled check during my tests to be sure I set everything up like them (to validate whether or not their code would work). I'm just a bit baffled at this point.

Sorry for long question, but I wanted to be as clear as possible about my problem. Also, I kindly request that any suggestions to switch toolsets/formats/etc be withheld. The project is 100% 'academic' at this time, and the whole point (and requirement) is to use PE format and work in VS.

[xenos]

Did you include sections like .data, .rodata and .bss in your linker script? Most likely your global data resides in one of these sections, and if they are not present in the executable, only random junk is printed.

[gedd]

Did you include sections like .data, .rodata and .bss in your linker script? Most likely your global data resides in one of these sections, and if they are not present in the executable, only random junk is printed.

Can't be done with PE

[Combuster]

I just have to quote this warning. Some people are just so predictable:

Seriously, if you are expert enough and know you can deal with the stream of problems you'll get, feel free to use PE. But please don't complain if you do. If you want things to just work, use ELF. There's a reason why people use it over PE.

XenOS has the point in question: your (read-only) data sections are off. Go find them and put them where they belong, VC should have some linkerscript-like functionality, although maybe hard to find.

[midir]

I use PE too (maybe I'm just an idiot or something, but I was already familiar with PE, and knew nothing about ELF), and as others have said, your sections are off alignment.

Your .text (code), .data, .rdata, .rodata, .bss, etc. sections are going to each be given virtual addresses by the linker. But GRUB knows nothing about this -- it uses the location of the Multiboot header to orient the initial load address -- that is, to load that section (.text) where the linker prepared it to be loaded. But it will load the entire file as-is relative to this point. It's not going to parse the PE header and section directory to figure out where the linker instructed it to put the .data sections and so on.

So, to use PE, you have two choices:
(1) Write some sort of middle-stage loader (using no non-local variables) that will read the PE section directory when your kernel boots and move all the sections to the real addresses the linker expected them to end up at. It will have to move them in such a way that you won't overwrite bits you still need (tricky).
(2) Arrange your PE file so that the physical and/or virtual addresses of all the sections relative to the one containing the Multiboot header are exactly the same as the offsets of the bytes of the file. Then it can just be shoved into memory at one location (like the way GRUB will load it) and it will all work.

The second solution is the one I use. Unfortunately, it not nearly that simple. Not only you do need to be able to control the padding between the load addresses of the sections, you need to control the padding of how the sections are arranged in the file. I use only GCC, so unfortunately I have no idea if this is possible with Visual C. (Does it even support the concept of linker scripts? I have no idea. Sorry.) I do know that even with GCC's linker, using its --file-alignment and --section-alignment options, I could not get it to stop putting some padding between the PE sections in the file.

Therefore, with GCC, building a PE file, I resorted to writing a PHP script that would read in the resulting exe, extract each section, and dump it into a new raw file arranged exactly as I wanted it. (Using objcopy in the GNU toolchain should have been good enough, but I use a higher-half kernel at 0xF0000000, loaded initially at 0x00100000 physical, and the stupid program kept trying to produce a 4-gigabyte file spanning the entire address space).

I really don't recommend doing what I do. I stick with it because it already works, so there's no point fiddling with it, and I'd already done it before I understood exactly what I was doing. But since you have your heart set on doing this via PE, and since I don't know whether VC might offer a better way, here is my GNU LD linker script and [the relevant parts of] the PHP script I use to arrange things properly. Maybe you'll find it useful (or at least informative). (Though honestly, these guys are probably right about ELF making things easier. Perhaps there's a different linker that will let you build an ELF file from VC's object files?)

linker.ld:
(Note: I omitted the list of input files for brevity; ".text.boot" contains the asm code that initialises paging so I run the rest of it at a high virtual address. )

Code: Select all

ENTRY( _multiboot_entry )

SECTIONS
{
	. = 0x00100000;
	
	.text.boot . : {
		*(.text.boot)
	}
	
	. = 0xF0000000 + SIZEOF(.text.boot);
	
	.text . : {
		*(.text .rdata .rodata .data)
		*(.text.hot)
		*(.text.unlikely)
	}
	
	.bss . : {
		*(.bss)
	}
	
	_bss_section_end = . - 0xF0000000 + 0x00100000;
}

convertexe.php:

Code: Select all

<?php

$input_file = 'kernel.sys';
$output_file = 'kernel.sys';

// convertexe.php extracts the program sections from the PE header and places
// them with no padding into the output file.


error_reporting(E_ALL);

$data = file_get_contents($input_file);
$fout = fopen($output_file, 'wb') or die('Cannot open output file');

if (substr($data, 0, 2) != 'MZ') {
	die("Convertexe: Not a PE file or already converted.\n");
}

$out_size = 0;
$bss_size = null;
$sections = GetPESections($data);
//print_r($sections);
foreach ($sections as $section) {
	if ($section['Name'] == '.comment') continue; // skip
	
	if ($section['Name'] == '.bss') {
		$bss_size = $section['VirtualSize'];
		continue;
	}
	
	$section_data = substr($data, $section['PointerToRawData'], $section['VirtualSize']);
	$out_size += strlen($section_data);
	fwrite($fout, $section_data);
}

fclose($fout);

echo 'Convertexe: Output file: ' . number_format($out_size) . ' bytes';
if ($bss_size !== null) {
	echo '; BSS section: ' . number_format($bss_size) . ' bytes';
}
echo "\n";

exit(0);


function GetPESections($data) {
	$x = unpack('VPEOffset', substr($data, 0x3C, 4));
	$PEOffset = $x['PEOffset'];

	$PEHeader = unpack(
		'VSignature/vMachine/vNumberOfSections/VTimeDateStamp/' .
		'VPointerToSymbolTable/VNumberOfSymbols/vSizeOfOptionalHeader/' .
		'vCharacteristics',
		substr($data, $PEOffset, 24)
	);

	$sections_offset = $PEOffset + 24 + $PEHeader['SizeOfOptionalHeader'];
	
	$sections = array();
	
	for ($i = 0; $i < $PEHeader['NumberOfSections']; $i++) {
		$sections[] = unpack(
			'a8Name/VVirtualSize/VVirtualAddress/VSizeOfRawData/' . 
			'VPointerToRawData/VPointerToRelocations/VPointerToLineNumbers/' .
			'vNumberOfRelocations/vNumberOfLineNumbers/VCharacteristics',
			substr($data, $sections_offset + $i * 40, 40)
		);
	}
	
	return $sections;
}

*EDIT*: Maybe there's a simpler way; I just thought of this. For all the primary sections of my kernel I use the linker script to actually combine them into one big fat .text section to eliminate most of the padding annoyances (and make the file tighter). (This was something I figured out much later.) So nowadays, my PHP script's main purpose is to remove only a single piece of padding between two sections. If you can generate the output exe like this, the only remaining issue is if you want to use a higher-half kernel or load/run things at sparse virtual addresses. Otherwise, the resulting PE file should be fine -- with everything packed in one section, padding disappears.

[ru2aqare]

I use PE too (maybe I'm just an idiot or something, but I was already familiar with PE, and knew nothing about ELF), and as others have said, your sections are off alignment.

+1

So, to use PE, you have two choices:
...
(2) Arrange your PE file so that the physical and/or virtual addresses of all the sections relative to the one containing the Multiboot header are exactly the same as the offsets of the bytes of the file. Then it can just be shoved into memory at one location (like the way GRUB will load it) and it will all work.

This is what I do. Just specify /ALIGN:512 to the linker and it will align all sections on a 512 byte boundary as opposed to aligning them on a page boundary. The downside is that you can no longer execute code linked with this option in a Win32 environment.

Also I put all Multiboot-related stuff in its own segment, .text$1 and the linker will automatically combine it with the rest of the code segment (.text section), but still keep said segment at the lowest virtual address. The $ functionality, as far as I know, is not officially documented, although the MS C runtime makes use of it. It is pretty handy though.

[ATC]

Thanks for the information, guys. Com, trust me, bro. It has nothing to do with me being "expert enough". And I'm definitely not here to complain and make demands. I'm enjoying this, and overcoming the challenges are all part of the experience. And thanks again to everyone for being kind enough to offer some insight.

I, by a fluke, discovered that further increasing those aforementioned numbers gets rid of the local vs global variable problem AND the corrupted memory. This, in turn, suddenly made all of the I/O code I worked hard on work beautifully. That seems to further imply what I and you all have suggested. But, I don't call that a "solution". It's an ugly hack that just happened to look right on the screen, lol. So I need to figure out why that is and what I can do to implement a robust and (re-)scalable fix. I suppose the only way to do that is to become more intimately familiar with the specifications and read up on everything else I can find (no shocker there). I notice that everyone who does this has done it a bit differently. So, I'm going to need to do some broad experimentation as well.

There is also another option for me, which won't go against the design requirements: making an initialization program which is multi-boot compliant to perform all the setup and THEN execute the PE kernel. Only problem there is that I lose my "home-field advantage" of working with VS and formats I know and love. But, I have the GCC tools and there is a wealth of information about it should I really need to do it. This idea does have some immediate advantages though:

1) Code/resources needed for initialization only need to exist the duration of initialization itself and can be freed once the kernel executes
2) Less problems to worry about in "forcing" a PE executable to comply with multiboot specs
3) Could potentially allow more advanced options and setup logic for the system and minimize code repetition

Another problem though is how I'll read the CD/DVD drive being in Pmode (possibly Unreal). Sort of pushing the limits of my greenhorn experience level, but I guess this whole thing has in the first place! I'll learn, one way or the other; hard or easy (seems to always be the hard way though, lol).

Thanks again, and I'll be eagerly awaiting any more information you folks can provide. It's quite an honor that you've taken a bit of your time to help and provide some insight!

[midir]

Just specify /ALIGN:512 to the linker and it will align all sections on a 512 byte boundary as opposed to aligning them on a page boundary.

Ah, good idea, working with the padding rather than against. I never thought of that solution because I didn't want blank space in my kernel, so I battled obsessively to get the padding down to 0-4 bytes.

[ru2aqare]

Ah, good idea, working with the padding rather than against. I never thought of that solution because I didn't want blank space in my kernel, so I battled obsessively to get the padding down to 0-4 bytes.

If you use the linker that comes with VS, I don't think you can get the alignment lower than 128 or something like that. I only tried down to 256 though, there is little point to go below that. And unless you have a ridiculous amount of sections (or a very, very small bootloader), the section alignments don't amount to a significant amount of wasted space either.

[Gigasoft]

I have got alignment down to 1 byte with the VS linker, but it required that all my sections had 1 byte alignment and that I had no .drectve section in my object files (meaning no entry point set with the end directive).

[ATC]

/bragging:OFF
#include "humble_happiness.h"

Knock on wood (*knock knock*), but I think I've pretty much gotten it "perfect".

Everything appears to be working fine; boots correctly, no memory corruption, graphics are fine, all sections load/no chopping, etc. Quite thrilled! I was starting to worry it was a lost cause myself, and I wouldn't blame you guys for thinking I wouldn't get it, lol. Been loads of fun getting to this point, and I'm thoroughly excited about the challenges that will come next! It's a HUGE relief, and I'm so glad I can focus on a good design and some solid code, rather than the shifty, tricky and practically uncharted waters of PE multiboot compliance. If anyone wants help with this or information, feel completely welcome to ask/pm/whatever. I learned that this has little to do with how "good" or skillful of a programmer you are. It has more to do with how much TIME you have to read, read, read and test, lol. If you stick to the books and build just right with a 100% valid (&& correct) header, it works like a charm (go figure, lol). But note the distinction between "valid" and "correct". Valid just means Grub is happy with it and loads the kernel. Correct means your header correctly describes to the bootloader how it is to handle your kernel. And you'd for darned sure better have the compiler and linker configured correctly to the n'th degree, or ph41l. Yup, one slip and you get a nasty curse from Grub or even a triple-fault.

Well, thanks again for all of your kindness and helpfulness, and best of wishes to all of you with your OS's and projects!

Regards, and I'll be 'round to hang out!

[gravaera]

Hi,

Why not create a Portable Executable/PE category in the Wiki and all of you PE fans add to it and thus create a small PE dev base of tools, and guide articles so that we can reduce the number of questions associated with it? The number of {PE based questions has been increasing lately, and it would be much appreciated.

--A suggestion,
gravaera