Truly portable executables (bootable PE32)

[technik3k]

Original contest was at http://forum.osdev.org/viewtopic.php?f=2&t=21791. No entries

I am attaching my own pe2boot utility with a couple of examples for MSVC, MinGW/GCC, and TinyCC.

Resulting PEs run on Windows and Linux and are bootable on a bare system from CDROM, USB, or PXE/BOOTP.

Runtime stub runs on a bare hardware or under DOS and handles IRQs, Exceptions, and int86() - and it all takes <1.3k!

[technik3k]

The updated version of pe2boot is here and has new features:

1. 1. Ability to boot directly from network (via PXE/BOOTP)
   2. Can create bootable CDROM images in both "No Emulation" and "1.44 Emulation" modes.
   3. Can make bootable USB sticks (then just copy your executable to the root directory and it boots!)
   4. Better executable compression and updated C and ASM examples for MinGW/gcc, fasm, tcc, and msvc compilers.

Most of barebone examples rely on GRUB. Unlike them pe2boot makes a single executable that can boot from a variety of sources, such as network (PXE/BOOTP), USB drives, CDROMs, Floppies, or run natively inside Windows and Linux operating systems. Another distinct feature is that you don't have to worry about compiler specific static linking rules. Rebasing is done dynamically during decompression.

This is hello.c example with gcc and msvc entry point declarations (other compilers like tcc or bcc32 could be used with small tweaks as well):

Code: Select all

#include <stdio.h>
#include <stdlib.h>

#ifdef __MINGW32__
void debug(void){asm("int $1");}	void __stdcall DllMainCRTStartup(int a, int b, int c)
#else //__MSVC__
void debug(void){__asm {int 1};}	void mainCRTStartup(int a)
#endif
{
	char *p = "Hello World";
	char *v = (char*)(0xB8000+0x6c7);

	if( (int)&a < (int)v )		// if stack < 640k - it is not Win32
	{
		for(;v++,*v++=*p++;);	// write directly to the video memory
		debug(); 					// invoke debug monitor built into the stub
	}
	else // Win32
	{
		printf(p);
		exit(0);
	}
}

You may use virtually any compiler that can generate PE with base relocations:

Code: Select all

   C:\test>cl hello.c /link /fixed:no /subsystem:console /nodefaultlib msvcrt.lib
   C:\test>c:\mingw\bin\gcc -s -shared -nostdlib hello.c -o hello.exe -lmsvcrt
   tux$ i586-mingw32msvc-gcc -s -shared -nostdlib hello.c -o hello.exe -lmsvcrt

Now you need to compress hello.exe and create bootable USB or CDROM images (pe2boot.exe will run either on Windows or Linux with Wine):

Code: Select all

   pe2boot.exe -s pe_stub.bin  -i hello.exe -o hello.exe     <- compress PE

   pe2boot.exe -b boot_fat.bin -i hello.exe -CD0 hello.iso   <- No emulation
   pe2boot.exe -b boot_fat.bin -i hello.exe -CD2 hello.iso   <- 1.44 emulation
   pe2boot.exe -b boot_fat.bin -i hello.exe -FLP hello.flp   <- 1.44 floppy
   pe2boot.exe -b boot_fat.bin -L "HELLO   EXE" -FAT \\.\U:  <- USB drive
   copy hello.exe u:\	 <- if you boot from USB drive it will load hello.exe

Finally, you are ready to test how it runs or boots on various environments:

Code: Select all

  hello.exe            							<- "Hello World" Win32/Linux
  qemu -fda hello.flp  							<- "Hello World" boot floppy
  qemu -cdrom hello.iso							<- "Hello World" boot cdrom
  qemu -tftp . -bootp hello.exe -boot n  	<- Direct boot with BOOTP

Plans for the near future are to be able to create hybrid USB/CDROM images like ISOLINUX.

Please, let me know if you have any specific comments, suggestions, or feature requests. If you are about to comment, that linker scripts aren't hard and static linking rocks,etc... Please, remember that the goal of this project is truly portable executable that can run on the bare hardware as well as inside an existing OS. In this scenario static linking is not a good option.

[bluemoon]

Sound like your're working on an binary compatibility layer (For example, Linux Binary Compatibility).
I'm surprised you do it on bare-bone, since most people would implement it last when the OS has richer functionality.

Anyway, I'm interested on how many functions you provide in the bulit-in runtime, since it affect the usability on the whole environment.

[technik3k]

Sound like your're working on an binary compatibility layer (For example, Linux Binary Compatibility).
I'm surprised you do it on bare-bone, since most people would implement it last when the OS has richer functionality.

Anyway, I'm interested on how many functions you provide in the bulit-in runtime, since it affect the usability on the whole environment.

Actually, I am not doing that myself, but use this concept on Linux.

Pe2boot's output is a PE32 executable. The runtime it adds to an EXE is only 1216 bytes. Think of it as an empty shell for your app with several handles bolted outside that different loading systems can hook to.

When run on Windows it simply lets you call Win32 functions that you normally import from various DLLs.

When Linux loader sees PE32 signature it calls Wine loader, which handles any Win32 calls. In the hello.c example those are printf() and exit() normally found in MSVCRT.dll just as it would happen on Windows. In addition to Win32 functions, you can also call Linux syscalls via "int 80h" (unless it is blocked by security policy).

When you run on DOS or bare hardware, the runtime will detect available memory, and switch CPU into protected mode and call your app. Your interface to the outside world is int86(regs,regs) function and any port I/O, if you like so.

Normally, if you want to make a system utility that runs and boots on various platforms, you would need to create separate:

• Win32 executable to run on Windows 9x, XP, Vista, 7 (or their server siblings)
  ELF executable to run on Linux, FreeBSD, etc...
  DOS executable with some sort of 32-bit extender for FreeDOS, MS-DOS, etc...
  Bare hardware version of the executable packaged onto
  • Bootable CDROM image
    Bootable Floppy/USB image
    Image for network boot using PXE/BOOTP
    Boot from HD partition, CHS/LBA, MBR/EFI, large sector size headaches...

Things will get complicated fast...

If you want to limit number of binaries the first logical step is to use PE32 because, thanks to Wine developers, virtually any linux box could run it or is one apt-get away from being able to.

Then to take care of DOS and direct bare hardware boot you need a small runtime that switches CPU into protected mode. This has been done by many people for the last 25 years (80386 was introduced by Intel in 1985). What I did was adding an integrated dynamic loader that could rebase Win32 code to any address and made entire thing ridiculously small (1216 bytes).

Finally, for various boot methods there are few tricks inside runtime itself and and additional 512 byte FAT1x boot sector that works on CDROMs, Floppies, USB sticks, and HardDrives of various sizes and geometries. It is possible to create a hybrid image that can be both burned onto CD/DVD or rawwritten onto floppy or USB. When you boot from it your app.exe boots, when you insert it under an OS you can run app.exe under that OS.

Conceptually, it might even be possible to create a binary that is a valid PE32 executable, valid PXE/BOOTP binary, valid OPTION ROM image AND a valid hybrid bootable CDROM/Floppy/USB/HD image all at the same time. But that would be an overkill Any takers?

[AndrewBuckley]

Very cool stuff. Barring Windows support, could the same thing be made with ELF files?

[technik3k]

Very cool stuff. Barring Windows support, could the same thing be made with ELF files?

Yes, probably, it is possible. I don't know know enough about ELF file format and dynamic loading to say for sure. Also, just like MultiBoot, PXE/BOOTP has specific requirements of where the magic number and header should reside. I was able to squeeze them into MZ/PE32 header, but it might be a problem with ELF.

As I said in the previous post, my goal is to run everywhere and running PE32s on Linux is only one apt-get away, while running ELFs on Windows is a major headache. It is much harder to install and configure Cygwin or similar emulator than Wine and you will be limited to 64k with .com files under DOS.

[bluemoon]

When run on Windows it simply lets you call Win32 functions that you normally import from various DLLs.

When Linux loader sees PE32 signature it calls Wine loader, which handles any Win32 calls. In the hello.c example those are printf() and exit() normally found in MSVCRT.dll just as it would happen on Windows. In addition to Win32 functions, you can also call Linux syscalls via "int 80h" (unless it is blocked by security policy).

When you run on DOS or bare hardware, the runtime will detect available memory, and switch CPU into protected mode and call your app. Your interface to the outside world is int86(regs,regs) function and any port I/O, if you like so.

I'm a bit confused, although the resulting binary is a single file, you still have to write different code segments for different target platform and branch internally? What will happen if I called winapi in the code and the program loaded on bare-bone? do it fail to resolve symbols or just call a stub function and result fail? How about I use in86() and run the program under windows?

What's the advantage over the following environment detection stub + zip/concat solution?
binary layout:
stub
[executable map]
PE executable for Windows, with MZ stub for DOS
executable for Linux

workflow:
1. The loader pass control to stub
2. stub detect environment
3. get offset of suitable executable from the map
4. perform initialization, relocation and pass control to the suitable executable

[technik3k]

I'm a bit confused, although the resulting binary is a single file, you still have to write different code segments for different target platform and branch internally? What will happen if I called winapi in the code and the program loaded on bare-bone? do it fail to resolve symbols or just call a stub function and result fail? How about I use in86() and run the program under windows?

What's the advantage over the following environment detection stub + zip/concat solution?
binary layout:
stub
[executable map]
PE executable for Windows, with MZ stub for DOS
executable for Linux

You are right about workflow, and that is what I am doing. The difference from your layout is that there is no executable for Linux. Thanks to Wine, I could run native Win32 executable on Linux. The switch happens when I am under DOS or BIOS. Here is my layout:

• MZ header
  -> BOOTP entry point
  ...
  PE32 Header
  -> DOS entry point
  ...
  BOOTP magic
  -> BIOS CDROM/USB/HD boot entry point
  ...
  commonBOOTP_DOS_BIOS:
  • Detect memory, A20, XMS
    Initialize GDT, IDT, IRQ handles
    Switch CPU to 32-bit protected mode
  jmp common32
  -> Win32 entry point
  common32:
  • Decompress and rebase executable
    if Win32 foreach DLL { LoadLibrary(); GetProcAddresses(); }
    jmp app entry point
  (compressed payload app)

Then one of arguments passed to the app is environment id. If you call Win32 function under DOS you crash. If you call int86() under Win32 you crash. Implementing a layer that provides common functionality depending on the platform is the subject of the next project. What I did so far is just built a container that is recognized as native on the various platforms.

[bluemoon]

I see, thanks for the explanation, that make senses now.

[mooseman]

Looks very cool!
Just wondering..... what license is the code released under? There seems to be no license info in the download, and none in the source files that I looked at.
- m

[Combuster]

When Linux loader sees PE32 signature it calls Wine loader, which handles any Win32 calls. In the hello.c example those are printf() and exit() normally found in MSVCRT.dll just as it would happen on Windows. In addition to Win32 functions, you can also call Linux syscalls via "int 80h" (unless it is blocked by security policy).

When you run on DOS or bare hardware, the runtime will detect available memory, and switch CPU into protected mode and call your app. Your interface to the outside world is int86(regs,regs) function and any port I/O, if you like so.

So you still need to write two applications. Plus that any attempt to use printf or anything like it fails because it will either give a linking conflict if you provide your own (something you need to do for bare metal support) or you use windows' and you will most likely crash on bare metal.

In other words, welcome to Jurassic Park. Why aren't the raptors in their enclosure?

[technik3k]

Looks very cool!
Just wondering..... what license is the code released under? There seems to be no license info in the download, and none in the source files that I looked at.
- m

It is released under ISC license. Two main sources files have it. I will add a separate licence file as well.

So you still need to write two applications. Plus that any attempt to use printf or anything like it fails because it will either give a linking conflict if you provide your own (something you need to do for bare metal support) or you use windows' and you will most likely crash on bare metal.

In other words, welcome to Jurassic Park. Why aren't the raptors in their enclosure?

So what would be your suggestion? If you intend to have a utility running on multiple platforms you have to deal with multiple backends anyway.

[Combuster]

So what would be your suggestion? If you intend to have a utility running on multiple platforms you have to deal with multiple backends anyway.

The obvious thing: write a program in strict ANSI C. The only deal with multiple backends is that I need to compile more often, but my code is shorter since it's oblivious to backend implementation details and it will run on more platforms than what your tool can ever hope to provide me.

That doesn't mean it's not a nice gimmick, but it's also just that: a nice gimmick, which like Raptors should not roam in the wild because it will lead to accidents .



Edit: Do you know Java?