True cross-platform development

[SpyderTL]

I've been looking for a true cross-platform development technology for a long time, and I've even spent years trying to build up my own language, compiler and tools with the eventual goal of being able to write a single program (i.e. myprogram.exe) that runs on virtually all platforms.

The obvious answer would be Java, or more recently .NET or even WebAssembly. But each of these have their own issues with performance or lack of support, etc. So, I'm still looking for another alternative.

I've recently been looking at the different object file formats, and library file formats, and trying to learn how code can be shared and executed. Of course, there are several different file formats for both of these types of files, and these tend to be either Windows based, Linux based or Mac based, and none of them are supported across all platforms.

What I would like is an assembly library that abstracts the Operating System, which is simple enough to create. However, even if you had this library, you would still have to create an .EXE for Windows, an ELF file for Linux, a Mach-O file for MacOS, etc.

Java and .NET get away with being cross platform by having an actual platform specific program pre-installed on the machine, which is responsible for both loading and executing the actual program. So, running interpreted language programs, and running intermediate language programs are both fairly well covered. What is missing is loading and running natively compiled programs on multiple platforms.

So, could you create a platform specific loader that would load a natively compiled executable program in a common format, and execute it? All of the OS specific abstraction could either be written by hand, or handled by a static library.

Does anyone know of an existing tool that can load ELF/MachO files on Windows, or .EXE/ELF files on MacOS, etc.? Or has anyone here attempted something like this, or does anyone see a problem with this type of solution?

Thanks.

[mmdmine]

Hey bro! we have same goals!
take a look at my article here: https://mmdmine.github.io/blog/dynamic_abstraction.html
My idea is defining a global executable format and writing a simple kernel module/driver to add support of executing it on the host. the system API also must be abstracted by a global library. this way we can have a cross platform native executable.

[iansjack]

To be truly cross-platform you are going to have to abstract not only the operating system, but also the underlying hardware. Else a program that runs on an x86 platform won't run on an ARM platform (for example). This takes you back to some form of intermediate code. So Java, SmallTalk, or something like that, would seem to be the solution.

[PeterX]

If I understand the issue correctly, You have to (a) program the executable-file loader for every OS, or (b) have a layer ontop of the existing OS, like "Wine".

Unfortunately Windows and Apple-OSs are proprietary so you won't get a chance to extend the exec-loader (option a) for these OSs. (Or does someone know a way to do so? I don't understand how mmdmine's suggestion would work.)

You can stick to open source systems like ReactOS, Linux, etc. But there ain't a free Apple-OS-"clone" as far as I know.

Or choose option b.

In contrast to iansjack I think you don't need high level contructs like Java or Smalltalk and you don't need to abstract the underlying hardware.

[SpyderTL]

I understand that to be cross-platform, you would need to recompile to the native hardware byte code. But I also said that this area is pretty well covered by the technologies that I mentioned above. What is missing is the ability to load the same compiled binary on any OS, or even from a boot loader, and execute it. With a hint or two from the loader, the exact same binary could run with no OS, under MS-DOS, as a Windows Console application, a Windows Desktop application, a Linux Desktop application, a Linux Console application, a Windows Phone, etc. (Either by writing a huge switch statement, or by abstracting the OS details away with a static library.)

I was just curious if anyone had tried this, or could think of any reason that it wouldn't work, not whether it was a good idea, or worth the effort or not.

[bzt]

I understand that to be cross-platform, you would need to recompile to the native hardware byte code. But I also said that this area is pretty well covered by the technologies that I mentioned above.

Just a thought, instead of supporting PE, ELF whatever, have you considered using IB files? In theory you could simply cut out the native bytecode converter from LLVM and reuse that in your executer, but I have no clue how big project that would be (I'm not that familiar with LLVM source). But seems possible, and IB files are OS independent. I do not call this interpreter, because this is rather an executer that actually compiles the code into memory (with the other segments) and then it runs that. This also cuts out the need for PE/ELF etc., as it creates segments in memory from the IB file right away.

What is missing is the ability to load the same compiled binary on any OS, or even from a boot loader, and execute it. With a hint or two from the loader, the exact same binary could run with no OS, under MS-DOS, as a Windows Console application, a Windows Desktop application, a Linux Desktop application, a Linux Console application, a Windows Phone, etc. (Either by writing a huge switch statement, or by abstracting the OS details away with a static library.)

You are perfectly right. Once you have converted the independent bytecode into native one, all you need is an interface to the underlying functions. A huge switch wouldn't be good imho, as it would be unmaintainable on the long run. Most existing solutions are aiming towards a library (one for each platform), with more or less success. If you think about it, BIOS and UEFI Run-Time are such libraries (with unique calling conventions, and more or less success providing a common abstract interface).

I was just curious if anyone had tried this, or could think of any reason that it wouldn't work, not whether it was a good idea, or worth the effort or not.

Well, for Java there's SanOS with exactly that goal (from the site: "was developed as part of an experiment on investigating the feasibility of running java server applications without a traditional operating system")

For WASM, there's WASI to provide a platform independent interface (something that you would consider a static abstration library for each platform). But it is far from being finished, you'll have to wait a couple of years before you could call it mature. There's no minimalistic, boot-loader like implementation yet that I know of, but certainly doable just like SanOS, and it certainly can be done on UEFI.

Imho they are doing WASI the wrong way (but that's just imho). Instead of designing a simple and straightforward API towards the WASM code that hides away all the OS and hardware specific details (like libc does for BSDs and Linux for example), they are planning to create a system of incompatible modules. This will never work properly imho, there'll be always problems about missing and or conflicting modules with incompatible interfaces.

Cheers,
bzt

[eekee]

This is a very old problem! The common solution in the 70s and 80s was P-code; the bytecode invented for Pascal. P-code wasn't a good fit for some of the languages or hardware it interfaced, but portability was so valuable that it was a selling point for mainframe software. P-code was bytecode, Java .NET and others also use bytecode, and I can't imagine another way of doing it. Well, I can: multiple machine-specific executables could be bundled together as Apple did for their transition to Intel, but that doesn't allow for new target architectures to be added without recompiling everything.

If you really want executables, you could maybe put multiple interpreters/jit-compilers into each executable you want to distribute, and I suppose that would do for demoing the programs. Or you could just include a variety of such interpreters/whatever as separate files. Apple could have done this, I thought they had everything in place to do it, but they chose to bloat their binary files instead. (I didn't look too closely.) IMO, the better way is to offer the interpreter+framework separately, although that does tend to raise compatibility issues over time. It's the way everybody does this, anyway. (Unless you count scripts distributed as source code.)

For some papers on a particular implementation, have a look at Inferno. It runs machine-independent bytecode (called Dis), JIT or interpreted, on a wide variety of architectures, native (with kernel in C) or hosted under other OSs. Dis is close to the machine except for threading abstractions. Here's the papers online, and they are included in the distribution. Inferno itself would need a lot of work to bring it up to date, sadly.

tl;dr: Old problem, many solutions, almost all solutions use JITted bytecode.

Turning to the problem of launching the programs...

Unfortunately Windows and Apple-OSs are proprietary so you won't get a chance to extend the exec-loader (option a) for these OSs. (Or does someone know a way to do so? I don't understand how mmdmine's suggestion would work.)

There must be a way; this problem is at least as old as Java. When these OSs were made "Java-compatible" in the 90s, I don't know for sure whether they were given a flexible mechanism for adding executable types or just Java was added, but I strongly suspect the former. (I have a very vague memory of the phrase "not just Java" in the context of these changes.) Linux has a flexible mechanism; you don't need to touch its source code to add executable types.

Another mechanism is definitely available on both systems, file association, but it has disadvantages. In Windows, it prevents you launching your executable with arguments from the GUI, meaning you can't drag files to edit onto your portable editor. If launched from the command line the shell won't wait for it to complete. It's been a long time since I used Mac OS X, my memory has faded a bit, but I don't think it's faded too much. I don't think I tried the drag & drop. I'm pretty sure the command line is slightly more restrictive: you have to prefix `open` to launch file associations from the command line, and like Windows, it doesn't wait.

[mmdmine]

If I understand the issue correctly, You have to (a) program the executable-file loader for every OS, or (b) have a layer ontop of the existing OS, like "Wine".

Unfortunately Windows and Apple-OSs are proprietary so you won't get a chance to extend the exec-loader (option a) for these OSs. (Or does someone know a way to do so? I don't understand how mmdmine's suggestion would work.)

You can stick to open source systems like ReactOS, Linux, etc. But there ain't a free Apple-OS-"clone" as far as I know.

Or choose option b.

In contrast to iansjack I think you don't need high level contructs like Java or Smalltalk and you don't need to abstract the underlying hardware.

I think it should be possible to add new executable type to OS by adding a driver or kernel module, or at the least by patching kernel, even to Windows. universal executable is a native code with some special unnative opcodes that should be compiled to kernel call or universal library call at runtime, it's possible by adding a simple JIT compiler to kernel and it would be fast. this JIT compiler doesn't do anything other than loading the universal library and linking it to the executable that's going to be run. universal library is an abstraction of OSes' API. this way you just need to recompile your program for each machine e.g. intel or arm. this is just theory.

[Candy]

> I think it should be possible to add new executable type to OS by adding a driver or kernel module, or at the least by patching kernel, even to Windows. universal executable is a native code with some special unnative opcodes that should be compiled to kernel call or universal library call at runtime, it's possible by adding a simple JIT compiler to kernel and it would be fast. this JIT compiler doesn't do anything other than loading the universal library and linking it to the executable that's going to be run. universal library is an abstraction of OSes' API. this way you just need to recompile your program for each machine e.g. intel or arm. this is just theory.

You can do this in Windows for example by registering a new file type handler with your loader. Your loader is then invoked as "loader.exe myExecutableType.elf" and does the loading.

To load it, you allocate new memory blocks for the memory it will occupy (VirtualAlloc), load the relevant data from disk (ReadFile), mark it as the correct protections (VirtualProtect) and then call the entry point. You may need to load dynamic libraries too, and probably provide an equivalent dynamic library loader functionality for it to do the same.

[mmdmine]

> I think it should be possible to add new executable type to OS by adding a driver or kernel module, or at the least by patching kernel, even to Windows. universal executable is a native code with some special unnative opcodes that should be compiled to kernel call or universal library call at runtime, it's possible by adding a simple JIT compiler to kernel and it would be fast. this JIT compiler doesn't do anything other than loading the universal library and linking it to the executable that's going to be run. universal library is an abstraction of OSes' API. this way you just need to recompile your program for each machine e.g. intel or arm. this is just theory.

You can do this in Windows for example by registering a new file type handler with your loader. Your loader is then invoked as "loader.exe myExecutableType.elf" and does the loading.

To load it, you allocate new memory blocks for the memory it will occupy (VirtualAlloc), load the relevant data from disk (ReadFile), mark it as the correct protections (VirtualProtect) and then call the entry point. You may need to load dynamic libraries too, and probably provide an equivalent dynamic library loader functionality for it to do the same.

sure. as I've written in the article it's the plan B if it's not possible to patch kernel on some platforms. An alternative to file type registery is a batch script that runs program on top of runtime, like this:

Code: Select all

@loader.exe %~dp0\program.bin %*

[SpyderTL]

You can do this in Windows for example by registering a new file type handler with your loader. Your loader is then invoked as "loader.exe myExecutableType.elf" and does the loading.

To load it, you allocate new memory blocks for the memory it will occupy (VirtualAlloc), load the relevant data from disk (ReadFile), mark it as the correct protections (VirtualProtect) and then call the entry point. You may need to load dynamic libraries too, and probably provide an equivalent dynamic library loader functionality for it to do the same.

sure. as I've written in the article it's the plan B if it's not possible to patch kernel on some platforms. An alternative to file type registery is a batch script that runs program on top of runtime, like this:

Code: Select all

@loader.exe %~dp0\program.bin %*

Yep. This is what I had in mind. A single binary that can be loaded on any OS, or even by GRUB, which would then determine what OS services are available, and use those services, or provide it's own, where needed. Obviously you would need separate binaries for each CPU, but there would be no performance hit (or maybe a very small performance hit), because the code is still native to the CPU, and there is no JIT and no interpreter.

I'm currently working on getting a working 1) console application and a working 2) desktop (windowed) application working on both 3) Windows (EXE) and 4) Linux (ELF). Once this is done, I'll try combining all 4 projects into a single project with 4 different loaders, and see if I can pull off a simple Hello World program that runs natively on all four platforms.

After that, assuming I make it that far, we'll see if there is any reason to take the concept any further.

Thanks, guys.

[mmdmine]

@SpyderTL It is not as easy as you might think. At first you should design your executable format and your universal library then implement a custom compiler and assembler that targets your platform. After that you need to write the loader and implement the universal library for each OS (or first write the loader? Someone advised me to implement runtime then compiler).
How would you call the library? Do you jump to constant offsets that is set up by loader? Do you use something like JIT compiler to generate dynamic library call code on run runtime? Or something else?
And grub is a bootloader, not an OS. You may boot kernels, chainload something on it or load modules but it's not a target for programming applications. It would even harder.
Good luck, I'll check you when I got back my PC.

[bzt]

Yep. This is what I had in mind. A single binary that can be loaded on any OS, or even by GRUB

Obviously you would need separate binaries for each CPU, but there would be no performance hit (or maybe a very small performance hit), because the code is still native to the CPU, and there is no JIT and no interpreter.

I'm confused. Above you wrote that you want a single binary, and here you write you are planning separate binaries for each CPU. The payload can be a single file, with separate loaders for each platform.

What do you have in mind? Just to clearify, there are two options here, and I'm not sure which one you are after:

1. you have a single binary (like .NET, Java, WebAssembly etc. as in single binary means single text section which implies single file)
in this case you must convert the code into native code in run-time (either by creating a native text segment on load or by interpreting as you execute). For this, I'd suggest IB files as the code that converts those into native code is already written (and free to use as LLVM is FOSS).

2. you have multiple binaries, and it's only the container that's common (in this context binary means multiple text sections, which could mean more files, but not necessarily)
this is pretty much like FAT executables under MacOSX, which had two text sections in the file, one for PPC, and one for Intel. The loader just created the text segment out of the one for the architecture upon load. Is this what you want? For this, I'd recommend ELF (with a specific ABI/machine magic) as that's much more flexible than PE (you can't store multiple binaries in a PE, but you can in ELF).

Both ways has advantages and disadvantages, and both have been done, so both are viable. It merely depends on what is your goal exactly, at which layer you want to abstract: code level or container level?

Thanks, guys.

Welcome, and good luck!

Cheers,
bzt

[SpyderTL]

To be clear, I want to write an application once, and run it on any OS (or even no OS), as long as the CPU matches the compile target for the executable.

If Windows, Linux, OSX and GRUB could all load the same ELF file, then the application would just need to figure out which one actually loaded it, and it could call the appropriate syscalls for that environment. With some static libraries to abstract this away, the application wouldn't even necessarily be aware of the platform details.

Unfortunately, Windows can't load ELF files, and GRUB has some limitations on it's ELF support. See here: https://stackoverflow.com/questions/253 ... 1#25492131

But the ELF format is probably the closest thing to a universally supported format, so it may just be a matter of writing a custom ELF loader for Windows, and working out an ELF format that works on all other platforms.

If I can somehow get my one executable loaded into memory on all platforms, without any interpreter, and without any Just-in-time compiler, I would consider that to be a success. Write once, compile once, run nearly anywhere, at native speeds.

To put it another way, .NET and Java both use native runtimes that are responsible for both loading the actual program, and re-compiling it into native code for the current platform. In my situation, the code is already in the correct format for the current platform. It just needs to be loaded into memory. So, similar to Java and .NET, you could create an install for your loader, just like you "install" .NET or the JRE now. Then, you could run any program that was compiled for your CPU, regardless of whether you had Windows, or Linux, or MacOS, or just wanted to put the program on a boot floppy with GRUB. The program would work virtually the same on all 4 environments, either in console mode or in windowed mode.

It seems like this should be possible, if not entirely practical. But in either case, it's a good excuse for me to learn the PE and ELF file formats, if nothing else.

[zaval]

does anyone see a problem with this type of solution?

It's not hard, so I guess easier would be to find one, who doesn't. because what you described you want is Utopia. looking not as futuristically unreal as the unfortunate natural language programming topic, however having the same chances to become a usable solution. in short, first, unified executable image format is the least significant problem for it if any, what you casually omitted alluding to some magical static library, that would make it all abstracted and deal with all the myriads of differencies between OS environments is the real problem. if this "truly" crossplatform design would be able to produce some results at all, it would be either as primitive as being able to run the simplest programs like "hell world" printers or it would hopelessly sink in tons of bloat of indirections, compatibility layers, translators etc, turning any program doing more, than printing "hello world" into a bloat hell, comparable to javascript hog in modern browsers, that are able to easily hang even monstrous rigs, with loads of computational power.
but if it's what you like, it's good for you, no matter the aforementioned. I just wanted to note, that contrary to the bzt misinformation, PE format is not less flexible, than ELF, not a bit, and oppositely, there is no a bit in ELF, that makes it flexible. and of course you can produce multiarch binaries with PE - there is a directory mechanism inside of it, you can use that for keeping there info on what archs' binaries are present there and where exactly they reside, some kind of special section, that is discardable and serves as a driver for the loader, or you may differentiate code (and other) sections another way, any you wish, PE doesn't limit this. but unlike ELF, you can provide a normal DLL-like import export mechanism, not just that moronic PIC of ELF shared objects. actually, executable format here is not the problem, said that already. neglecting the complexity of an executional environment is.
to PeterX, expandability is not a virtue of being opensource, it's a virtue of a good design and being well documented. Candy demonstrated how one can add ELF loading support to Windows. just having million of lines of foreign code in front of your eyes is not a guarantee you are at the easiest way of achieving your goals of expanding something over there. I'd rather want to deal with a well structured system, that describes well how it can be extended. you got the point.