running 32-bit code in LM64

[kerravon]

Why do you think I don't need boot services? That's what I'm using to access disks and screen. Currrently I'm even accessing individual files with them rather than using my own FAT code.

You're clearly capable of writing your own drivers to access disks and screen. So why don't you do that? Then it won't matter what the firmware is doing because you won't be relying on the firmware.

I would like there to be lots of Win32 clones. They don't have to come from me. Vendors are shipping an entire OS these days (called UEFI), and I am after minimal code (bootx64.efi) to convert that into a Win32 clone. I will provide one such example. If others (even me) wish to provide a more complicated Win32 implementation, including their own FAT drivers etc - that's fine. The more options to run Win32 executables the better. And I want Microsoft's monopoly to be ended and there to be a "Win32 consortium" that ensures Win32 OSes progress together. And ideally this process would have started in 1986, but it's better late than never (in my opinion - maybe not anyone else's). I bought ArcaOS as a vehicle for running Win32 executables too.

[davmac314]

... I will (nominally) only do one, for proof of concept.

...

As per above - it would be the vendor making this minimal change, not me.

No, as per above, you were going to modify the firmware for one system yourself (the proof-of-concept). That is what my question was about. How are you going to do this?

No, why run Win32 software under 64-bit UEFI? Why not just compile it as a 64-bit UEFI application, or even as a Win64 application? In short: why's it so important that you can run a single executable on two very different architectures?

What I want is programming rules issued in 1986 that would allow me to future-proof my executables. I have also constructed rules for the mainframe to do the same thing. The mainframe was much easier - they didn't do what x64 did - they didn't invalidate any 32-bit instructions. It's quite odd to me to see instructions invalidated - but if that's what manufacturers are allowed to do (or, in 1986, going to do), so be it. I'll write my software according to the rules.

I expect to build a PE32+ executable in 1986, according to the rules (rules which were formulated between 1980 and 1985), and have it work forever. I do not expect to be forced to recompile it post-1986.

"Forever" will be "until LM64 is permanently retired".

I'm going to guess that what you really mean is that you want a set of rules that could have been issued in 1986, not that you actually have a time machine and intend to use it. But that doesn't answer the underlying question. Why? What's the point of all this? Especially given 1986 is long gone?

[kerravon]

... I will (nominally) only do one, for proof of concept.

...

As per above - it would be the vendor making this minimal change, not me.

No, as per above, you were going to modify the firmware for one system yourself (the proof-of-concept). That is what my question was about. How are you going to do this?

I haven't investigated it much, but I believe Mr Chromebox has an open source UEFI implementation to be flashed on to modern Chromebooks. So - I would download that source code, build it, find out where it detects how much available memory there is, change that to ensure the maximum is 4 GiB (maybe 1 line of code), then change the paging to get 4-8 GiB mapped to 0-4 GiB (I've never done this before, but I suspect that it will be about 50 lines of code). I believe this code is still maintained, so I would join the appropriate forum and simply ask "where do I put this 1 + 50 lines of code?". They probably won't support my goal, and nor will they write the code themselves, but they will likely point me to the right source code anyway, minimizing my effort.

I'm going to guess that what you really mean is that you want a set of rules that could have been issued in 1986, not that you actually have a time machine and intend to use it. But that doesn't answer the underlying question. Why? What's the point of all this? Especially given 1986 is long gone?

I really haven't written much code. I've been trying to sort out the rules since 1986 (or close). I bought my PC XT in 1987, but I didn't learn MSDOS programming. I stuck to pure C90. I am not wedded to any particular compiler because I haven't yet decided what code I want generated.

The fact that it has taken me 37 years and counting to try to get the infrastructure in place does not change the actual goal started 37 years ago.

Note that it is only in the last approximately one year that I have even done UEFI programming. I've been deliberately sticking to the BIOS all this time. Not exceeding the architecture available in 1986. Also with an eye on eventually porting to the Amiga so that the Amiga takes over from the PC.

The fact that Commodore went bankrupt doesn't perturb me.

My software is all text, and uses ANSI controls for fullscreen apps (including UEFI apps). I should be ready for the Amiga.

I can remember someone complaining to me that I was producing MSDOS executables and not Amiga executables (although I did produce some) - even though I had an Amiga.

I'm still not yet ready to produce Amiga executables. That will come after I have 32-bit x86 executables that I am happy with.

And that won't happen until I have a compiler that I am happy with.

In more than 50 years, no-one has produced a public domain C90 compiler, which is what I am after. There are multiple lines of attack, and we are getting there.

Why do people physically climb mountains? If I wanted to see the top of a mountain (I don't), I would hire a helicopter.

Those (strange to me) people probably think I am strange too - fighting 1980s wars. Other people reenact wars from centuries ago.

My goal will at least produce (and in fact, has already produced) a standalone public domain operating system. The mountain that was physically climbed produced nothing at all. People actually buy crossword puzzle books, solve them, and then the book goes in the bin. But no-one ever calls them out on it.

This entire forum is full of people who are producing operating systems that they are confident will never be used by anyone. But they're not calling themselves out on that.

Personally I do not even consider myself to be in that group. I believe it is possible that my Win32 semi-clone (or a derivate, or something inspired by the proof of concept) will one day challenge Microsoft. And there will likely be Win64 clones too. Commercial and closed source. And a Win32 and Win64 consortium. I suspect it may well happen. In the same way that IBM lost control of PC hardware, I am expecting Microsoft to lose control of PC software.

Regardless, that's my hobby. Can you suggest a better hobby? You're presumably in this exact group because of a similar, but not identical, hobby.

[davmac314]

Why do people physically climb mountains? If I wanted to see the top of a mountain (I don't), I would hire a helicopter.

This part I think I comprehend. I don't pretend to understand the rest of your motivations, but I wish you well.

And there will likely be Win64 clones too. Commercial and closed source.

Indeed, there is already ReactOS, for one.

Regardless, that's my hobby. Can you suggest a better hobby? You're presumably in this exact group because of a similar, but not identical, hobby.

Your choice of hobby is yours to make. When I ask the question "why" it is to ascertain the underlying motivations, partly because I feel that you are setting artificial limitations that will limit what you can achieve, and it is common for people (in OS development) to set or operate under limitations due to a misconception, in which case it is usually good to correct it. In your case I have largely failed. It appears that the answer is in part "for the challenge" and in part "to topple Microsoft from dominance in the OS arena". Given the latter, I still don't understand why you wish to artificially constrain yourself to creating executables which will continue to run both on ancient hardware and in the constrained environment of x64 UEFI. Regardless, I wish you all the best.

[kerravon]

Indeed, there is already ReactOS, for one.

A company can't pick that up and make it commercial quality and close source it to protect their investment (if that's the business model that they innately think is going to turn a profit). I'm creating something that can be picked up with no strings attached.

It appears that the answer is in part "for the challenge" and in part "to topple Microsoft from dominance in the OS arena". Given the latter, I still don't understand why you wish to artificially constrain yourself to creating executables which will continue to run both on ancient hardware and in the constrained environment of x64 UEFI. Regardless, I wish you all the best.

I won't personally be doing the toppling (or at least it would be very unlikely). It will need someone like IBM or Fujitsu. I don't expect them to constrain themselves at all. As for me restraining myself - that's not quite true either. I already said that CM32 is the better choice so I will probably be going for that option. And I have already said that I am in the process of creating Win64 executables. I spent all today trying to resolve a code generation error in cc64 (in the PDOS source tree). What I said is that I would like universal 32-bit executables (constrained to work on both 80386 and LM64) as an OPTION. I have no idea what environment that may be useful in - maybe Fujitsu wants to go in that direction for some reason to do with chip fabrication - I just want to prove the technology (including negative indexes and scaling which were only teased out in the last few days thanks to people in this forum). MVS/380 needed to be proven before people believed it too. Then they still didn't believe that VM/380 was possible until I proved that. Undaunted, they then didn't believe VSE/380 was possible. Until it was proven again. And it continued - AM32, accessing 4 GiB, running on z/Arch. Every single thing had to be proven. Even when it was proven on Hercules they still didn't believe it would work on real hardware so I had to find someone with real hardware. I even had someone who was helping me write some code who insisted what I was asking for wouldn't work, but wrote the code anyway, under protest. When I showed him it finally working, he somehow missed what I had shown him and still insisted it wouldn't work, and I pointed out that I didn't expect him to still be saying that after I just showed it to him working.

I suspect a lot of people wouldn't believe (and may still not believe) that it is possible to build a 32-bit executable that works on both an 80386 and in LM64 if you follow certain rules. And in fairness, I don't know myself whether it is possible or not, because I've only demonstrated a single small executable - I don't know what 80386 instructions stop working on LM64 because I know very little about LM64. What I do know is that in the free marketplace of ideas (ie this forum), no-one has been able to name a specific show-stopping instruction. Before I personally thought that without "push eax" etc, LM64 was impossible. It took a long time before I thought of a workaround for that. But the fact that no-one here has named an instruction is not definitive (it does give me a lot of confidence though).

[davmac314]

I suspect a lot of people wouldn't believe (and may still not believe) that it is possible to build a 32-bit executable that works on both an 80386 and in LM64 if you follow certain rules. And in fairness, I don't know myself whether it is possible or not, because I've only demonstrated a single small executable - I don't know what 80386 instructions stop working on LM64 because I know very little about LM64. What I do know is that in the free marketplace of ideas (ie this forum), no-one has been able to name a specific show-stopping instruction. Before I personally thought that without "push eax" etc, LM64 was impossible. It took a long time before I thought of a workaround for that. But the fact that no-one here has named an instruction is not definitive (it does give me a lot of confidence though).

You may have mentioned it earlier, but I would assume push/pop can be replaced just by loads/stores followed or preceded by addition/subtraction to the stack pointer for example. So "push eax" could be replaced with:

Code: Select all

sub esp, 4
mov [esp], 4

Call instructions will push a 64-bit return value which if not dealt with would be be slightly problematic, though. A function which takes parameters on the stack will expect them to be at a certain offset from the stack pointer on function entry, which would normally be different when running in 32-bit vs 64-bit mode due to a differently-sized return address being pushed by the call. Even this is probably not intractable; you could perform a fixup of the stack pointer as first thing to ensure that it is always 8-byte aligned for example (assuming that it is also 8-byte aligned at the call), maybe pushing (not with a 'push' instruction!) an address containing a 'ret' instruction to pad out a 32-bit EIP to a 64-bit RIP, meaning that a 'ret' executed in 32-bit mode will immediately then perform a 2nd 'ret' and get back to the caller that way. All this is assuming you are writing or modifying your own compiler of course.

So, no, I don't believe it is impossible at all. My main doubt is whether it would be worthwhile. But you are certainly free to make your own judgement regarding that.

[kerravon]

Call instructions will push a 64-bit return value which if not dealt with would be be slightly problematic, though. A function which takes parameters on the stack will expect them to be at a certain offset from the stack pointer on function entry, which would normally be different when running in 32-bit vs 64-bit mode due to a differently-sized return address being pushed by the call. Even this is probably not intractable; you could perform a fixup of the stack pointer as first thing to ensure that it is always 8-byte aligned for example (assuming that it is also 8-byte aligned at the call), maybe pushing (not with a 'push' instruction!) an address containing a 'ret' instruction to pad out a 32-bit EIP to a 64-bit RIP, meaning that a 'ret' executed in 32-bit mode will immediately then perform a 2nd 'ret' and get back to the caller that way. All this is assuming you are writing or modifying your own compiler of course.

If I understand your proposal, that sounds like conditional execution.

As per the first message in this thread, this is what I did:

https://sourceforge.net/p/pdos/gitcode/ ... demo32.asm

So, no, I don't believe it is impossible at all.

The negative index issue was a show-stopper for a long time (year/years - can't remember - but could look it up) on the mainframe before someone suggested using paging.

My main doubt is whether it would be worthwhile. But you are certainly free to make your own judgement regarding that.

Someone once asked me why I was spending so much effort writing something (a C runtime library) that came free with every compiler. In the mid-late 1990s when I accidentally got access to a mainframe, I made an initial port to MVS (mainframe) - and once again, not for any definite purpose. In 1998-1999 I made an attempt to get the i370 target of gcc to work on EBCDIC, but failed. Then Hercules was created. Then someone else got gcc to work on the mainframe in EBCDIC (using IBM's C compiler and IBM's C runtime). At that point no-one knew whether GCC could run on the free MVS 3.8J (restricted to 16 MiB - more like 8 or 9 MiB). We didn't know how big the executable would be, and we didn't know how much runtime memory it would use, and we didn't know whether any other environmental restrictions existed (the assembler did indeed run out of symbols or something - but someone updated the assembler - on VSE we ran into a linker relocation problem - bypassed by using a fixed memory address). It was all made to work, and people had a free C compiler available, which opened up other development, such as REXX on VM/370. In actual fact my initial interest was only MVS, and I was surprised when someone from the VM side submitted code changes to PDPCLIB so that it would work there too.

I didn't know any of this when I started writing PDPCLIB in 1994 and I didn't really know how to answer the original question. Although I did have a bit of an answer - I wanted to know why people were coding the non-C90 open() instead of fopen() and wanted to see if I could create an fopen() that prioritized fast access to binary files so that no-one had a reason to use the non-C90 call. I don't think anyone actually used it for that original intended purpose.

[davmac314]

The negative index issue was a show-stopper for a long time (year/years - can't remember - but could look it up) on the mainframe before someone suggested using paging.

At least for the case discussed in this thread (i386 code in x64 environment, plus you control the compiler) - you could just avoid generating them.

[Octocontrabass]

Vendors are shipping an entire OS these days (called UEFI), and I am after minimal code (bootx64.efi) to convert that into a Win32 clone.

But a Win32 clone can run Win32 binaries. Win32 binaries won't run in your environment because your environment is limited in ways Windows is not. You've come up with a different environment and a way to build binaries that run in both Win32 and your environment.

It will need someone like IBM or Fujitsu.

In 1986, they might have been interested in a standard API for PC programs across operating systems. In 2023, they'll use Javascript.

I suspect a lot of people wouldn't believe (and may still not believe) that it is possible to build a 32-bit executable that works on both an 80386 and in LM64 if you follow certain rules.

I have no doubt that it's possible. Whether it's useful is a different question. If you're still writing software like it's 1986, you don't need to worry about future CPUs removing support for compatibility mode because current CPUs are already fast enough to emulate an entire 1986 PC in Javascript. I suspect any 32-bit software from today will run perfectly fine through similar emulation if and when future CPUs remove support for 32-bit compatibility mode.

[kerravon]

The negative index issue was a show-stopper for a long time (year/years - can't remember - but could look it up) on the mainframe before someone suggested using paging.

At least for the case discussed in this thread (i386 code in x64 environment, plus you control the compiler) - you could just avoid generating them.

On the mainframe, where I controlled the compiler (but didn't know how to update - and no-one was willing to assist), I tried to do exactly that - generate an addition/subtraction instead of using an index register. Finally someone came along who suggested a way that that might have been possible (peephole optimization), and I went to try to change the code generation. While doing that, I realized that it wasn't just that one case in question that an index variable could be used. I can't remember exactly - it could have been the LA instruction - I think maybe even without an index variable this would have failed when an external variable happened to be negative. As such, I would have needed to check the reference to every single variable to see if it was negative - in fact, maybe not even that would have been enough - a positive value would need to be manually truncated too. The use of negative/wrap is so fundamental that it ceased to be a C compiler generating assembler code, it became some sort of interpreter that was being generated, which was too far away from what I actually wanted - I may as well simply run a real interpreter. I think I posted in a group that I had given up because of this, and that is when finally someone had the idea of using paging, so that the compiler didn't need to be changed one iota.

I don't know if this is the same for the x64 - I'm not really familiar with that.

[kerravon]

Vendors are shipping an entire OS these days (called UEFI), and I am after minimal code (bootx64.efi) to convert that into a Win32 clone.

But a Win32 clone can run Win32 binaries.

Even Linux Wine doesn't run ALL Win32 binaries. I assume ReactOS doesn't run ALL of them either.

Win32 binaries won't run in your environment

Some will, some won't. If people don't "follow the rules" (not really their fault - it took 37 years to write them), that's on them, not me.

because your environment is limited in ways Windows is not. You've come up with a different environment and a way to build binaries that run in both Win32 and your environment.

My environment is a Win32 subset. Not a full clone. Not unusual. And nothing wrong with that. It takes a lot of effort to match the market leader. Anyone who is interested in having the broadest acceptance of their application binaries needs to look (and preferably test) ALL the Win32 partial-clones on the market and write for the lowest common denominator (or whatever they wish to support).

It will need someone like IBM or Fujitsu.

In 1986, they might have been interested in a standard API for PC programs across operating systems. In 2023, they'll use Javascript.

Don't be so sure that everyone is happy to do interpretation and take a corresponding speed loss. I'm not sure anyone knows what 100% of the market is currently, and be able to extrapolate the future based on that with 100% certainty.

I suspect a lot of people wouldn't believe (and may still not believe) that it is possible to build a 32-bit executable that works on both an 80386 and in LM64 if you follow certain rules.

I have no doubt that it's possible.

I'm not talking about interpretation. Obviously that was always possible.

Whether it's useful is a different question. If you're still writing software like it's 1986, you don't need to worry about future CPUs removing support for compatibility mode because current CPUs are already fast enough to emulate an entire 1986 PC in Javascript. I suspect any 32-bit software from today will run perfectly fine through similar emulation if and when future CPUs remove support for 32-bit compatibility mode.

I don't want to rely on CPUs being fast enough for all applications. BTW - I have to do interpretation (using Hercules) to run 32-bit mainframe code. It takes something like an hour for GCCMVS to rebuild itself under emulation. I don't think modern computers are so fast that emulation is a universal solution that everyone is happy with.

And speed isn't the only consideration. Power consumption is (or used to be) another. You need(ed) to cut a human open to replace a pacemaker when the battery runs out.

[Octocontrabass]

Even Linux Wine doesn't run ALL Win32 binaries.

Neither does Windows!

Some will, some won't. If people don't "follow the rules" (not really their fault - it took 37 years to write them), that's on them, not me.

The same is true of Wine, except that Wine's rules are the same as Windows's rules. Programs that break the rules tend to work in Windows and not Wine because those programs are only tested in Windows.

Don't be so sure that everyone is happy to do interpretation and take a corresponding speed loss.

Anyone unhappy with the speed of modern Javascript compilers will also be unhappy with the speed of polyglot code.

I'm not talking about interpretation.

Neither am I.

I don't think modern computers are so fast that emulation is a universal solution that everyone is happy with.

It's not supposed to be a universal solution. It's a solution for obsolete software that can't be recompiled to work on modern CPU architectures.

[kerravon]

It's not supposed to be a universal solution. It's a solution for obsolete software that can't be recompiled to work on modern CPU architectures.

This is the thing though - I don't expect my binaries to have to be recompiled. I'm expecting the vendor to provide upward compatibility.

That's how it is on the mainframe - my 32-bit S/370 (or even S/360) software runs in AM64 with no modification required. There is some quibbling about that, but the generated S/370 instructions that gcc started generating in literally 1989:

C:\devel\gcc\gcc\config\i370>grep Copyright i370.md
i370.md: ;; Copyright (C) 1989, 1993, 1994, 1995, 1997, 1998, 1999, 2000, 2002

C:\devel\gcc\gcc\config\i370>

run with no change (unless you want to quibble about bugs)

Ok, so I want to do the same thing in the same timeframe. AMD made it real difficult, but not impossible. For some reason they obsoleted a lot of instructions. They could have mentioned that in some sort of roadmap. IBM didn't do that with the mainframe. The instructions continue to work just fine. I'm not aware of a single user-space instruction that was obsoleted when you activate AM64 or z/Arch mode.

Note that that i370.md could have been written in 1965. There is almost no change between S/360 and S/370 (well - that i370.md did in fact generate some of the handful (maybe just 2?) of instructions that weren't in S/360, ie MVCL and CLCL).

[Octocontrabass]

This is the thing though - I don't expect my binaries to have to be recompiled. I'm expecting the vendor to provide upward compatibility.

How many vendors have intentionally provided a 64-bit instruction set that's binary-compatible with 32-bit applications? The only one I know of is MIPS. You've come up with interesting workarounds for other architectures, but it only works by coincidence, not by design.

[kerravon]

This is the thing though - I don't expect my binaries to have to be recompiled. I'm expecting the vendor to provide upward compatibility.

How many vendors have intentionally provided a 64-bit instruction set that's binary-compatible with 32-bit applications? The only one I know of is MIPS. You've come up with interesting workarounds for other architectures, but it only works by coincidence, not by design.

You don't accept IBM mainframes as an example?

If you do, then why is the MIPS/IBM way not the "correct way to do things"?