32-bit kernel that runs 64-bit apps

[milouz]

Hi,

I thought that it was impossible to have a 32-bit kernel running 64-bit apps... until I read that some versions of Mac OS X kernel, despite being 32-bit, can effectively run 64-bit apps

Does anyone knows a little bit more about this ? I really can't figure out how it can be possible

[mallard]

Mac OS X effectively has two main parts to the kernel; the Mach microkernel and the BSD layer. (See here, particularly the "Architecture" section).

Since Mach handles memory management, scheduling, etc, it would need to be 64-bit, but Apple may have kept the BSD layer 32-bit initially, especially if they used PAE to access "high" RAM from 32-bit code.

[Combuster]

I think that the point in question is actually a reference to Rosetta emulating 64-bit PowerPC apps on a 32-bit intel machine, which is basically a form of cheating.


You could perform a completely different "cheat" on an intel machine by having a 32-bit kernel and still entering long mode and set up enough handling to route all incoming requests to 32-bit handlers - so you can run most of your kernel on 32-bit-only processors, but still take advantage of 64-bit mode when it's available.

[iansjack]

I think that the point in question is actually a reference to Rosetta emulating 64-bit PowerPC apps on a 32-bit intel machine, which is basically a form of cheating.

I'm pretty sure that is not true. The 32-bit Intel kernel could run 64-bit Intel userspace applications. And the real question is why shouldn't it be able to do so?

[jbemmel]

For x86, there is a bit in the code segment descriptor which determines whether it contains 32-bit or 64-bit code. It is possible to have a 32-bit kernel segment with 64-bit application segments

[iansjack]

Hi,

I thought that it was impossible to have a 32-bit kernel running 64-bit apps... until I read that some versions of Mac OS X kernel, despite being 32-bit, can effectively run 64-bit apps

Does anyone knows a little bit more about this ? I really can't figure out how it can be possible

A few more details here: http://hothardware.com/News/Apples-64Bi ... lt/#!22CL2

But why shouldn't it be possible? What exactly leads you to think that a 32-bit kernel can't launch and run 64-bit programs? After all, a 64-bit kernel has no problem running 32-bit applications.

And to be clear, this wan't a trick. You genuinely got better performance running 64-bit applications than the 32-bit equivalents. Hardly surprising with all the extra registers and more efficient addressing available.

[Brendan]

Hi,

But why shouldn't it be possible? What exactly leads you to think that a 32-bit kernel can't launch and run 64-bit programs? After all, a 64-bit kernel has no problem running 32-bit applications.

In long mode all interrupt handlers have to be 64 bit (even when you're running 32-bit code). You could have 64-bit "stubs" that switch to 32-bit to work around that, but right from the start "100% pure 32-bit kernel" is impossible.

Next, what happens when you're running a 64-bit application and it causes a page fault? 32-bit code can't even access the highest bits of CR2 to find out what the problem was, and the paging structures are different anyway. You'd have to rewrite the page fault handler for 64-bit (and can't just have a 64-bit stub that calls old 32-bit code). You'd also need to rewrite the lower level parts of virtual memory management.

Then; everything that used any virtual address from user-space or returned virtual addresses to user-space will have to be changed to handle a 64-bit virtual addresses; including whatever structures you're using to keep track of "in progress" transfers to/from disk (swap, memory mapped files) and file IO and networking, etc. If you want to be able to debug and profile 64-bit applications then that includes the breakpoint and debugging exception handlers plus everything else that could be involved (performance monitoring counter code, last branch recording, etc).

Most of the scheduler would be fine; but whatever you're using to save the previous task's state and load the new task's state is going to have to handle "64-bit state". This might mean that (e.g.) if a 64-bit application blocks or something you switch to old 32-bit kernel code to decide which task to switch to, then switch to 64-bit code to do the actual task switch.

If it was a micro-kernel, then "minimum changes to support 64-bit applications" means you probably need to rewrite half the micro-kernel just to end up with something that resembles the Flying Spaghetti Monster.


Cheers,

Brendan

[iansjack]

Eppur si muove

It appears that they forgot to tell the Apple software engineers that it was impossible. And, as we all know, bumble bees are incapable of flight.

[Brendan]

Hi,

It appears that they forgot to tell the Apple software engineers that it was impossible. And, as we all know, bumble bees are incapable of flight.

More likely is that Apple software engineers know that it wasn't a 32-bit kernel and that Apple's marketing people lie. For example, maybe it was actually a 64-bit "darwin core" with 32-bit "old puss written by Apple" slapped on it with minor changes.


Cheers,

Brendan

[iansjack]

I can't think of a plausible explanation as to why Apple's marketing people should pretend that the kernel was 32-bit when it wasn't, especially as 64-bit was then the best thing since sliced bread. And if the kernel wasn't 32-bit then it wouldn't have worked with 32-bit device drivers, which was the whole point of the excercise. Anyone who used XP_64 knows how important that point was. They had earlier followed the same procedure with PPC versions of OS X. The excellent Mac OS X Internals explains the process.

Just because you don't understand how something is done doesn't mean that it can't be done.

[Brendan]

Hi,

I can't think of a plausible explanation as to why Apple's marketing people should pretend that the kernel was 32-bit when it wasn't, especially as 64-bit was then the best thing since sliced bread. And if the kernel wasn't 32-bit then it wouldn't have worked with 32-bit device drivers, which was the whole point of the excercise. Anyone who used XP_64 knows how important that point was. They had earlier followed the same procedure with PPC versions of OS X. The excellent Mac OS X Internals explains the process.

Just because you don't understand how something is done doesn't mean that it can't be done.

It's impossible to have 32-bit interrupt handlers in long mode, impossible for old "32-bit only" code to handle (e.g.) PML4, and impossible for old "32-bit only" code to save and restore a 64-bit task's state.

Now; are you denying these facts, or are you self-deluded?


Cheers,

Brendan

[iansjack]

Now; are you denying these facts, or are you self-deluded?

Clearly, along with Apple's software engineers, I am deluded to believe that they did something that they did.

There's is little point in continuing to argue as to whether bumble bees can fly, so I'll leave you with your certainties.

Meanwhile, in the real world ... Eppur si muove

[Antti]

I thought that it was impossible to have a 32-bit kernel running 64-bit apps

What you thought was more right than wrong but there is no correct answer. It all depends on how we define this whole issue. If we emulated 64-bit apps, it would be possible. If we used "native instructions", it would not be possible. See the debate above.

Being able to run 64-bit apps natively means that the kernel is very aware of "64-bit" being present. Being in Long Mode, the kernel could run instructions that are 16-bit, 32-bit or 64-bit. All the things required for being in this mode mean that the kernel effectively is a "64-bit kernel". The bitness of the actual instructions is irrelevant.

[Combuster]

So basically, the cheat is that Mac OS comes with two kernels, one 32-bit and one true 64-bit. Since binaries are actually universal, the 32-bit kernel simply runs the 32-bit binaries from the same package, and prefer the 64-bit ones on a 64-bit kernel.

So the real deal is that a 32-bit kernel can still run 64-bit-enabled apps because all apps also happen to ship with 32-bit code in the same package.

[iansjack]

So basically, the cheat is that Mac OS comes with two kernels, one 32-bit and one true 64-bit. Since binaries are actually universal, the 32-bit kernel simply runs the 32-bit binaries from the same package, and prefer the 64-bit ones on a 64-bit kernel.

No, that is completely wrong. The kernel comes in 32-bit and 64-bit versions (obviously the 32-bit one is 64-bit aware). The 32-bit kernel runs in 32-bit mode, and so can use 32-bit kernel extensions, but can run applications in 64-bit mode. It doesn't pick the 32-bit version from the universal binary but uses the 64-bit version, with all the advantages that entails. The 64-bit kernel cannot use 32-bit kernel extensions and so, in the early days when most extensions only existed in 32-bit form, initially had little use. By default it was not enabled, though you could choose to do so (depending upon the model of your Mac).

This is not any sort of trickery, obfuscation, or smoke and mirrors; it's just the way things were. It was a very smart move on Apples's part as those who adopted XP64 in the early days could tell you. Of course they had a head start on Microsoft as they had already been through the same process with the PPC. How it all works is a matter that is well documented. One of the salient points is that applications and the kernel don't pass pointers to each other. Apart from anything else, the source code of those versions of the Darwin kernel is available for those with an open mind to study.

Nowadays (I believe) the 32-bit kernel is history.