How to make safe updates of descriptors

[rdos]

I have an issue with making updates of size & base of a selector while other cores might use or load it. I think I solved the forced reload part by using the TLB shoot-down logic, but updating the descriptor is tricky too.

The main issue is that a descriptor is 8 bytes, but in protected mode you cannot make 64-bit atomic writes. My idea is that I first set the access byte to not-present, update base and size, and then restore the access byte. If another core happens to load the selector while the access byte indicates non-present, the non-present exception should happen, and in the exception handler I could simply check if the descriptor is really non-present, in which case the fault processing continues, or if it is present now, just re-execute the instruction.

Any better solution to this?

[Octocontrabass]

The main issue is that a descriptor is 8 bytes, but in protected mode you cannot make 64-bit atomic writes.

If your CPU supports CMPXCHG8B, any aligned 64-bit write will be atomic. Same for aligned 64-bit reads.

GCC and Clang will use (V)MOVQ, (V)MOVLPS, FILD/FISTP, and LOCK CMPXCH8B to implement atomic reads and writes according to the available instruction sets and microarchitecture optimizations.

If you use an instruction without a LOCK prefix to implement an atomic write, you may need to follow it with a store buffer flush to ensure other processors will see it at the expected time. GCC and Clang use MFENCE or "lock or [esp], 0" to implement the strongest sequential ordering; I didn't check to see which instructions they use for weaker orderings.

[rdos]

cmpxchg8b is not available in processors before Pentium, and I think I want to be compatible at least back to 486DX, but ideally to 386SX. OTOH, I would only need this on multicore processors, and cli/sti around the descriptor modification code should be enough on non-multicore.

fild and fistp is an interesting alternative that should work.

I don't know how the processor handles selector loads, but I suppose the normal locks should be used for bus-transactions, and the processor probably needs to read the complete 8 byte descriptor as a locked operation. If that is the case, then it should work.

[reapersms]

Anything old enough to not have cmpxchg8b is exceedingly unlikely to be multi-processor or multicore. In particular, any of those systems would likely have rather different boot process requirements. MPS didn't exist until around '94. 486s would be covered, just barely.

Aiming for a single system binary to cover the entire 32-bit range from 386 to late Core is likely more trouble than it is worth.

I'd say modifying GDT entries should be rare enough that a full, cross-CPU lock might not be particularly noticeable, and LDT entry modification would be restricted enough to just the process in question, and modifications there would already be in kernel code, so halting all scheduling for that process would probably be fine -- but I don't know how well that assumption tracks with your segment-heavy approach.

-- addendum

The issues you can run into here are part of why Linux dropped 386 support back in 4.8; maintaining the extra SMP related code became more trouble than it was worth.

[Octocontrabass]

cmpxchg8b is not available in processors before Pentium, and I think I want to be compatible at least back to 486DX, but ideally to 386SX. OTOH, I would only need this on multicore processors, and cli/sti around the descriptor modification code should be enough on non-multicore.

Given the incredible rarity of multiprocessor 386 and 486 systems, I don't think you need to worry about those. (And even if you find one, odds are it won't follow the MP specification, so good luck.)

I don't know how the processor handles selector loads, but I suppose the normal locks should be used for bus-transactions, and the processor probably needs to read the complete 8 byte descriptor as a locked operation. If that is the case, then it should work.

As long as it's only reading the descriptor it should be fine. If the CPU needs to update the "accessed" bit then it won't work.

I'd say modifying GDT entries should be rare enough

In most OSes, yes, but I don't think we're talking about most OSes here.

The issues you can run into here are part of why Linux dropped 386 support back in 4.8; maintaining the extra SMP related code became more trouble than it was worth.

Linux never supported SMP 386 systems. The extra code was for working around missing features like CR0.WP or INVLPG.

[rdos]

Anything old enough to not have cmpxchg8b is exceedingly unlikely to be multi-processor or multicore. In particular, any of those systems would likely have rather different boot process requirements. MPS didn't exist until around '94. 486s would be covered, just barely.

Aiming for a single system binary to cover the entire 32-bit range from 386 to late Core is likely more trouble than it is worth.

It depends on how the binary is created and how syscalls are implemented.

For instance, I handle PAE vs protected mode paging by patching a call table to cover the relevant type of paging depending on processor type, highest RAM address and a few other parameters. I have an unitifed API towards the paging system, and isolating the details has clear benefits.

I'd say modifying GDT entries should be rare enough that a full, cross-CPU lock might not be particularly noticeable, and LDT entry modification would be restricted enough to just the process in question, and modifications there would already be in kernel code, so halting all scheduling for that process would probably be fine -- but I don't know how well that assumption tracks with your segment-heavy approach.

A lock is not possible because I cannot take a lock everytime I load a segment register. The issue is with one core modiying the GDT entry while another core loads a selector with the entry. The modification of the entry would be done when the 4k area for allocation is exhausted so another 4k of memory needs to be added. This would change the size, but also likely the base, of the descriptor.

Speed is not an issue, but selector loads must not result in exceptions or malfunctioning code, regardless of how unlikely it is.

The issues you can run into here are part of why Linux dropped 386 support back in 4.8; maintaining the extra SMP related code became more trouble than it was worth.

My impression is that Linux 32-bit support was rather poorly designed, particularly in regards to using available physical memory in an effective way.

I've redesigned my physical memory manager and now also the disc cache & how the VFS works to avoid unneccesary mapping of buffers in kernel space.

[rdos]

cmpxchg8b is not available in processors before Pentium, and I think I want to be compatible at least back to 486DX, but ideally to 386SX. OTOH, I would only need this on multicore processors, and cli/sti around the descriptor modification code should be enough on non-multicore.

Given the incredible rarity of multiprocessor 386 and 486 systems, I don't think you need to worry about those. (And even if you find one, odds are it won't follow the MP specification, so good luck.)

I support systems that use PIC and have no MP support, and I don't need to recompile or relink that kernel for that. The scheduler have a few primitives that either use spinlocks or cli/sti, depending on the number of cores in the system. Some syscalls are linked to different versions based on multicore or single core operation. I also load different hardware drivers, like PIC driver vs APIC driver.

I don't know how the processor handles selector loads, but I suppose the normal locks should be used for bus-transactions, and the processor probably needs to read the complete 8 byte descriptor as a locked operation. If that is the case, then it should work.

As long as it's only reading the descriptor it should be fine. If the CPU needs to update the "accessed" bit then it won't work.

Right, so the accessed bit should be set to avoid processor updates of it.

Linux never supported SMP 386 systems. The extra code was for working around missing features like CR0.WP or INVLPG.

I have variants of the TLB flush code. The 386 variant needs to reload CR3, while on 486 or higher invlpg is used for minor changes, and reloading CR3 for major. This is handled by linking the flushtlb syscall to different variants based on processor type.