How to make safe updates of descriptors

Discussions on more advanced topics such as monolithic vs micro-kernels, transactional memory models, and paging vs segmentation should go here. Use this forum to expand and improve the wiki!
Post Reply
rdos
Member
Member
Posts: 3289
Joined: Wed Oct 01, 2008 1:55 pm

How to make safe updates of descriptors

Post by rdos »

I have an issue with making updates of size & base of a selector while other cores might use or load it. I think I solved the forced reload part by using the TLB shoot-down logic, but updating the descriptor is tricky too.

The main issue is that a descriptor is 8 bytes, but in protected mode you cannot make 64-bit atomic writes. My idea is that I first set the access byte to not-present, update base and size, and then restore the access byte. If another core happens to load the selector while the access byte indicates non-present, the non-present exception should happen, and in the exception handler I could simply check if the descriptor is really non-present, in which case the fault processing continues, or if it is present now, just re-execute the instruction.

Any better solution to this?
Octocontrabass
Member
Member
Posts: 5546
Joined: Mon Mar 25, 2013 7:01 pm

Re: How to make safe updates of descriptors

Post by Octocontrabass »

rdos wrote:The main issue is that a descriptor is 8 bytes, but in protected mode you cannot make 64-bit atomic writes.
If your CPU supports CMPXCHG8B, any aligned 64-bit write will be atomic. Same for aligned 64-bit reads.

GCC and Clang will use (V)MOVQ, (V)MOVLPS, FILD/FISTP, and LOCK CMPXCH8B to implement atomic reads and writes according to the available instruction sets and microarchitecture optimizations.

If you use an instruction without a LOCK prefix to implement an atomic write, you may need to follow it with a store buffer flush to ensure other processors will see it at the expected time. GCC and Clang use MFENCE or "lock or [esp], 0" to implement the strongest sequential ordering; I didn't check to see which instructions they use for weaker orderings.
rdos
Member
Member
Posts: 3289
Joined: Wed Oct 01, 2008 1:55 pm

Re: How to make safe updates of descriptors

Post by rdos »

cmpxchg8b is not available in processors before Pentium, and I think I want to be compatible at least back to 486DX, but ideally to 386SX. OTOH, I would only need this on multicore processors, and cli/sti around the descriptor modification code should be enough on non-multicore.

fild and fistp is an interesting alternative that should work.

I don't know how the processor handles selector loads, but I suppose the normal locks should be used for bus-transactions, and the processor probably needs to read the complete 8 byte descriptor as a locked operation. If that is the case, then it should work.
reapersms
Member
Member
Posts: 48
Joined: Fri Oct 04, 2019 10:10 am

Re: How to make safe updates of descriptors

Post by reapersms »

Anything old enough to not have cmpxchg8b is exceedingly unlikely to be multi-processor or multicore. In particular, any of those systems would likely have rather different boot process requirements. MPS didn't exist until around '94. 486s would be covered, just barely.

Aiming for a single system binary to cover the entire 32-bit range from 386 to late Core is likely more trouble than it is worth.

I'd say modifying GDT entries should be rare enough that a full, cross-CPU lock might not be particularly noticeable, and LDT entry modification would be restricted enough to just the process in question, and modifications there would already be in kernel code, so halting all scheduling for that process would probably be fine -- but I don't know how well that assumption tracks with your segment-heavy approach.

-- addendum

The issues you can run into here are part of why Linux dropped 386 support back in 4.8; maintaining the extra SMP related code became more trouble than it was worth.
Octocontrabass
Member
Member
Posts: 5546
Joined: Mon Mar 25, 2013 7:01 pm

Re: How to make safe updates of descriptors

Post by Octocontrabass »

rdos wrote:cmpxchg8b is not available in processors before Pentium, and I think I want to be compatible at least back to 486DX, but ideally to 386SX. OTOH, I would only need this on multicore processors, and cli/sti around the descriptor modification code should be enough on non-multicore.
Given the incredible rarity of multiprocessor 386 and 486 systems, I don't think you need to worry about those. (And even if you find one, odds are it won't follow the MP specification, so good luck.)
rdos wrote:I don't know how the processor handles selector loads, but I suppose the normal locks should be used for bus-transactions, and the processor probably needs to read the complete 8 byte descriptor as a locked operation. If that is the case, then it should work.
As long as it's only reading the descriptor it should be fine. If the CPU needs to update the "accessed" bit then it won't work.
reapersms wrote:I'd say modifying GDT entries should be rare enough
In most OSes, yes, but I don't think we're talking about most OSes here.
reapersms wrote:The issues you can run into here are part of why Linux dropped 386 support back in 4.8; maintaining the extra SMP related code became more trouble than it was worth.
Linux never supported SMP 386 systems. The extra code was for working around missing features like CR0.WP or INVLPG.
rdos
Member
Member
Posts: 3289
Joined: Wed Oct 01, 2008 1:55 pm

Re: How to make safe updates of descriptors

Post by rdos »

reapersms wrote:Anything old enough to not have cmpxchg8b is exceedingly unlikely to be multi-processor or multicore. In particular, any of those systems would likely have rather different boot process requirements. MPS didn't exist until around '94. 486s would be covered, just barely.

Aiming for a single system binary to cover the entire 32-bit range from 386 to late Core is likely more trouble than it is worth.
It depends on how the binary is created and how syscalls are implemented.

For instance, I handle PAE vs protected mode paging by patching a call table to cover the relevant type of paging depending on processor type, highest RAM address and a few other parameters. I have an unitifed API towards the paging system, and isolating the details has clear benefits.
reapersms wrote: I'd say modifying GDT entries should be rare enough that a full, cross-CPU lock might not be particularly noticeable, and LDT entry modification would be restricted enough to just the process in question, and modifications there would already be in kernel code, so halting all scheduling for that process would probably be fine -- but I don't know how well that assumption tracks with your segment-heavy approach.
A lock is not possible because I cannot take a lock everytime I load a segment register. The issue is with one core modiying the GDT entry while another core loads a selector with the entry. The modification of the entry would be done when the 4k area for allocation is exhausted so another 4k of memory needs to be added. This would change the size, but also likely the base, of the descriptor.

Speed is not an issue, but selector loads must not result in exceptions or malfunctioning code, regardless of how unlikely it is.
reapersms wrote: The issues you can run into here are part of why Linux dropped 386 support back in 4.8; maintaining the extra SMP related code became more trouble than it was worth.
My impression is that Linux 32-bit support was rather poorly designed, particularly in regards to using available physical memory in an effective way.

I've redesigned my physical memory manager and now also the disc cache & how the VFS works to avoid unneccesary mapping of buffers in kernel space.
Last edited by rdos on Wed Feb 22, 2023 2:51 pm, edited 2 times in total.
rdos
Member
Member
Posts: 3289
Joined: Wed Oct 01, 2008 1:55 pm

Re: How to make safe updates of descriptors

Post by rdos »

Octocontrabass wrote:
rdos wrote:cmpxchg8b is not available in processors before Pentium, and I think I want to be compatible at least back to 486DX, but ideally to 386SX. OTOH, I would only need this on multicore processors, and cli/sti around the descriptor modification code should be enough on non-multicore.
Given the incredible rarity of multiprocessor 386 and 486 systems, I don't think you need to worry about those. (And even if you find one, odds are it won't follow the MP specification, so good luck.)
I support systems that use PIC and have no MP support, and I don't need to recompile or relink that kernel for that. The scheduler have a few primitives that either use spinlocks or cli/sti, depending on the number of cores in the system. Some syscalls are linked to different versions based on multicore or single core operation. I also load different hardware drivers, like PIC driver vs APIC driver.
Octocontrabass wrote:
rdos wrote:I don't know how the processor handles selector loads, but I suppose the normal locks should be used for bus-transactions, and the processor probably needs to read the complete 8 byte descriptor as a locked operation. If that is the case, then it should work.
As long as it's only reading the descriptor it should be fine. If the CPU needs to update the "accessed" bit then it won't work.
Right, so the accessed bit should be set to avoid processor updates of it.
Octocontrabass wrote: Linux never supported SMP 386 systems. The extra code was for working around missing features like CR0.WP or INVLPG.
I have variants of the TLB flush code. The 386 variant needs to reload CR3, while on 486 or higher invlpg is used for minor changes, and reloading CR3 for major. This is handled by linking the flushtlb syscall to different variants based on processor type.
Post Reply