alternatives to swapgs

[Ethin]

TBH I think AMD has the right of it in this instance. FRED seems neat but it adds even further complexity to the already over-complex x86 architecture, and we don't need more of that. I like how with AMD all one needs to do (from what I've read) is check to see if eSC is supported and set a bit in EFER. With FRED you have to set at least 4 other MSRs and completely redesign your interrupt handling code, and for people like me who aren't using C or asm, that either means waiting for LLVM or <insert compiler backend here> to implement that new ABI or mechanism to simplify the design (my code, for example, uses the x86-interrupt calling convention in LLVM) or we have to do other weird hacks to get it working. And that's not even adding in backwards compatibility either. That just means a lot more kernel code and a lot more points of failure unless one only wants to support FRED or the IDT mechanism. And we haven't even gotten started on all the code that will need to be completely rewritten should Intel make FRED the default (or even make it required) in the future.

[sj95126]

should Intel make FRED the default (or even make it required)

They still support the smsw instruction after telling us to stop using it in 1985. I don't think you have to worry.

[nullplan]

FRED seems neat but it adds even further complexity to the already over-complex x86 architecture, and we don't need more of that.

I disagree. I've just read the FRED proposal and it would massively reduce the complexity. You can throw away the IDT, including all the code to support it, and all existing entry stubs (OK, in practice you are going to need to keep those around for older CPUs). You only need two entry stubs, one for user space entry and one for kernel space entry. Since the FRED interrupt frame ends in the same structure as the normal interrupt frame, you do not need to redevelop all the exception handler functions. You can treat the first few bytes of the FRED frame like a new data structure and give that to a higher-level handler as second argument. Having to read CR2 or DR6 is no longer necessary, so no more special cases for page fault. And the stack level stuff in there prevents the problem you so often have with the IST: If the event does reoccur before the IRET instruction, your interrupt frame is now gone.

Oh, and the proposal makes it possible for kernel space to use the red zone. Of course, still not in practice, since we need to keep compat for older CPUs, but once FRED becomes ubiquitous, this becomes an option. OK, now to read the AMD proposal.

and for people like me who aren't using C or asm, that either means waiting for LLVM or <insert compiler backend here> to implement that new ABI or mechanism to simplify the design (my code, for example, uses the x86-interrupt calling convention in LLVM) or we have to do other weird hacks to get it working.

Well, I guess there's your problem. I have hand-written my entry stubs in assembler precisely because of this. I mean, how do you support SYSCALL? That's not supported in LLVM, as far as I know. You need to load the right stack pointer, and I wouldn't know how to communicate that to the compiler at all. I know that in writing an OS, I must accept at least a minimal amount of assembler code. Everything else will sooner or later turn out to be too limited.

It's unclear whether having two competing proposals is a good thing.

Yeah, given that AMD and Intel bicker like an old married couple, I assume this is going to end in us having to support two new mechanisms instead of one. It's going to be SYSCALL vs. SYSENTER reloaded.

[nullplan]

Having read the AMD proposal, it does appear a lot weaker than FRED to me. It only revamps SYSCALL/SYSRET, and adds a barbed wire to a configurable number of exception vectors. Now, instead of overwriting your IST stack frames, you can crash with a double fault! Not really an improvement, considering I cannot return from double fault. What I would want is a mechanism to make them actually stack exceptions onto the stack. Like, mark a stack as "in use", so RSP is not reset to TSS.ISTx, but it just stays where it is. That would allow nesting of those interrupts.

It does not address the need for conditional SWAPGS in interrupt entry/exit. Re-entrancy protection is weak. And I still don't know what all of that talk of shadow stacks is all about. In all, I much prefer FRED over the AMD proposal. And then they throw in chapter 5, where they finally fix an age-old bug in x86. Nice, but too little, too late. That ship has sailed, returned to port, and sailed again.

[Ethin]

For syscall I obviously will need to write some asm; the x86-interrupt convention isn't enough and I only receive the stack frame. I could (maybe) work with that... I'm not sure.
The reason I said that it would make the architecture even more complex is that yes, you'd need only two handlers, and you wouldn't need to read CR2 or any of the DR registers, but you'd still need 256 interrupt routines of some kind. Since this would be a completely new way of handling interrupts, and if the model for handling interrupts changed (e.g.: the data and way that an ISR is called right now is different in this new model, which it appears to be) I'd still need to either wait for LLVM to catch up (which usually doesn't take long but it might in this instance, its hard to know since this is still just a draft) or I'd need to write my two handlers in assembly to forward anything on to a handler stub in Rust that can then do the redirection to the real interrupt handler. Its hard to know though... This is still a draft, after all.
The shadow stack is stuff that has to do with SCET. Its designed to prevent ROP/JOP attacks.

[nullplan]

but you'd still need 256 interrupt routines of some kind.

Not so. You need exception handlers for all the exceptions you want to handle, and in practice, a lot of those are just "if userspace, signal process/raise exception, else panic". And then one handler for the external interrupts. This is the way I have set up my interrupt handling already. I do have 224 IRQ entry points, but each of them only pushes the interrupt number before transitioning to the common handler. And FRED would remove that responsibility completely, because now the interrupt number is part of the FRED frame.

Since this would be a completely new way of handling interrupts,

Again, not quite. Current stub pushes all registers, conditionally swaps GS, calls the handler, conditionally tests for late handling (signals and scheduling), conditionally swaps GS back, pops all registers, and IRET. New stub pushes all registers, calls a centralized handler (that can dispatch to the old handlers based on the FRED frame), then only the user space version tests for late handling, then both pop all registers, and ERETU/ERETS. The difference is, instead of using a macro to generate exception handler stubs all over the place, I only need two of them, and I can write the interrupt dispatcher in C instead of assembler. And you can probably write one in Rust, as long as you can call a Rust function from assembler.

(e.g.: the data and way that an ISR is called right now is different in this new model, which it appears to be)

Again, the FRED frame ends in an interrupt frame. Even though they do define a few more bits in there, you can continue to use your interrupt frame data structure. You just need a new one for the additional information.

I'd still need to either wait for LLVM to catch up [...] or I'd need to write my two handlers in assembly to forward anything on to a handler stub in Rust that can then do the redirection to the real interrupt handler

That is one reason I chose to write the interrupt entry stubs in assembler: I am not dependent on any compiler features there.

[sj95126]

I do have 224 IRQ entry points, but each of them only pushes the interrupt number before transitioning to the common handler. And FRED would remove that responsibility completely, because now the interrupt number is part of the FRED frame.

I can't believe they didn't do this from the very beginning. I think this might irk me more than any design flaw in x86. I hate that you have to create hundreds of stubs just to be able to log "something called int 0x9f by mistake."

It would have been so completely trivial to push the interrupt number on the stack along with CS:IP, FLAGS, and SS:SP. In fact, since they would have only needed 8 bits for the interrupt number, they could have used the rest as flags, including a flag that would indicate whether an error code is on the stack, instead of weirdness like #DF pushing an error code that's always 0.

[nexos]

I do have 224 IRQ entry points, but each of them only pushes the interrupt number before transitioning to the common handler. And FRED would remove that responsibility completely, because now the interrupt number is part of the FRED frame.

I can't believe they didn't do this from the very beginning. I think this might irk me more than any design flaw in x86. I hate that you have to create hundreds of stubs just to be able to log "something called int 0x9f by mistake."

It would have been so completely trivial to push the interrupt number on the stack along with CS:IP, FLAGS, and SS:SP. In fact, since they would have only needed 8 bits for the interrupt number, they could have used the rest as flags, including a flag that would indicate whether an error code is on the stack, instead of weirdness like #DF pushing an error code that's always 0.

I think I once spent an hour creating all my interrupt stubs. Why didn't Intel do something that is, well, very simple, and push the interrupt number? They are making our lives miserable!

[Korona]

It's not that bad though. Your IRQ stub will just push a constant to the stack and jump to a common handler. And with MSIs, you'll actually use a substantial amount of IRQ vectors anyway. For example, on this laptop, 76 IRQs are in use on a stock Linux installation.

[nullplan]

I can't believe they didn't do this from the very beginning.

Me neither. Although it wasn't a big problem in the beginning. With the 8259 PICs, the interrupts were all static. It wasn't until the advent of the IOAPIC that the number of interrupts became a runtime variable, and with the advent of MSI, the number of interrupts truly exploded.

In fact, since they would have only needed 8 bits for the interrupt number, they could have used the rest as flags, including a flag that would indicate whether an error code is on the stack, instead of weirdness like #DF pushing an error code that's always 0.

Or just push an error code in every exception. Or no exception, and have an error code register. Either way would be fine to have a uniform interrupt frame. But no, can't have that.

I think I once spent an hour creating all my interrupt stubs.

Honestly, I lifted mine from Linux. Oh, err... I mean, I was inspired by Linux. The idea is to encode the interrupt number in a signed byte, so we can always use the small push encoding. Which I do like this:

Code: Select all

.global irq_base
.align 8
irq_base:
.set irq, 32
.rept 256-32
  pushq $127-irq
  jmp irq_common
.align 8
.set irq, irq+1
.endr

Then in the common handler, I only need to reduce that variable by 127, and I have the negative interrupt number on stack.

And for the exceptions, I have an assembly label named for the exception, and a C function called the same thing, but with a "do_" in front, and the assembler stub calls the C function. And everything can be generated from one macro:

Code: Select all

.macro excpt label,push_error=1,read_cr2=0
\label:
.if \push_error
    pushq $0
.endif
    save_regs
    movq %rsp, %rdi
    movq %rsp, %r12 /* non-volatile register */
.if \read_cr2
    movq %cr2, %rsi
.endif
    andq $-16,%rsp
    xorq %r13,%r13
    testq $3, 18*8(%rsp)
    jz 1f
    incq %r13
    swapgs
1:
    cld
    call do_\label
    jmp interrupt_return
.endm

Oh yeah, and one more idea lifted from Linux: The interrupt return code can be unified into a single place, because after calling the specific handler, it is all the same.

[nexos]

I think I once spent an hour creating all my interrupt stubs.

Honestly, I lifted mine from Linux. Oh, err... I mean, I was inspired by Linux. The idea is to encode the interrupt number in a signed byte, so we can always use the small push encoding. Which I do like this:

Code: Select all

.global irq_base
.align 8
irq_base:
.set irq, 32
.rept 256-32
  pushq $127-irq
  jmp irq_common
.align 8
.set irq, irq+1
.endr

Then in the common handler, I only need to reduce that variable by 127, and I have the negative interrupt number on stack.

And for the exceptions, I have an assembly label named for the exception, and a C function called the same thing, but with a "do_" in front, and the assembler stub calls the C function. And everything can be generated from one macro:

Code: Select all

.macro excpt label,push_error=1,read_cr2=0
\label:
.if \push_error
    pushq $0
.endif
    save_regs
    movq %rsp, %rdi
    movq %rsp, %r12 /* non-volatile register */
.if \read_cr2
    movq %cr2, %rsi
.endif
    andq $-16,%rsp
    xorq %r13,%r13
    testq $3, 18*8(%rsp)
    jz 1f
    incq %r13
    swapgs
1:
    cld
    call do_\label
    jmp interrupt_return
.endm

Oh yeah, and one more idea lifted from Linux: The interrupt return code can be unified into a single place, because after calling the specific handler, it is all the same.

Wow, that's a great solution!

[sj95126]

And for the exceptions, I have an assembly label named for the exception, and a C function called the same thing, but with a "do_" in front, and the assembler stub calls the C function.

That is a good approach, but what I think I may end up doing is extending the same framework I used for handling IRQs, which is that there's a macro to generate each IRQ stub, which pushes the IRQ number and calls the common handler, which then calls the real IRQ handler (in C) from a jump table. Each driver registers its handler to be added to that table; extending the same for exceptions should work too. There's a bit of risk to the extra layer of indirection but compartmentalizing the code with sufficient protections should be ok (I may exempt #DF from it anyway).

[sj95126]

Well, soooo... we went off topic a little bit, but the answers to my original question ranged from "sure, do it however you like" to "swapgs isn't THAT bad." So I decided to just punt the issue for now.

I did get syscalls working without swapgs, by accessing the CPU struct with %rip-relative addressing. Since it already worked, I just decided to leave it there. I'll figure out how to handle multiple CPUs if and when I need to. Maybe I'll address it before FRED 2.0 comes out. Until then, I can use either one with a #define.

Code: Select all

#ifdef USE_SWAPGS
        swapgs
        movq    %rsp, %gs:CPU_STATE_SAVED_RSP
        movq    %gs:CPU_STATE_SYSCALL_STACK, %rsp
#else
        movq    %rsp, cpu0_state+CPU_STATE_SAVED_RSP(%rip)
        movq    cpu0_state+CPU_STATE_SYSCALL_STACK(%rip), %rsp
#endif          /* USE_SWAPGS */