OSDev.org

cyr1x wrote:

gerryg400 wrote: Is it possible to return from an NMI without enabling further NMIs ?

Yes, just don't use IRET, use RETF, etc...

turdus wrote:In long mode you can skip the CPL check, as CPU can switch stacks with consistent data regardless to CPL change.

turdus wrote:IST also allows separate stacks for NMI, exception handlers and irq handlers for example (up to 7 stacks).

As far as I can tell; after the CPU has started the NMI handler but before the CPU has had a chance to execute the NMI handler's very first instruction, a machine check exception or an SMI could occur and do IRET; and immediately after the IRET (or after a return from SMM) a second NMI can occur before you've managed to execute one instruction for the first NMI.

The only "bullet-proof" way to handle NMI is to cope with nested NMI. This implies that using a task gate (in protected mode) or using the "IST" (in long mode) for the NMI handler is a bad idea (as they can't handle nesting).

gerryg400 wrote:

As far as I can tell; after the CPU has started the NMI handler but before the CPU has had a chance to execute the NMI handler's very first instruction, a machine check exception or an SMI could occur and do IRET; and immediately after the IRET (or after a return from SMM) a second NMI can occur before you've managed to execute one instruction for the first NMI.

The only "bullet-proof" way to handle NMI is to cope with nested NMI. This implies that using a task gate (in protected mode) or using the "IST" (in long mode) for the NMI handler is a bad idea (as they can't handle nesting).

I'm pretty much out of my depth here. Don't know much about SMIs or MC exceptions. I thought (perhaps hoped ?) that SMI code shouldn't do an IRET for this very reason.

Intel wrote:26.8 NMI HANDLING WHILE IN SMM

NMI interrupts are blocked upon entry to the SMI handler. If an NMI request occurs during the SMI handler, it is latched and serviced after the processor exits SMM. Only one NMI request will be latched during the SMI handler. If an NMI request is pending when the processor executes the RSM instruction, the NMI is serviced before the next instruction of the interrupted code sequence. This assumes that NMIs were not blocked before the SMI occurred. If NMIs were blocked before the SMI occurred, they are blocked after execution of RSM.

Although NMI requests are blocked when the processor enters SMM, they may be enabled through software by executing an IRET instruction. If the SMM handler requires the use of NMI interrupts, it should invoke a dummy interrupt service routine for the purpose of executing an IRET instruction. Once an IRET instruction is executed, NMI interrupt requests are serviced in the same “real mode” manner in which they are handled outside of SMM.

A special case can occur if an SMI handler nests inside an NMI handler and then another NMI occurs. During NMI interrupt handling, NMI interrupts are disabled, so normally NMI interrupts are serviced and completed with an IRET instruction one at a time. When the processor enters SMM while executing an NMI handler, the processor saves the SMRAM state save map but does not save the attribute to keep NMI interrupts disabled. Potentially, an NMI could be latched (while in SMM or upon exit) and serviced upon exit of SMM even though the previous NMI handler has still not completed. One or more NMIs could thus be nested inside the first NMI handler. The NMI interrupt handler should take this possibility into consideration.

Also, for the Pentium processor, exceptions that invoke a trap or fault handler will enable NMI interrupts from inside of SMM. This behaviour is implementation specific for the Pentium processor and is not part of the IA-32 architecture.

gerryg400 wrote:And I thought that MC exceptions weren't recoverable anyway so there was no need to return from them. Too simplistic ?

gerryg400 wrote:Bottom line is I've been assuming that I could prevent NMIs from nesting.

Brendan wrote:

Code: Select all

interruptHandler:
    push dword 0                       ;Dummy error code (remove for some exception handlers)

    test byte [esp+4],3                ;Was interrupted code running at CPL=0?
    je .isCPL0                         ; yes

Code: Select all

    popad
    add esp,4                          ;Remove error code
    iretd

Intel Vol. 3 Chap. 6.14.2 wrote:In legacy mode, the stack pointer may be at any alignment when an interrupt or exception causes a stack frame to be pushed. This causes the stack frame and succeeding pushes done by an interrupt handler to be at arbitrary alignments. In IA-32e mode, the RSP is aligned to a 16-byte boundary before pushing the stack frame. The stack frame itself is aligned on a 16-byte boundary when the interrupt handler is called. The processor can arbitrarily realign the new RSP on interrupts because the previous (possibly unaligned) RSP is unconditionally saved on the newly aligned stack. The previous RSP will be automatically restored by a subsequent IRET.

Brendan wrote:Interrupt handlers have to check the interrupted code's CPL

turdus wrote:

Brendan wrote:For "one kernel stack per CPU".... Interrupt handlers have to check the interrupted code's CPL

No, that's the point of IST. Read the manuals ("The IST mechanism provides a method for specific interrupts, such as NMI, double-fault, and machine-check, to always execute on a known- good stack."). I don't have such a check and I can assure you it works fine and fast without. And there's plenty of space on TCB page for nested stacks (a thread struct usually not bigger than 512 bytes).
Let's say you cannot image a handler without it, but it's not a "have to".

XenOS wrote: @all:

Regarding the issue of NMI handling - how should an NMI be handled anyway? I always thought that NMIs indicate some critical hardware failure that cannot be dealt with in any sane way without shutting down the system (or letting it crash). (Except for IPIs with NMI delivery mode, of course - but these are always software generated.)

XenOS wrote:Regarding the issue of NMI handling - how should an NMI be handled anyway? I always thought that NMIs indicate some critical hardware failure that cannot be dealt with in any sane way without shutting down the system (or letting it crash). (Except for IPIs with NMI delivery mode, of course - but these are always software generated.)

Cognition wrote:I also get the impression that NMIs shouldn't really nest in the first place.

XenOS wrote:@gerryg400:
Would you mind posting your ret_to_user code? I guess I can imagine what it should look like, I just want to make sure that I don't miss anything important.

ret_to_user:

        /* put core_id in edi */
        GET_COREID %edi
        call    ksysexit
        /* ksysexit returns the new stack pointer */
        movq    %rax, %rsp

        /* Back to the new thread */
        POP_GP_REGS
        iretq

void *ksysexit(int core_id) {

    kobj_thread_t *newt;

    kassert(core_nest[core_id] == 1);

    ksignal_check();

    newt = curr_thread[core_id];

    /* Now run the continuation function */
    if (newt->continuation_func) {
        kdebug("POST PROCESSING\n");
        newt->reg.rax = newt->continuation_func(newt);
        newt->continuation_func = 0;
    }

    __cli();

    /* Set the current core state */
    core_state[core_id] = 0;

    /* Set the core nesting level to 0 */
    core_nest[core_id] = 0;

    /* Fix up the TLS */
    wr_msr(0xc0000100, (uintptr_t)curr_thread[core_id]->pptls);

    /* Fixup the TSS */
    tss[core_id]->rsp0 = (uint64_t)&curr_thread[core_id]->stk0top;

    /* And get the stack for the next thread */
    return &curr_thread[core_id]->reg.r15;
}

XenOS wrote:One more minor issue with the "stack points to TCB" approach I (possibly) found is that in long mode one needs to make sure that the register save area pointed to by RSP0 is always aligned at a 16-byte boundary, because otherwise the pushed registers may end up at the wrong place:

Brendan wrote:Hi,

turdus wrote:

Brendan wrote:For "one kernel stack per CPU".... Interrupt handlers have to check the interrupted code's CPL

No, that's the point of IST. Read the manuals ("The IST mechanism provides a method for specific interrupts, such as NMI, double-fault, and machine-check, to always execute on a known- good stack."). I don't have such a check and I can assure you it works fine and fast without. And there's plenty of space on TCB page for nested stacks (a thread struct usually not bigger than 512 bytes).
Let's say you cannot image a handler without it, but it's not a "have to".

You're using "one (small) kernel stack per thread plus a bunch more kernel stacks for various interrupt handlers" and not using "one kernel stack per CPU"; and because you're having trouble seeing the difference you're not comprehending what everyone else here is talking about.

rdos wrote:I don't think that matters a lot. In a design that is tolerant of stack overflows in kernel, either the stack or the double fault handler must be a task / IST. Otherwise, any such things will generate a tripple-fault, and you would have no idea what happened. This is not related to if there is one kernel stack per thread or CPU.

gerryg400 wrote:

rdos wrote:I don't think that matters a lot. In a design that is tolerant of stack overflows in kernel, either the stack or the double fault handler must be a task / IST. Otherwise, any such things will generate a tripple-fault, and you would have no idea what happened. This is not related to if there is one kernel stack per thread or CPU.

Perhaps a task gate or an IST could be used to find kernel bugs, but in this type of kernel design it is intended that 'normal' operation never use these CPU features. Remember that most work in a microkernel is done in ring 3. And in this particular type of microkernel the goal is for a very small, very flat (not very nested) kernel. My kernel is not tolerant in any way of kernel stack overflow. It doesn't need to be because (aside from bugs), the kernel stack is used in a very predictable way and will never overflow.

Brendan wrote:You're using "one (small) kernel stack per thread plus a bunch more kernel stacks for various interrupt handlers" and not using "one kernel stack per CPU"; and because you're having trouble seeing the difference you're not comprehending what everyone else here is talking about.

Note: Ignoring space used for things like thread name, amount of CPU time used by the thread, etc; we'd be looking at about 512 bytes for FPU/MMX/SSE state alone; plus another 32 bytes (for protected mode) or 128 bytes (for long mode) to store the thread's general registers. There are no stacks in the TCB at all.