Reading Local APIC registers returns all zeroes

[gmfun]

Hi,

I am learning the APIC programming in the qemu-kvm environment. I checked the local APIC address using both IA32_APIC_BASE (0x1b) MSR register and ACPI MADT table, all yielding the same address: 0xfee00000:
I "initialize" APIC using the below procedure:

Code: Select all

#define IA32_APIC_BASE_MSR 0x1B
#define IA32_APIC_BASE_MSR_ENABLE 0x800
#define IA32_APIC_BASE_MSR_BSP 0x100

void local_apic::init(void *base_address)
{
    if (0 == base_address) {
        const uint64_t apic_msr = arch_read_msr(IA32_APIC_BASE_MSR);
        lapic_ptr_ = reinterpret_cast<volatile uint32_t *>(apic_msr & 0xfffff000);
    } else {
        lapic_ptr_ = reinterpret_cast<volatile uint32_t *>(base_address);
    }
    const uint64_t apic_msr_enable = ((uint64_t)lapic_ptr_ & 0xfffff000) | IA32_APIC_BASE_MSR_ENABLE | IA32_APIC_BASE_MSR_BSP;
    arch_write_msr(IA32_APIC_BASE_MSR, apic_msr_enable);
    uint64_t verify = arch_read_msr(IA32_APIC_BASE_MSR);
    if (IA32_APIC_BASE_MSR_ENABLE == (verify & IA32_APIC_BASE_MSR_ENABLE)) {
        immediate_console::print("xapic ok %016x\n", verify);
    } else {
        immediate_console::print("xapic fail\n");
    }
    // Write to spurios interrupt vector register
    *((volatile uint8_t *)lapic_ptr_ + 0xf0) = *((volatile uint8_t *)lapic_ptr_ + 0xf0) | 0x100;
}

However, whenever I try to read any of the LAPIC registers, they all have value 0 regardless of the value I wrote there earlier.

I checked the Intel manual regarding the APIC and found that when paging is enabled, the APIC registers should be mapped to a UC (strong uncached) page.
My small kernel switches to the long mode and sets up page table to identity-map first 4G of the address space. For testing, I set bits 3 (write-through enable), 4 (caching disable), and 12 (PAT bit for huge pages) when mapping 2MB huge pages. Below is the code:

Code: Select all

static constexpr auto page_table_alignment = 4096;

alignas(page_table_alignment) static uint64_t p4_table[512];
alignas(page_table_alignment) static uint64_t p3_table[512];
alignas(page_table_alignment) static uint64_t p2_tables[4][512];

constexpr auto PAGE_PRESENT  = 1LU << 0;
constexpr auto PAGE_RW       = 1LU << 1;
constexpr auto PAGE_WT       = 1LU << 3;
constexpr auto PAGE_CD       = 1LU << 4;
constexpr auto PAGE_HUGE     = 1LU << 7;
constexpr auto PAGE_PAT_HUGE = 1LU << 12;

static uint64_t make_p4_entry(const uint64_t p3_addr)
{
    return PAGE_PRESENT | PAGE_RW | p3_addr;
}

static uint64_t make_p3_entry(const uint64_t p2_addr)
{
    return PAGE_PRESENT | PAGE_RW | p2_addr;
}

static uint64_t make_p2_entry_huge(const uint64_t phys_addr)
{
    return PAGE_PRESENT | PAGE_RW | PAGE_PAT_HUGE | PAGE_CD | PAGE_WT | PAGE_HUGE | phys_addr;
}

void init_identity_mapping()
{
    for (int i = 0; i < 4; i++) {
        for (int j = 0; j < 512; j++) {
            p2_tables[i][j] = make_p2_entry_huge(j * (2 << 20));
        }
        p3_table[i] = make_p3_entry(reinterpret_cast<uint64_t>(p2_tables[i]));
    }
    p4_table[0] = make_p4_entry(reinterpret_cast<uint64_t>(p3_table));
    const uint64_t p4_addr = reinterpret_cast<uint64_t>(p4_table);
    asm volatile("mov %0, %%rax\n\tmov %%rax, %%cr3" : : "m" (p4_addr) : "memory", "eax");
}

I also checked the IA32_PAT MSR (0x277) which has the value 0x0000000000070406, which means the PAT 3 that is being selected in my page table (0b111), is UC (all 3 bits are zero in PAT3 entry in IA32_PAT MSR).

The environment I run in is qemu-kvm (2.8.0), with GRUB2 as bootloader.

Does anybody have an idea of what is wrong with my approach?

Best regards,
Rost

[nexos]

You shouldn't need to set bit 12. Have you declared the APIC base pointer as volatile?

[Octocontrabass]

I also checked the IA32_PAT MSR (0x277) which has the value 0x0000000000070406,

Did you intentionally change it from its default value of 0x0007040600070406? If not, you've found a bug - either in how you print the value or in how you read the MSR.

which means the PAT 3 that is being selected in my page table (0b111),

You're selecting PAT7, not PAT3.

[gmfun]

You shouldn't need to set bit 12. Have you declared the APIC base pointer as volatile?

Agree, this is excessive, but should not be harmful unless it was already set to 0 earlier.

Did you intentionally change it from its default value of 0x0007040600070406? If not, you've found a bug - either in how you print the value or in how you read the MSR.

Indeed, I used the wrong format specifier. PAT is equal to 0x0007040600070406.

You're selecting PAT7, not PAT3.

Yeah, thanks for pointing this out. Still, the memory should be marked as UC.

I also noticed I had wrong cast when accessing LAPIC registers (uint8_t instead of uint32_t):

Code: Select all

*((volatile uint8_t *)lapic_ptr_ + 0xf0) = *((volatile uint8_t *)lapic_ptr_ + 0xf0) | 0x100;

Yet, there was no bug in the original code, only in the simplified version I rewrote for this post is wrong.

On a side note, I was able to access x2APIC registers via MSR interface, so I will probably continue using this method. Maybe one day I or someone else will figure out the reason why using MMIO does not work here

[nexos]

On a side note, I was able to access x2APIC registers

x2APIC is fairly new. For this reason, shouldn't rely on its existence. Also be sure to check for it with CPUID first.

[Octocontrabass]

I also noticed I had wrong cast when accessing LAPIC registers (uint8_t instead of uint32_t):

Casts have higher precedence than arithmetic operators, so you're performing arithmetic on pointers to uint32_t. Throw some parentheses in there (or divide the offsets by the size of uint32_t) and see if that helps.