APIC & AMD64 [SOLVED]

[ezeaguerre]

Hi !!!
I was making a little OS to test some recently acquired skills but I got stucked. The problem is this:

First I change the mapping of the APIC through the IA32_APIC_BASE MSR to something like 0x28000, then I read some APIC parameters ( Local APIC ID, Max LVT, APIC Version ) from 0xC0082020 & 0xC0082020 ( I use paging and I have a whole GB from 0xC0000000 to the end ). In Bochs and VMWare works great, but when I really boot into this little application ( I mean like a really OS ) it gets totally frozen. First I thought it could be something wrong with the video, so, I tried to make it triple fault by loading the GDT with a wrong address, using floating point math and making a page fault but it didn't work... I don't know what's going on... I'm going mad !!!
Here is a little of code:

Code: Select all

// Enable APIC
	reg_t *pLVT = ( reg_t * ) ( ( reg_t ) pApicBase + 0xf0 );
	*pLVT = 0x1ff;
	
	// Error LVT
	pLVT = ( reg_t * ) ( ( reg_t ) pApicBase + 0x370 );
	*pLVT = 0x100ff;
	
	// Timer LVT
	pLVT = ( reg_t * ) ( ( reg_t ) pApicBase + 0x320 );
	*pLVT = 0x100ff;
	
	// LINT0 LVT
	pLVT = ( reg_t * ) ( ( reg_t ) pApicBase + 0x350 );
	*pLVT = 0x100ff;
	
	// LINT1 LVT
	pLVT = ( reg_t * ) ( ( reg_t ) pApicBase + 0x360 );
	*pLVT = 0x100ff;
	
	// Perf Mon Counters
	pLVT = ( reg_t * ) ( ( reg_t ) pApicBase + 0x340 );
	*pLVT = 0x100ff;
	
	// Thermal Sensor
	pLVT = ( reg_t * ) ( ( reg_t ) pApicBase + 0x330 );
	*pLVT = 0x100ff;
	
	reg_t *pPointer;
		
	// Local Apic
	pPointer= ( reg_t * ) ( ( reg_t ) pApicBase + 0x20 );
	iLocalApic = *( ( int * ) pPointer ) >> 24;
		
	pPointer = ( reg_t * ) ( ( reg_t ) pApicBase + 0x30 );
	iVersionApic = *( ( volatile int * ) pPointer );
	iMaxLVT = ( iVersionApic >> 16 ) & 0xff;
	iVersionApic &= 0xff;

The code from "Enable APIC" to "Thermal Sensor" wasn't there originally, but it didn't make any difference anyway. I've tracked down the problem to the reading of APIC Version & Local APIC ID.

Also, I've found this on the Linux kernel:

Code: Select all

static int modern_apic(void)
{
	unsigned int lvr, version;
	/* AMD systems use old APIC versions, so check the CPU */
	if (boot_cpu_data.x86_vendor == X86_VENDOR_AMD &&
		boot_cpu_data.x86 >= 0xf)
		return 1;
	lvr = apic_read(APIC_LVR);
	version = GET_APIC_VERSION(lvr);
	return version >= 0x14;
}

That means I have no APIC Version? But what about the Local APIC ID? OK... I can use CPUID in place... but... I've read AMD's "BIOS & Kernel developers guide" and I think it should work. At least the APIC Register is there, maybe is an old version ( 0x10 vs 0x14 I think... ) but it should work...

Thanks so much for reading !!!
And sorry for my English ( specially the verbs... )

[Combuster]

The manuals state that all accesses that address the APIC must be uncacheable. Normally, the APIC's address is made uncacheable by the bios/hardware, but once you relocate it, you have to make sure that condition still holds.

Wether it is causing the problems, I don't know, but it never hurts to check.

[Brendan]

Hi,

Combuster is right...

AFAIK if the CPU thinks the local APIC's area can be cached, then it'll try to read an entire cache line (128 bytes) from the physical address. The local APIC only responds to dword transactions and will ignore a 128 by cache line fill, so the CPU will end up getting data from RAM.

Bochs, Qemu and probably other emulators don't emulate L1 and L2 caches (emulating caches doesn't make much sense when the emulated cache is in normal RAM and you have the overhead of doing cache lookup in software). Instead they operate as a cacheless system. In this case there is no cache line fill and the emulator tries to get a dword instead.

For fun, you could set the "cache disable" flag in CR0 and do a WBINVD instruction before trying to read from the local APIC. This should work (especially if it's a single CPU system and you don't have to worry about cache coherency with other CPUs), but it's not a good solution (blowing away CPU caches every time you want to send an EOI would severely effect performance). You could also try to mess with the MTRRs, but it's better to save the limited number of variable range registers for mapping other hardware like video card's display memory.

First I change the mapping of the APIC through the IA32_APIC_BASE MSR to something like 0x28000, then I read some APIC parameters ( Local APIC ID, Max LVT, APIC Version ) from 0xC0082020 & 0xC0082020 ( I use paging and I have a whole GB from 0xC0000000 to the end ).

Did you have any reason to change the IA32_APIC_BASE MSR? With paging it'd be easy to leave it at the same physical address and map it to any virtual address you like, without causing problems with the CPU's caching.

There's only 2 reasons I can think of to remap the local APICs and I/O APIC/s. The first is a real mode OS where you can't access them at the default address and need to shift them below 1 MB (this isn't a good reason - it doesn't apply to your OS as you're using paging, and for a real mode OS it'd be easier to use unreal mode).

The other reason is to make more space for memory mapped PCI devices, but you'd need a very strange computer for this to become necessary (a large number of memory mapped 32-bit PCI devices, a lot of RAM and a broken BIOS that doesn't leave enough space free). Even then you'd only gain about 16 MB of physical address space (depending on how large the BIOS is), and you'd have to make sure you relocate everything else in this area (including I/O APIC/s, HPET and any local APICs in disabled CPUs).


Cheers,

Brendan

[ezeaguerre]

Hi guys, thank you so much for replying !!
After remapping the APIC I set the "Cache Disable" bit on the page table entry and invalidate it, but I'll try the "Cache Disable" on CR0 and WBINVD just to see what happens.
About the MTRRs I think they start all on 0 which means UC for all memory, and here comes another question: If I have UC on the MTRRs for all memory, does the "Cache Disable" bit on CR0 and/or the page table has any meaning?
And no, there isn't any special reason to remap the Local APIC, I just wanted to try it
Well, I'll try everything you told me when my brother leaves my computer ¬¬, thank you so much !!!

P.S: Do I have to invalidate the caches after remapping the APIC or marking the page as "Cache Disable" and invalidate it is enough?

[Brendan]

Hi,

After remapping the APIC I set the "Cache Disable" bit on the page table entry and invalidate it, but I'll try the "Cache Disable" on CR0 and WBINVD just to see what happens.

The cache disable flag in a page directory or page table has slightly strange behaviour. From a technical perspective, because other CPUs in the system may have the same physical page mapped as "cachable" the CPU will still do "cache coherency" stuff to ensure correctness. This means sending MESI messages over the bus when the area is modified or read to tell other CPUs to flush the cache line.

For a single CPU system (and multi-CPU systems where the same physical page is never cached elsewhere) these MESI messages are pointless, but the CPU doesn't know that. The end result is that it's mostly harmless but also increases bus traffic for no reason.

This doesn't apply if the MTRRs say the area is non-cacheable as the MTRRs must be the same on all CPUs, and therefore it's impossible for one CPU to cache an area while another doesn't, and the CPU doesn't need to send the MESI messages over the bus. Of course in this case it won't matter what the cache disable flag in the page directory or page table says - it's uncachable regardless.

About the MTRRs I think they start all on 0 which means UC for all memory,

When the CPU first boots all MTRRs are disabled and everything is treated as uncachable. Then the BIOS does it's tricks and enables the MTRRs, so that by the time the OS is started all normal RAM is set to the "write-back" caching type (and other areas may be set to other caching types).

and here comes another question: If I have UC on the MTRRs for all memory, does the "Cache Disable" bit on CR0 and/or the page table has any meaning?

In this case the setting of cache disable flag in CR0 wouldn't make any difference. However if the MTRRs are set to cache anything, then the flag in CR0 will disable that caching.

P.S: Do I have to invalidate the caches after remapping the APIC or marking the page as "Cache Disable" and invalidate it is enough?

You must use WBINVD to flush all caches (or use the new CLFUSH instruction to flush effected cache lines) if, for any reason, the CPU may have the effected cache line/s in it's cache and you disable caching using any method. Please note that "for any reason" includes the BIOS using that cache line/s for something before your OS runs, and also includes "speculative execution" (where the CPU decides it might need the data and pre-loads the data into it's cache just in case).

The correct method (given that you can't guarantee that the old data isn't in the cache) would be to set the cache disable flag in CR0, then flush all caches, then change the address of the APIC and map the APIC into the address space with the cache disable flag in the page table set, and finally clear the cache disable flag in CR0. This ensures that the old data isn't in the cache beforehand, while also making sure the APIC's data isn't pre-loaded into the CPUs cache after the address of the APIC is changed but before it's mapped into the address space.

And no, there isn't any special reason to remap the Local APIC, I just wanted to try it

Patient: It hurts when I poke myself in the eye with a screwdriver!
Doctor: Why are you poking yourself in the eye with a screwdriver?
Patient: There's no special reason, I just wanted to try it.
Doctor: Perhaps we can modify the screwdriver so it doesn't hurt so much...



Of course it's also entirely possible that there's other problems. I'm mostly familiar with Intel CPUs, but modern AMD CPUs have differences. Specifically they use Hyper-transport, which includes a "top of memory" field in it's configuration. It's entirely possible that AMD CPUs send all transactions that are below the "top of memory" to it's memory controller without checking if the transaction hits the local APIC or not.


Cheers,

Brendan

[ezeaguerre]

Well, I can't make it work. I've tried this in my initialization code:

Code: Select all

...
	# DISABLE MTRRS, Wich means Strong Uncachable
	IA32_MTRR_DEF_TYPE = 0x2ff
	xorl %edx, %edx
	xorl %eax, %eax
	movl $IA32_MTRR_DEF_TYPE, %ecx
	wrmsr
	

	# Cache Disable = 1
	movl $0xC000003B, %eax
	movl %eax, %cr0
	
	# INVALIDATE CACHE
	wbinvd 
...

and I've also made the Cache Disable bit on the page table = 1.
However, if I just make a mapping to the normal APIC address it works great, it's not what I wanted in the first place, but it works... I'm gonna try that thing of "top of memory" to see what happens.

Thanks so much !!!

[Brendan]

Hi,

However, if I just make a mapping to the normal APIC address it works great, it's not what I wanted in the first place, but it works...

Did you try remapping it to something like 0xFED00000? This is such a small difference that it shouldn't cause problems with caching, etc and it'd rule out annoying bugs (like accidentally clearing the global enable/disable flag, or using the wrong MSR).

I'm gonna try that thing of "top of memory" to see what happens.

I'd suggest that remapping the local APIC to something like 0x28000, and then changing the top of memory to 0x28000 to make that work isn't very sane - you'd limit the computer to 160 KB of usable RAM (although this may depend on how the northbridge and hyper-transport routing is programmed).

I also wouldn't assume this is easy to do correctly. For AMD there's 2 top of memory registers, the MTRRs and a routing table that all determine which hyper-transport link is used for different physical memory accesses, and all of these things (for each CPU) should be consistant with each other.


Cheers,

Brendan

[ezeaguerre]

Hi,

Did you try remapping it to something like 0xFED00000? This is such a small difference that it shouldn't cause problems with caching, etc and it'd rule out annoying bugs (like accidentally clearing the global enable/disable flag, or using the wrong MSR).

No, but I'll try

I'd suggest that remapping the local APIC to something like 0x28000, and then changing the top of memory to 0x28000 to make that work isn't very sane - you'd limit the computer to 160 KB of usable RAM (although this may depend on how the northbridge and hyper-transport routing is programmed).

Off course I'm not gonna do that. I mean I'll research the subject

I also wouldn't assume this is easy to do correctly. For AMD there's 2 top of memory registers, the MTRRs and a routing table that all determine which hyper-transport link is used for different physical memory accesses, and all of these things (for each CPU) should be consistant with each other.

Yes, I've been reading, the second top of memory is for memory beyond the 4GB mark.
I've also found this:

The access destination is based on the highest priority of the following ranges that the access falls in:
1. (Lowest priority) Compare against the top-of memory registers (MSR C001_001A and MSR
C001_001D).
2. The Fixed-Size MTRRs (MSRs 02[6F:68, 59, 58, 50]).
3. The IORRs (MSRs C001_00[18, 16] and MSRs C001_00[19, 17]).
4. TSEG & ASEG (MSR C001_0112 and MSR C001_0113).
5. (Highest priority) NB AGP aperture range registers.

Maybe I'll have to set the appropriate type to one of the fixed MTRRs and use AMD extensions to select I/O destination ( Quoting the manual: Each fixed MTRR includes two bits, RdDram and WrDram, that determine the destination based on the access type. ). I think the IORRs should be fine, so I'll try the MTRRs...

Bye !!

PS: I tested my OS with an Athlon XP and it worked fine, so I'm quite sure is a problem with the integrated north bridge of the K8

[ezeaguerre]

It WORKS !!!! Finally !!!
It seems neither disabling nor enabling the MTRRs works.
To make it work I had to enable the AMD extensions I previously mentioned, wich seems weird because they say that if the extensions are disabled the access go to I/O by default, anyway, I made a simple check like this:

Code: Select all

if ( ! strcmp ( strVendor, "AuthenticAMD" ) )
   enableMTRR (); 

and this is enableMTRR:

Code: Select all

IA32_MTRR_FIX64K_00000 = 0x250
IA32_MTRR_DEF_TYPE = 0x2ff
SYSCFG = 0xC0010010
enableMTRR:
   pushl %eax
   pushl %ecx
   pushl %edx
	
   # First turn off the cache
   movl %cr0, %eax
   orl $0x40000000, %eax
   movl %eax, %cr0
   wbinvd # I think this is unnecessary
	
   # Enable AMD extensions to the MTRRs
   movl $SYSCFG, %ecx
   rdmsr
   orl $0xC0000, %eax
   wrmsr

   # Make sure MTRRs are ON
   movl $IA32_MTRR_DEF_TYPE, %ecx
   rdmsr
   orl $0xC00, %eax
   andl $0xFFFFFF00, %eax
   wrmsr
	
   # Let's make the memory mapped APIC: I/O UC
   movl $IA32_MTRR_FIX64K_00000, %ecx
   rdmsr
   andl $0xFF00FFFF, %eax
   wrmsr
	
   # Get cache back on !!!
   movl %cr0, %eax
   andl $0x9FFFFFFF, %eax
   movl %cr0, %eax
   wbinvd # It seems ridiculous, but I'm too scared :P
	
   popl %edx
   popl %ecx
   popl %eax
   ret

Off course there's a lot to polish in my code ( for example: Not to call enableMTRR when I'm on K7 ), but for the moment I'm happy

Thank you very much guys !!!
Specially you Brendan for all your wise advises