Can't get APs to boot up, trying to add SMP to my OS.

[wrschlanger]

Please see this file: http://gaia.ecs.csus.edu/~schlangw/smp.cpp
For complete source code, download the file at the top of this site: http://code.google.com/p/vm64dec/downloads/list.

I have parsed the floating MP structure, done everything I thought I should, but still the APs won't start up! The APs, if they start up, will fill the screen at 0xb800:0 with 0x1f20's thus giving it a blue background. It never happens.

I am using the x86-64 version of Qemu with -smp 4 on a 32-bit Ubuntu Linux.

Here is relevant code:

Code: Select all

	// Send Startup IPIs if not an old pre-integrated APIC.
	if(proc->apic_ver >= APIC_VER_NEW)
	{
///video_printf("[new]");
imps_lapic_write(LAPIC_ESR, 0);
accept_status = imps_lapic_read(LAPIC_ESR);
		int i;
		for (i = 1; i <= 2; i++)
		{
			send_ipi(apicid, LAPIC_ICR_DM_SIPI | ((bootaddr >> 12) & 0xFF), 0xfff0f800);
			init_delay(10);
		}
	}
	
for(;;) ;

Now the infinite loop is there because I'm waiting for a sign that at least one AP is starting up. It never does.

Any help would be greatly appreciated.

Thanks,
Willow

[JohnnyTheDon]

The startup for APs is not a single IPI. You need to go through the INIT-SIPI-SIPI sequence outlined in the Intel manuals:

15. Broadcasts an INIT-SIPI-SIPI IPI sequence to the APs to wake them up and initialize them:
MOV ESI, ICR_LOW; Load address of ICR low dword into ESI.
MOV EAX, 000C4500H; Load ICR encoding for broadcast INIT IPI
; to all APs into EAX.
MOV [ESI], EAX; Broadcast INIT IPI to all APs
; 10-millisecond delay loop.
MOV EAX, 000C46XXH; Load ICR encoding for broadcast SIPI IP
; to all APs into EAX, where xx is the vector computed in step 10.
MOV [ESI], EAX; Broadcast SIPI IPI to all APs
; 200-microsecond delay loop
MOV [ESI], EAX; Broadcast second SIPI IPI to all APs
; 200-microsecond delay loop

[wrschlanger]

I don't understand, there is a for() loop here which sends the IPI twice.
Here is more complete code:

Code: Select all

int Smp::send_ipi(U4 dst, U4 v, U4 mask)
{
	int to, send_status;
	
U4 old1 = 0;// imps_lapic_read(LAPIC_ICR+0x10) & 0xffffff;
mask = 0;
	imps_lapic_write(LAPIC_ICR+0x10, old1|(dst << 24));
	imps_lapic_write(LAPIC_ICR, (imps_lapic_read(LAPIC_ICR) & mask) | v);

	/* Wait for send to finish */
	to = 0;
	do {
		init_delay(1);
		send_status = imps_lapic_read(LAPIC_ICR) & LAPIC_ICR_STATUS_PEND;
	} while (send_status && (to++ < 1000));
	
///if(to == 1000) video_printf("[ERROR]"); else video_printf("[OK]");

	return (to < 1000);
}

// This should return 0 on success.
int Smp::boot_cpu(imps_processor *proc)
{
	int apicid = proc->apic_id;//, success = 1, to;
	unsigned accept_status;
	U8 bootaddr = 0x70000;
	U8 bios_reset_vector = kernel_load_address + BIOS_RESET_VECTOR;

	U1 *bootdest = (U1 *)(kernel_load_address + bootaddr);
	for(int i = 0; i < 4096; ++i)
	{
		bootdest[i] = 0;
	}
	bootdest[0] = 0xea;
	*(U4 *)(bootdest + 1) = 0xffff0510;
for(U8 i = 0x500; i < 0x1000; ++i)
  bootdest[i-0x500] = *(U1 *)(i + kernel_load_address + 0x100000);

	// set BIOS reset vector
	cmos_write_byte(CMOS_RESET_CODE, CMOS_RESET_JUMP);
	*((volatile U4 *) bios_reset_vector) = ((bootaddr & 0xFF000L) << 12);
	
	// clear the APIC error register
	imps_lapic_write(LAPIC_ESR, 0);
	accept_status = imps_lapic_read(LAPIC_ESR);

	// assert INIT IPI
	send_ipi(apicid, LAPIC_ICR_TM_LEVEL | LAPIC_ICR_LEVELASSERT | LAPIC_ICR_DM_INIT, 0xfff0f800);

	init_delay(100);

	// de-assert INIT IPI
	send_ipi(apicid, LAPIC_ICR_TM_LEVEL | LAPIC_ICR_DM_INIT, ~0xcdfff);

	init_delay(100);

	// Send Startup IPIs if not an old pre-integrated APIC.
	if(proc->apic_ver >= APIC_VER_NEW)
	{
///video_printf("[new]");
imps_lapic_write(LAPIC_ESR, 0);
accept_status = imps_lapic_read(LAPIC_ESR);
		int i;
		for (i = 1; i <= 2; i++)
		{
			send_ipi(apicid, LAPIC_ICR_DM_SIPI | ((bootaddr >> 12) & 0xFF), 0xfff0f800);
			init_delay(10);
		}
	}
	
for(;;) ;
	init_delay(10000);

	video_printf("(Unimplemented)");
	video_printf("\n");
	return 0;
}

As you can see, the IPIs are in a for loop that sends 2 of them:

Code: Select all

		for (i = 1; i <= 2; i++)
		{
			send_ipi(apicid, LAPIC_ICR_DM_SIPI | ((bootaddr >> 12) & 0xFF), 0xfff0f800);
			init_delay(10);
		}

I also remembered to enable the APIC via spurious vector, but it doesn't help. I tried the Intel code verbatim (OK I translated it to 64bit C++ code) but it Still fails to start up the APs.

Using qemu -smp 4.

[Matthew]

Are you certain that you are correctly linking and setting up the AP boot code? I am not seeing anything wrong with the startup sequence, and Qemu is fairly forgiving, but are you sure that the APs are finding the code?

Also try Bochs and examine the output log, it is very detailed.