OSDev.org

Hello, this is my first post here. Currently I am reading the source code of xv6 , a small teaching OS, and got lost in the part of APIC and PIC initialization. I have tested xv6 on QEMU 0.11.1 and BOCHS 2.4.5, and found some problems. I am sorry but the question is so long because I tested many possible combinations.

First, I think I have to briefly explain what has been done about APIC and PIC in xv6. If you are not interested in xv6, feel free to skip to the questions:

1. In the main() of xv6, it parse the MP table at first, save the ids of LAPIC and IOAPIC, and save the address of IOAPIC without checking the IOAPIC enable flag. Other information is ignored.

2. If the MP table exists, variable ismp will be set. If it is set, then xv6 initializes LAPIC: 1) setup the APIC Timer and the error register, 2) mask LINT0,1 and Performance Counter, 3) clear the ESR, 4) send one EOI, and 5) send the INIT de-assert.

3. Regardless of MP table and APIC, xv6 then initializes the PIC with usual setting that mask all irqs.

4. If ismp is set, initialize IOAPIC. Compare the ioapic id saved from MP table and the id read from IOAPICID register, and output an error message if they are not equal. Then, disable all irqs.

(Actually, two ids are never equal on QEMU, and the result won't effect anything!)

5. If ismp is not set, setup the PIT.

6. There are two devices using irqs: keyboard and IDE disk. IDE disk is accessed in LBA mode, and only slave disk will be used. Both of them enables the irq 1, 14 on PIC, and enables on IOAPIC if ismp is set. The irq offset is 32.

Now I got a question:

Q1: Why enable irqs on BOTH PIC and IOAPIC?

Trying to answer it, I have checked the Intel Multiprocessor Specification, the Intel IA-32 3A Manual chapter 5, 7, 8, the Intel 8259A datasheet, and the IOAPIC datasheet. But I still can't make it clear about the relation between APIC and PIC in a real machine. So I tried to modify something and run it on QEMU and BOCHS. This is the result:

QEMU 0.9

First I just run it on the qemu from apt of debian. Suddenly after sti, xv6 gets 'unexpected trap 39'. This happens only on some machines including my Thinkpad T60. I have no idea about it and just compiled and use the newer QEMU 0.11.1.

QEMU 0.11.1

Everything works fine. QEMU uses APIC and filled the MP table, though the IOAPIC ids are different as mentioned above.

QEMU 0.11.1 with -cpu 486

I want to see how does it run on a machine without APICs, so I tried the 486 cpu, and found that QEMU implements 486 utilizing APICs. The MP table is filled as the default cpu does. Everything seems unchanged.

Then, I started to try to make it enable irqs on either PIC or IOAPIC, and initialize either PIC or APICs only. Here is the result:

1. Initialize both, enable on PIC only: default hangs after boot, 486 runs normally.

2. Initialize both, enable on IOAPIC only: default runs normally, 486 hangs after boot just like the default hangs above.

3. Initialize PIC only: both run normally. In smp, only one CPU is initialized.

4. Initialize LAPIC and PIC: default hangs after boot, 486 passes the bootstrap, but is unable to pass the usertests in xv6.

5. Initialize LAPIC and IOAPIC: default runs normally, 486 gets 'unexpected trap 118'.

Besides the reasonable result #3, how to exlain:

Q2: If both of them can initialize both PIC and APICs, why default relies on IOAPIC and 486 relies on PIC? Is it defined in any specifications? Or is it just the implementation of QEMU?

Q3: Why QEMU (also does BOCHS) gets unexpected traps, which should have been masked both on PIC and IOAPIC?

I think all of theses questions may be just what QEMU implements, so I decided to try BOCHS and see what is different. I am sorry that I have no old machines handy to try out an old cpu without APICs.

BOCHS 2.4.2

It is installed by apt. Suddenly gets 'unexpected trap 47'. The gdb stub is not enabled, so I compiled 2.4.5.

BOCHS 2.4.5 --enable-gdb-stub

I tried this, and everything works fine, but an 'unexpected trap 39' appears again after several commands executed. Later, I found that the MP table is not built. BOCHS will write it only if --enable-pci is also added, thus xv6 was just using the PIC, which is always promised to work.

BOCHS 2.4.5 --enable-gdb-stub --enable-pci

Now MP table is built, and everything should work like QEMU default does...here the unexpected trap 47 comes again! so I tried those combinations with BOCHS again, including --enable-cpu-level=4 and --enable-smp.

Here is the result:



There are several patterns:

1. 'trap #' means an 'unexpected trap' happens during bootstrap. This is the most annoying one and I have no ideas about it.

2. 'boot passed, trap # latter' means the bootstrap is success, but get 'unexpected trap' after several commands are executed. Usually follows a wrong command or command error. This is also a strange pattern.

3. 'hangs after boot' means the bootstrap is success, but the irqs are not correctly enabled, so the user process hangs.

4. 'hangs waiting for LAPIC INIT De-Assert' means LAPIC INIT De-Assert should not be sent and won't be sent in this configuration of machine, so it hangs.

5. 'ok' means the bootstrap is success, it won't get 'unexpected trap' after several commands executed, and the usertests is passed.

In BOCHS, xv6 gets more chaotic:

Q4: Why xv6 in BOCHS can not work properly with initialization and enabling of PIC only? What may cause the trap 39 (irq 7) happens even that it is masked? Is it related to PCI? (currently I do not know anything about PCI though)

Q5: BOCHS seems have completely different result with QEMU. Anyway, what exactly should an OS do when it is initializing PIC or APIC, assuming that it will run on old machines (486), old machines with APICs, and new machines with APICs?

Just a request: Don't use colours to indicate questions. Use Bold or similar instead; some of us use dark coloured themes and the blue is really impossible to read.