Oxide and Friends

[thewrongchristian]

Interesting Oxide and Friends podcast:

https://oxide-and-friends.transistor.fm ... corruption

It's a slow burn to get to the ultimate root cause, but worth a listen.

In summary, the episode is about tracking down an issue causing data corruption in their firmware bootstrap system, but seemingly at random. A cautionary tail for budding OS developers, but I won't add spoilers right now.

[thewrongchristian]

Interesting Oxide and Friends podcast:

https://oxide-and-friends.transistor.fm ... corruption

It's a slow burn to get to the ultimate root cause, but worth a listen.

In summary, the episode is about tracking down an issue causing data corruption in their firmware bootstrap system, but seemingly at random. A cautionary tail for budding OS developers, but I won't add spoilers right now.

Spoiler:

The code in question was invalidating a kernel level TLB mapping for a page, but doing so before the corresponding page table entry was cleared. They did it in this order on the assumption that they wouldn't be referencing the page again in their code.

But, before they got to clearing the PTE, they were getting an interrupt, the ISR contained code that speculatively was referencing the page being invalidated, and that speculative reference was reloading the TLB from the still valid PTE.

The TLB entry remained, because it wasn't being further invalidated, even though the page table had no valid entry for the page. So any references to that page were referring to some now phantom PTE, until some other mechanism invalidated it.

TL; DR

Remove your PTE from the page table, then invalidate the TLB.

[eekee]

Interesting, thanks. I saw speculative execution was involved. Every time I hear of a spec exec bug, I get more interested in embedded and ancient hardware! But I'm making a note about it anyway.