Emulator with non-ideal conditions

[Prochamber]

I've notice how emulators always (attempt to) create an environment that is ideal for compatibility and has hardware that works perfectly and is free of buggy or non-standard implementations.
This environment works very well when the requirement is to use an existing operating system for what ever purpose, however a perfect environment is not useful for testing your own.
It would be great if there was an emulator that could simulate buggy hardware or hardware errors. I am not taking about essential pieces of hardware randomly failing. The idea is something along the lines of simulating troublesome hardware, like randomly occurring read errors on a floppy disk controller or a VGA controller with some VESA modes as non-VGA modes with weird padding or a machine uses an old CPU like an 8086.
What do you think, would it be useful? Has anyone done this?

[bluemoon]

This will be very useful for os development, I guess bochs got some cpu-specific (and perhaps device specific) triggers to generate problematic situations for debug purpose (debug of the emulator itself).

However, for general usage, user only care if it runs popular OS like Windows or Linux; therefore the motivation of such features is limited.

[AbstractYouShudNow]

You can always download the source code of, e.g. Bochs, and introduce code that generates buggy situations. However, make sure that these situations are POSSIBLE, or you'll work for nothing.

[Casm]

I would be more worried about software bugs than hardware bugs. The one probably outnumbers the other by around 100,000 to one.

[Prochamber]

You can always download the source code of, e.g. Bochs, and introduce code that generates buggy situations. However, make sure that these situations are POSSIBLE, or you'll work for nothing.

It would be possible to change the BIOS return values. I'll look into the Bochs code, maybe I can use that to emulator hardware errors. I'm really not fan of Bochs but it looks like it's my best option.

[bluemoon]

If you want to simulate BIOS return faulty value, you could install ISR hooks as a test bed.

However, I discourage doing any serious work with BIOS or real mode.

[Brendan]

Hi,

What do you think, would it be useful? Has anyone done this?

It would be very useful; especially for things like "full coverage" unit testing for boot code (e.g. does it handle a seriously messed up memory map correctly) and device drivers (e.g. does the device driver correctly handle all of the "faulty hardware" cases correctly); plus things like fault tolerance testing.

For common hardware faults (e.g. RAM and disk errors), it has been done before - mostly researchers adding "fault injection" features to various emulators (Bochs, Qemu) and using it to do research on various things (hint: the right keywords to search for seem to be "fault injection emulator").

Sadly; I don't think any/many people have done much more than this.


Cheers,

Brendan

[Antti]

What about hardware? Could we have a special computer that has a physical switch to generate an NMI, another switch for "faulty RAM" activation etc.? It is possible but has anyone done anything like this?

[jnc100]

What about hardware? Could we have a special computer that has a physical switch to generate an NMI, another switch for "faulty RAM" activation etc.? It is possible but has anyone done anything like this?

I have a Dell server with an external NMI switch. It is useful for activating debugging modes on certain operating systems.

As regards 'broken' memory maps, just how extreme can they get? I have seen some with several overlapping areas, where I have always assumed the 'used' regions take priority over the 'free' ones, and have generally regarded anything under 1 MiB as used even if not marked as such, but do some actually report areas as free which don't exist?

Regards,
John.

[bluemoon]

What about hardware? Could we have a special computer that has a physical switch to generate an NMI, another switch for "faulty RAM" activation etc.? It is possible but has anyone done anything like this?

You may manually remove the RAM or make it bad contact.
Note that, there is risk of damaging the computer or get you electrocuted.

Mixing different brand of memory chip or insane over-clock also increase the chance of RAM malfunction.

[Brendan]

Hi,

As regards 'broken' memory maps, just how extreme can they get? I have seen some with several overlapping areas, where I have always assumed the 'used' regions take priority over the 'free' ones, and have generally regarded anything under 1 MiB as used even if not marked as such, but do some actually report areas as free which don't exist?

In general there's 3 cases:

• things that do comply with the expected behaviour
• things that don't comply with the expected behaviour; but do comply with the literal meaning of the relevant specification
• things that don't comply with anything; that can be classified as "someone else is to blame"

Some people only test the first case. For example (firmware's memory map) they might only test that their code works if the firmware's memory map is sane; and might have code that assumes areas of RAM start/end on page boundaries and never find out that their code is buggy.

Some people test the first case and the second case. For example (firmware's memory map) they might write special code to test that things like "zero size" areas, badly aligned area start/end, unknown area types, etc are all handled correctly by their code.

Some people test all 3 cases. For example (firmware's memory map) they might write special code to test that overlapping areas of any type (e.g. including overlapping "non-volatile ACPI" and "system" areas) are handled correctly by their code (where "handled correctly" may include correctly refusing to boot if the memory map is unusable).

For people that test all 3 cases; it doesn't really matter if systems with problems currently exist or not. What matters is if it's possible or impossible for systems with problems to exist (now and in the future). However; anything that might be possible is possible in the future. For example (firmware's memory map) you can't be certain that no system will ever report areas as free which don't exist.

Of course different people will do a different amount of testing. For example (firmware's memory map), someone might do 50 different tests the second case and no tests for the third case, and someone else might do 1 test for the second case and one test for the third case.

In practice (especially when you start looking at the last case), it can be impossible to test every possible problem. The goal is to ensure that the code is "robust enough" (not to spend an infinite amount of time attempting to obtain "100% robust"); where different people have very different definitions of "robust enough" (e.g. life support systems vs. games console).


Cheers,

Brendan

[Brendan]

Hi,

I was browsing and came across FAUmachine. It's an emulator (based on Qemu I think) that includes a bunch of features for automation and fault injection. I don't know much about it (haven't tried it and can't find any decent end-user documentation), but I thought it might be worth mentioning.


Cheers,

Brendan

[bewing]

This is one of the things I am trying to do with my Rebochs emulator. When it is fully implemented, it will have a selectable mode called "nasty mode" for most devices, and for the BIOS. The entire intention of nasty mode is to test your OS with typical non-standard behavior from the emulator, that you see on actual hardware. Nasty mode is already partially implemented for the BIOS in rebochs. But I am only working on it in my spare time, with no help, so it is going slowly. It will take me time to document cases of non-standard behavior, and time to add it to the emulator.