CPUID forgery detection...

[Combuster]

I see Smiddy make the point, but I'll confirm it anyway.
Problem is that some bioses have faulty code of their own, making you think you have a processor you do not BY ACCIDENT. Given the amount of non-spoofed bioses in existance, and the amount of revisions they sometimes hand out (i flashed my VIA mobo thrice the first year), i think that adds another problem to the issue: Am i being lied to, did some boot code make a mistake, or is this information genuine?

Talking about bugs: some processors' CPUIDs dont return the correct information as well - if you have read the intel manuals you'll probably have noticed half a page dedicated to one such bug (IIRC, fast system calls was mistakingly set)

I wont go the discussion what to do if the processor's info is spoofed on purpose. However, what if the information is incorrect by accident, then we ought to fix that. And detecting cpu type isnt that much of astrology - a simple DIV can already tell apart different processors, and apart from microcode updates, theres no way to spoof this one.

For me, the kernel should provide the most accurate information to the user. So i have a system call providing such information. That way i can provide the user with accurate information. Saving them from looking up known bugs as well as saving them from distributing patches, as you only need to update the kernel to check new processors.

For the rest i agree with Brendan, that fixable bugs should be fixed, independent of the processor saying its something its not, as some fixes might or will break other processors (like, writing Cyrix MSRs will go to the system bus on non-Cyrices. Not that some creep would try to replace the vendor string with "CyrixInstead" )

[smiddy]

Maybe I'm missing the point here or my own paradigm is askewed. Here's a thought that occured to me, we're all assuming that every boot of your OS you would check this, meaning either placing it in your boot code/loader or your kernel code. Would this be something that might a part of an installation code? In other words you would have a setup file that identifies all the expected devices that was written from the installation routines. Once the kernel is loaded it uses this file created by the installation program for the parameters it needs in order to run the kernel. It would likely be a safe bet that catching inadvertant BIOS issues would be caught then, it wouldn't catch problems after installation, or if the BIOS was updated, but at least you won't have the overhead of every boot checking, unless that is what you want to do...or perhaps running an occasional monitor of the CPU...

[Brendan]

Hi,

They could always release new hardware that does a better job of preventing you from detecting the processor's true identity.

AFAIK that's never the intent of the manufacturers. Old CPUs allow CPUID values to be changed so that it works properly with badly written "Intel only" software, and some CPUs do it for marketting purposes.

For example, if a manufacturer makes 1000 CPUs, some of them might fail testing at full speed and some might have faulty caches. In this case it makes sense to sell some of them as slower CPUs or CPUs with half the cache disabled, and use a different brand name (if you've ever wondered what the difference between a Celeron and a Pentium II/III/4 is, take a look at the cache sizes). This creates a small problem though - they don't know which brand string to use when they manufacture the CPU. I'd guess Intel set it using something like an initial micro-code update, while AMD expect the BIOS to figure it out.

At a certain point, you just have to say that your software may not work on broken hardware. CPUID is a nice, clean way to figure out what you can expect from a processor, unlikely to encourage bugs, and if that processor attempts to trick the OS, it can't be expected to work.

With all the bugs, badly designed fields, manufacturer specific extensions and other problems, CPUID has become one of the ugliest messes of the entire architecture. Just for fun, try writing some code to get the size of the L1 code & data caches that works on all CPUs that return this information (if you can get 2 integers in less than 100 lines of code you've missed something).


Cheers,

Brendan

[Brendan]

Hi,

Ok, I ran into some troubles and partially changed my plans to suit, so I thought I'd describe exactly what I'm doing (and cover Smiddy's "do it during OS install" idea at the same time).

First, the troubles:

• For AMD CPUs the official brand name string selection algorithm (intended to be done by the BIOS) relies on accurately measuring the speed of the CPU, and in some cases depends on the L2 cache size, the CPU speed and the FSB speed (e.g. AMD Athlon MP and XP CPUs) . I'm not convinced I can accurately measure the speed of the CPU in all cases (taking into account SMM and power/temperature management, that can make the CPU seem slower than it really is).
  
  For VIA/IDT/Centaur CPUs I can't find enough information for some models (Ezra-T, Esther), and for most of them different branded CPUs are practically identical - there's the "normal" model (like "C3/Samuel 2") and the equivelent "Eden ESP ???" model. I don't know what the difference is, but family/model/stepping, cache size, features, etc are all identical (might just be a different CPU socket for all I can tell).
  
  For GeodeGX2 and similar, if I do know it's one of these chips I can't do much about it because all CPUID information can be false. I can correct the vendor ID but that might be all, depending on what was changed.
  
  For SiS I can't get any information at all.

Now for what I'm doing...

During boot, a boot loader loads a boot image into memory which contains everything used after that (until some device drivers are working). All code is modular, where modules come from the boot image. The boot loader starts a "boot manager" module, which uses other modules including the CPU detection module.

This means I can have 100 different CPU detection modules without much trouble, and it will be possible for other people to write custom CPU detection modules. For e.g. for an embedded system where the OS is burnt onto a ROM and the entire system is sealed the manufacturer would probably replace the CPU detection code with some static "the CPU is X" code that doesn't do any detection. There's also a boot script (a "<variable> = <value>" format) which can be used to control the behaviour of any boot modules. Despite all of this, for normal desktop/server situations the code I'm writing is fast and compresses well, so there shouldn't be much reason to skip this checking each boot.

The CPU detection module is responsible for detecting how many CPUs there are, the relationships between the CPUs, and what each CPU is (features, caches, etc). Some of the results are used to determine which modules are used later on (for e.g. if all CPUs support long mode the OS automatically decides to use 64-bit kernel modules). This may involve loading a CPU specific kernel module to support vendor specific CPU features (like MTRRs, power management and machine check exception).

In addition, the CPU detection code sets 3 groups of flags. The first group is which features the CPU actually supports reliably (not what it says it supports), where each flag uniquely represents a specific feature, including things like 3DNow, cryptography extensions, VMX, SVM and power management (LongRun, LongHaul, etc).

The second set of flags ("CPU errata") are used for CPU bug work-arounds, for e.g. if a Pentium is detected the "F00F" flag would be set and later code would implement the work-around for it.

The third set of flags ("CPU flaws") are used for warning system administrators about CPU bugs that can cause "erratic behavior or reduced stability". The idea here is that the system administrator can easily decide if the CPU should be used or not. For a critical 24/7 server you might not want to use something that locks up occasionally due to bus problems (some Pentium II chips). For a server in a bank used for handling large transactions, calculating interest, etc, you might not want a CPU with certain FPU accuracy problems (Pentium).

There's also a smaller set of "instruction set" flags used to determine which applications can run (MMX, extendedMMX, 3DNow, extended 3DNow, SSE1/2/3). For e.g. if an application requires MMX it can set a flag in it's header that the OS will check before allowing it to start. In some cases (e.g. an SMP machine with mixed Pentiums) this can mean the application is restricted to the CPUs that support MMX and prevented from running on CPUs that don't.

Now, getting back to the "CPUID forgery", it will go like this. If the CPU is unforgable or if the CPU detection code can reliable detect the correct details, then everything is good. If the code can't reliably detect the CPU brand string, then it'll set a "CPU errata" flag to let others know that the brand string may be wrong (but it won't effect the OS). If the family/model/stepping details can't be detected reliably then it'll set a "CPU flaw" flag to indicate that (under certain conditions) erratic behavior or reduced stability is possible. If the CPUs features can't be detected reliably, the OS will just use a reduced set of features.

This means for most CPUs everything is fine. For some AMD CPUs you'd get the "brand string might be wrong" errata flag. For some IDT/VIA/Centaur CPUs and some NSC/Geode CPUs you might get the errata flag and the "may be unstable due to wrong CPU model" flaw flag, and for some NSC/Geode CPUs you might also get a reduced set of features.

In addition, there's another flaw flag which is set when my code doesn't know if the CPU has bugs or not. For most Intel and AMD CPUs the information is available from the manufacturers and this flag can be cleared (once I write all the CPU bug detection code for the CPUs). For almost all other manufacturers and for older 80486s this flag would be set.

The end result is that system administrators will be able to easily figure out which potential problems each computer has (without spending hours looking through manufacturer documentation), and my OS can (potentially) become far more reliable than OS's like Windows, Linux, etc.


Cheers,

Brendan

[smiddy]

I think you've nailed this pretty well. By making the potential flaws known the user/administrator can make decisions based on that information (as can the OS).

I am currently going down the system management BIOS path in my own research (one I expect you've already gone down) and am finding that nothing is set in stone and nearly everything can be forged one way or another. The permutations are endless. This solidifies the perspective of announcign the suspected issues thus trying and failing or succeeding with probes if so desired by the user/administrator is a way to approach it. The frustration of the ever changing world of hardware (which I hope doesn't stop anytime soon) adds to the complexity of what you're trying to accomplish. As a hobby it is difficult to keep on the front side of the wave (bleeding edge). Those with the financial backing and resources can further their OS if they wish, but then there is at some point a baseline and not allowing requirements to creap. Ah, the never ending battle...

Good luck, I will be anxiously awaiting a preview of what you finally come up with.

[Seven11]

In some cases (e.g. an SMP machine with mixed Pentiums)

Is that even possible? All SMP motherboards I have seen so far won't even allow processors with different model/stepping or frequency. Can you have different processors in a NUMA system?

I am currently going down the system management BIOS

I'm thinking about checking the detected hardware info against the SMBIOS to see if the results endup, anyone tried that?

I didn't even know that the CPUID info could be altered (except for Intels Micorcode update), what sort of tests are your cpu_info model conducting Brendan?

[Combuster]

Is that even possible? All SMP motherboards I have seen so far won't even allow processors with different model/stepping or frequency. Can you have different processors in a NUMA system?

In general, you can have any set of cpus on a board, different clocks, different brands, even different architectures - iirc a sega genesis (with upgrades) has a zilog Z80, two motorola 68k (one at 8mhz and one at 12) and two hitachi chips. That does not mean its easy to program though

As for the x86, i think you could logically put different speed cpus on a board, as long as the FSB speed remains the same, as the difference in clock only propagates within each cpu. I'm not sure though.
And there are more clues in this direction - I run a smp linux kernel on my own dual processor board, and it runs all processor checks twice, instead of just once.
Apart from that, Brendan's the expert in this area so...

I'm thinking about checking the detected hardware info against the SMBIOS to see if the results endup, anyone tried that?

If both the CPUID info and SMBIOS can be forged, it wouldnt be worth trying to detect the correct values by comparing the two, although if they dont match you know somebody did only half his work.

what sort of tests are your cpu_info model conducting

If you've read the first post, you'll probably have noticed that its supposedly about wether its worth the time or not. Nevertheless, there are some standard checks that can be done. To get the model, you can test eflags (usually seen as the de-facto test to check for a 386 or better), you can run a DIV test, which will get you the manufacturer. For new cpus, you can test all known cpuid entries for presence and contents to see what you have.

[Seven11]

As for the x86, i think you could logically put different speed cpus on a board, as long as the FSB speed remains the same, as the difference in clock only propagates within each cpu. I'm not sure though.

Yes that is the logical conclusion but I'm not so sure motherboard, BIOS and chipset makers agree with you, do you have an example of any board/chipset that allows this?

I run a smp linux kernel on my own dual processor board, and it runs all processor checks twice, instead of just once.

Of course it does... it runs det cpu init for both processors, and for every init there should be a function who detects the presence of special extensions (in this case (MMX, SSEx...)

although if they dont match you know somebody did only half his work.

Yes but who? It could be my OS that only detected some of the hardware and its extensions or it could be the BIOS who spoofed some of the info (as in smiddys example with the CPU)

[Combuster]

Uh right. Pretty smart questions i guess.

mixed cpus: if i had an example, would i be saying "i think you could"
smp init: you're kindof proving my point here
smbios: you dont know who, but you can be sure its NOT you. If you are the one that did half work you'll end up having too few information to conclude either one is spoofed, and consequently you cant conclude that somebody did half work. You need to be good enough to prove the statement.
Of course, that only goes as long as your code isn't bogus

[smiddy]

smbios: you dont know who, but you can be sure its NOT you. If you are the one that did half work you'll end up having too few information to conclude either one is spoofed, and consequently you cant conclude that somebody did half work. You need to be good enough to prove the statement.
Of course, that only goes as long as your code isn't bogus

Depending on who you ask you can get different answers here too. I assume you mean bogus (not sure where that technical term lies) as in an individual producing their own OS may not have all the information and only the information they know can be raight to their own references, yes. Then again you can have outdated information (which given the term above, can be bogus too) which was good yesterday but today doesn't work because something so dynamic has changed and the OS writer hasn't released an update in order to capture the oldness, but will once the new release is produced.

From what I've read system management BIOS was a way to have a seperate entity to produce a standard to which everyone could use in writing an OS. Unfortunately not everyone want/need/has to/does comply with the standards. And there are those with inside information that can use that information to their own benefit from a competitive standpoint.

As for ACPI, which I've only done a cursory overview of, is supposed to incorperate items PnP, SMBIOS, DMI, and others as well as have a seperate functioning system outside the OS and the machine, a limbo area so to speak (there's my technical term, limbo). This specific interface tries to also do this...but you always have issues.

Suffice it to say, everything can be spoofed, but you have to have a certain level of trust in order to move forward otherwise whats the point?

[Brendan]

Hi,

I am currently going down the system management BIOS path in my own research (one I expect you've already gone down) and am finding that nothing is set in stone and nearly everything can be forged one way or another. The permutations are endless. This solidifies the perspective of announcign the suspected issues thus trying and failing or succeeding with probes if so desired by the user/administrator is a way to approach it. The frustration of the ever changing world of hardware (which I hope doesn't stop anytime soon) adds to the complexity of what you're trying to accomplish. As a hobby it is difficult to keep on the front side of the wave (bleeding edge). Those with the financial backing and resources can further their OS if they wish, but then there is at some point a baseline and not allowing requirements to creap. Ah, the never ending battle...

A few years ago I did a very small SMBIOS test - I downloaded some software to display the details it returns and ran it on one machine. It reported wrong information. After a little searching I found plenty of people also getting wrong information, and FAQs like this one.

The problem is that SMBIOS is designed to make it easy for system administrators to create an inventory of hardware for maintenance/upgrade purposes, and isn't intended to be used by software for configuration purposes. Because of this BIOS manufacturers tend to have worse quality control for it, as they assume it's going to be used for it's intended purpose and therefore won't effect the reliability of software.

Good luck, I will be anxiously awaiting a preview of what you finally come up with.

Thanks .

So far I've got most of it "done enough". For some AMD CPUs I could make sure the brand name string is right instead of setting the "brand may be wrong" flag, and I could do a little more to detect caches for CPUs that don't return this information from CPUID. Then there's "CPU errata", which is going to take ages (reading through manufacturer revision guides and specification updates, determining what effect each CPU bug will have on my code and classifying them) - so far I've only done Intel CPUs from 80486 to Pentium II properly. All of these things can wait until later on.

It also doesn't detect the number of logical CPUs per core or the number of cores in the physical chip, and I haven't done the MP specification table scan, the ACPI table scan, or the AP CPU startup code (it's "single-CPU only" at the moment). These are my next steps - when I've got "multi-CPU" working right I'll release it and move on to other things...

As for the x86, i think you could logically put different speed cpus on a board, as long as the FSB speed remains the same, as the difference in clock only propagates within each cpu. I'm not sure though.

Yes that is the logical conclusion but I'm not so sure motherboard, BIOS and chipset makers agree with you, do you have an example of any board/chipset that allows this?

AFAIK, as long as the FSB is the same, the chip fits the socket, and there's no incompatabilities with bus signalling (e.g. same cache line sizes) then it should work. I have heard of a few cases where people have used different CPUs in an SMP system (similar CPUs with different internal clock multipliers) and found that it worked except for OS related timing problems. Almost identical processors (e.g. mixed steppings) is fairly common.

Intel's Multi-processor Specification has a paragraph that describes things nicely:

B.8 Supporting Unequal Processors

Some MP operating systems that exist today do not support processors of different types, speeds, or capabilities. However, as processor lifetimes increase and new generations of processors arrive, the potential for dissimilarity among processors increases. The MP specification addresses this potential by providing an MP configuration table to help the operating system configure itself. Operating system writers should factor in processor variations, such as processor type, family, model, and features, to arrive at a configuration that maximizes overall system performance. At a minimum, the MP operating system should remain operational and should support the common features of unequal processors.

Unfortunately "some operating systems that exist today" decided to take the easy way and support almost identical CPUs only. I'd assume BIOS and motherboard manufacturers also favour "almost identical CPUs only", as it reduces the amount of testing required. For e.g. with 8 different CPU types and a dual socket motherboard, you can test them in identical pairs and do 8 tests, or test each possible combination and end up doing 64 tests. Despite this there's a difference between "officially supported" and "works in practice".

My OS assumes that all CPUs may be radically different (e.g. an Intel Pentium, VIA Antaur, AMD Opteron and Intel Core2 in the same system). The amount of variation between CPUs is limited by hardware (motherboard and BIOS) and nothing else.


Cheers,

Brendan