Engineering for active threats versus passive ones

[Schol-R-LEA]

I was already thinking of linking to yesterday's XKCD here, but it is something that the Explain XKCD wiki page for the comic that really caught my attention, as it put into words something that is rarely said so clearly.

First, the cartoon in question:


The part of the explanation that caught my eye was this:

This is a result of a fundamental difference between computer security and other types of safety measures -- in cryptography, there is always somebody trying to undo what you've built. Not only that, but new advances in cryptography tend to point out vulnerabilities with previous versions, making them not only obsolete, but dangerously so.

This is actually a very big point that I think everyone sort of knows, but few really take into consideration in such a lucid fashion. It is as if every bridge designer had to include active protections against an army of saboteurs who not only could but invariably will be actively trying to bring the bridges down, every day for as long as the bridge stands - some of whom aren't targeting the specific bridge, just randomly targeting any bridge they find.

This leads to a situation where even game developers have to take a stance closer to military counter-intelligence than entertainers. It colors the entire field in a way which is so pervasive that it fades into the background, yet most developers ignore the topic, blithely assuming that the operating system and development tools will protect them, despite (or perhaps because of) the fact that no software can fix the biggest security risk - human nature and the strong possibility of the users being careless, ignorant, tired, intoxicated, greedy, gullible, or in some other way vulnerable to social engineering tactics.

Discuss.

[Solar]

Even the XKCD wasn't as lucid as it could have been.

Modern aircraft are incredibly resilient. Against weather, mechanical failure etc.; but not against someone exploding a bomb to bring it down.

Elevators are protected by multiple failsafe mechanisms. But not against someone intentionally taking out those mechanisms.

That is the one angle.

The other angle is that aircraft and elevators need to undergo rigid testing before getting a type permit (or whatever the English word is...). Software, however, is by definition a one-shot affair -- and some specific exceptions nonwithstanding, there does exist neither formal procedure nor authority for actually testing it. Not even against simple stupidity (which has been found in more than one computerized voting "solution"), let alone malicious tampering.

What is rigorous testing in the case of airplanes and elevators, is replaced with marketing and lobbying by software companies that want to make the quick sale. You basically have to take the manufacturer's word for it.

Aircraft are produced by Boeing, Airbus etc.; elevators are produced by ThyssenKrupp, Mitsubishi, Fujitec etc.; all these companies are interested in maintaining an image of quality.

Name a manufacturer of computerized voting software who's made a name for delivering quality. Anyone? This is a government call for bids we're talking here. That usually goes to the lowest bidder.

My favourite metaphor is a bridge engineer, who has learned the characteristics of steel, concrete, stone etc. while in training, and who can rely on knowledge acquired by bridge engineers over many decades. "Please build a bridge at this location." -- "{doing calculations} The best thing will be a suspension bridge. This will require X tons of steel bars and Y meters of gauge Z cabling. We will have to build foundations that need to be A meters deep and B meters wide. The overall cost will be roundabout THIS, unforeseen problems notwithstanding."

"OK."

Now compare that to a software engineer, who pretty much by definition will encounter something new in every project (because otherwise we'd be using the existing software, wouldn't we?). "Please build this software." -- "{doing calculations that involve a lot of guesswork} The best thing will be technology X. It will require about Y months, including Z months of testing. We should have a probation phase of A months before we can be sure the software works as intended. The overall cost will be roundabout THIS, unforeseen problems notwithstanding."

"No, we already decided on a different technology. It has to be finished in half the time; just skip the lengthy testing, and nobody needs documentation anyway. And you must be kidding about the price; we already made an offer and your budget may not exceed that number. You better get it working or you'll be fired."

See the difference? No bridge engineer worth his salt would work under these circumstances. He'd throw down his slide rule and leave, secure in the knowledge that any other bridge engineer would refuse those requirements as well.

But there is always some software company that says it can do it... and then delivers the equivalent of a rotten plank labelled "bridge".

And all that is still not taken malicious tampering into account.

[Schol-R-LEA]

So what your saying is, to quote Peter Welsh, "we don’t even worry about [crackers trying to trash your system or steal your information] because another nuke doesn’t make that much difference in a nuclear winter"?

I can see that. (or at least hear it. I doubt you will see the video below, hence the link to it here, but that's OK, because it's just a reading of the same essay.)



I've come to the conclusion that commercial software development isn't a career, it's an abusive co-dependent relationship with a spouse whom you never leave no matter how horrible they are to you because they keep enabling your addiction to awesome hacks and the power trip you get when things actually work.

[Solar]

Or you make yourself a niche and enough of a name in the department that, when you say "nope, not going to fly", that your superiors are actually listening.

But it's a handful of work, I can tell you.

[eekee]

I know this thread is a bit old now, but I recently learned for certain that it's not just software. I was talking to an engineer the other day. He made 'robots' for various purposes, from pick-and-place machines to teleoperators for keyhole surgery. He tried to do things properly and straightforwardly, such as using a linear track when something had to move in a straight line, instead of a rotational movement with complex calculation to make it go straight. (He complained a lot about other companies doing that.) He had a good sense of the physics involved when making a robot faster. He would make sure the mechanics of the machine worked properly instead of passing it off with, "We'll fix that in software." It seems very few companies actually work like this. He had to put his prices up by a factor of about 5, otherwise buyers would ignore him, thinking "It can't possibly be that cheap."

I don't know any more about his field, but in software I know many companies are just not interested in paying for designs, only feature labels. If a feature not on the spec sheet costs nothing, if it 'falls out of' other parts of the design, it doesn't matter to them. They WILL NOT buy a design with that feature. I think it must be major component of the cost disease infecting this world.

I don't entirely know whether it's a supply problem or a demand problem. Bridges existed and were known engineering challenges before, I think, this lunacy became normal. Self-propelled vehicles too, but not all their modern components. Or, perhaps it's only the things which can be designed by one person which are immune to this, wheras things which are designed by many people and sold as a unit get infected. But then that engineer was doing with a very small team what other companies employ hundreds or thousands of people to produce, and there had been others like him before.

I used to get angry about this sort of problem in the world, I wanted to do something about it, but I never figured out what. In the end, I gave up, returning to believing "the whole world is lying in the power of the wicked one," and looking to God to fix this root cause of problems in His appointed time.