Systems Software Research is Irrelevant, by Rob Pike

[Arto]

Trammell Hudson blogged today on a topic that should interest most folks here:

In 2000, Rob Pike wrote Systems Software Research is Irrelevant (aka utah2000 or utah2k), on the decline in operating system research. It included a chart lamenting the stagnation in software and programming languages over the ten years from 1990 to 2000. I've updated it for 2014, which supports his point even more strongly:

I've quoted below some relevant choice bits from the aforementioned original polemic by Rob Pike:

If systems research was relevant, we'd see new operating systems and new languages making inroads into the industry, the way we did in the '70s and '80s. Instead, we see a thriving software industry that largely ignores research, and a research community that writes papers rather than software. [...]

Linux's success may indeed be the single strongest argument for my thesis: The excitement generated by a clone of a decades-old operating system demonstrates the void that the systems software research community has failed to fill. [...]

New operating systems today tend to be just ways of reimplementing Unix. If they have a novel architecture -- and some do -- the first thing to build is the Unix emulation layer. How can operating systems research be relevant when the resulting operating systems are all indistinguishable?

There was a claim in the late 1970s and early 1980s that Unix had killed operating systems research because no one would try anything else. At the time, I didn't believe it. Today, I grudgingly accept that the claim may be true (Microsoft notwithstanding). A victim of its own success: portability led to ubiquity. That meant architecture didn't matter, so now there's only one. Linux is the hot new thing... but it's just another Unix. [...]

The world has decided how it wants computers to be. The systems software research community influenced that decision somewhat, but very little, and now it is shut out of the discussion. It has reached the point where I doubt that a brilliant systems project would even be funded, and if funded, wouldn't find the bodies to do the work. The odds of success were always low; now they're essentially zero.

What do you think? Are we doomed to keep reimplementing Unix for the next half a century?

[Cjreek]

Probably. Everybody* uses Windows or Linux. So people are influenced by their experience with operating systems - maybe in some cases despite their efforts NOT to be influenced by those operating systems.

I don't want my OS to be like windows (for example) but as a windows user for (almost) my whole life I can't help but notice that I often do things the way windows does it (or at least close to windows).

The problem also seems to be that there are no serious (at least very few?) indie developers for operating systems. Indie developers who really invest time and money in their project which is understandable because nobody will support an OS on kickstarter for example. And the risk of failing - even with a good concept and a decent amount of money - is just too high because windows and linux are just too huge and the at most 1% chance of being successful is just not worth loosing that much time and money.

But in general an OS indie scene would be beneficial to the future of operating systems I think.
And I don't count (most of?) us to those indie developers. ReactOS may be big enough but it's just a windows clone so there are no innovations.

[Kazinsal]

First thing's first: I haven't seen the name "Trammell Hudson" before, nor does he seem to have any notable projects attached to him on his website (other than "I wrote some network drivers at Lockheed Martin"). There's no real reason for his extension of the chart other than to state "hey, look at the software we've been using on a large scale for the past 20 years" and infer that having a stable base set of software and protocols for the computer world is somehow a bad thing.

The article also doesn't seem to source anything outside of Rob Pike's original article, which itself I find a few flaws with. Notably, the article is written admitting that it's "darker than reality" and focuses solely on the negative. Of course systems research is going to look like a declining failure of a field if you look at it with an "everything is sh!t" attitude.

Perhaps if the article had some substance and not just an opinion that Rob Pike's article was right, I'd be able to agree with it. Unfortunately, I can't. It's a post by some blogger I've never herad of who has no professional or even self-admitted experience in operating systems design saying that operating systems design is dead because Rob Pike said it was 14 years ago.

What do you think? Are we doomed to keep reimplementing Unix for the next half a century?

You can be if you want. I'm more than happy to find a better way to do things.

[sortie]

Who cares about research for the sake of research? We can do this better and deliver real products.

[AndrewAPrice]

First, there are plenty of non-Unix-wannabes being developed. Second, I'll echo Kazinsal - who is this guy to decide?

[Jezze]

Well we will probably not invent something truly groundbreaking so what I think will be the keyword for future operating systems will be refinement and specialization. Linux has been able to adapt to smp and virtualization but not without a cost as in ugly legacy interfaces. What would truly change the landscape would be new hardware that operates significantly different to today.

[konacake]

Yeah, I have to disagree too. I don't really see anything inherently wrong with Unix, it's a nice standard way to implement syscalls so you don't have to heavily modify any software you want to port. I don't think it actually impedes innovation, there's still plenty of stuff you can design yourself, the Unix standard doesn't specify anything about how threads actually work or what the filesystem does, just how to access them.

Also, do we even need that much innovation anymore? Software development is kind of an art, and when creating art you eventually perfect the craft and get bored of it. All that's left to do is remake old things or experiment for the sake of it. Although I'm a bit afraid of using this analogy and calling Windows or Linux "art". But the point is, all that innovating was happening because the field was still "new", there were no standards and everyone had to do everything from scratch, now we have standards and reusable, open code. It was a messy time, now we're a little more organized.

[mathematician]

Well we will probably not invent something truly groundbreaking so what I think will be the keyword for future operating systems will be refinement and specialization. Linux has been able to adapt to smp and virtualization but not without a cost as in ugly legacy interfaces. What would truly change the landscape would be new hardware that operates significantly different to today.

Intel's reported ambition is to have 64 cores on home PCs. If they succeed, it will probably be much harder to adapt to that.

[Combuster]

Talking about new technology and considering how slow the patent office is, it will be three years until Mill CPUs enter production.

There goes another hard brake on innovation.

[shikhin]

Hi,

First, there are plenty of non-Unix-wannabes being developed.

I see two crazy assembly-only OSs, one Unix clone, and two non-exciting projects. Not sure what you were talking about.

Second, I'll echo Kazinsal - who is this guy to decide?

How does that matter if he's right?

Who cares about research for the sake of research? We can do this better and deliver real products.

Not sure what's the point of delivering "real products" if no significant amount of users is ever going to use them. If your plan is to make deliverable products better, surely you'd have better success with removing crud from existing products.

Regards,
Shikhin

[Pancakes]

You guys should also keep in mind that there are lots of jobs out there maintaining, improving, and extending existing operating systems and a good knowledge on the architecture and modern operating systems makes you very valuable in that role. When you take on an operating system project you are forced to learn these things and even if you end up with a clone, partial clone, or something new you have really expanded your understanding of the computer and software.

So yeah maybe you are not going to write the next greatest operating system.. unless you see some radical changes in computer architecture and then it is my opinion and the doors may open up. I think computer architectures have stabilized and most ways of doing things have solidified and anything that can be done better that has not been done is mainly held back with the massive rewriting of software that would be required. Not saying changes are not being made all the time. There are a constant steam of improvements in networking, storage, and graphics - and all of these improvements have really been a part of the operating system at least to some degree in my opinion.

So I think he may be right in that system software research is down, but that does not mean it will forever be like that, and it does not mean that educating yourself in system software is fruitless.

[shikhin]

Hi,

I think computer architectures have stabilized and most ways of doing things have solidified and anything that can be done better that has not been done is mainly held back with the massive rewriting of software that would be required.

It's sad that you say that because I very recently rewatched a talk by Bret Victor (someone posted it on the forums, http://vimeo.com/71278954). At the end of it, he says that the reason why great innovation happened in the 1960's and 1970's is that although we had sufficient knowledge about things, we didn't have stagnated ideas on how things should be done. We didn't think of programming as just involving long streams of text, so we were willing to explore other ways. Once we do start having solidified ideas about everything, dogma sets in.

Cheers,
Shikhin

[Arto]

I think computer architectures have stabilized and most ways of doing things have solidified and anything that can be done better that has not been done is mainly held back with the massive rewriting of software that would be required.

It's sad that you say that because I very recently rewatched a talk by Bret Victor (someone posted it on the forums, http://vimeo.com/71278954).

The forum thread for that was: The Future of Programming, circa 1973.

And indeed, the people here working on the Mill CPU might be able to provide some solid counterevidence to the assertion that "ways of doing things have solidified". We're hardly at the end of history, here.

[Pancakes]

Hi,

I think computer architectures have stabilized and most ways of doing things have solidified and anything that can be done better that has not been done is mainly held back with the massive rewriting of software that would be required.

It's sad that you say that because I very recently rewatched a talk by Bret Victor (someone posted it on the forums, http://vimeo.com/71278954). At the end of it, he says that the reason why great innovation happened in the 1960's and 1970's is that although we had sufficient knowledge about things, we didn't have stagnated ideas on how things should be done. We didn't think of programming as just involving long streams of text, so we were willing to explore other ways. Once we do start having solidified ideas about everything, dogma sets in.

Cheers,
Shikhin

You may be right! Thanks for keeping me reminded. =)

[Geri]

currently, the system codes are overcomplicated, becouse the x86/arm io is overcomplicated itself. why the hell we even have interrupts? insanity! its too large to everything can be properly understanded. its an ugly dinosaur living within us. just let these platform slowly die, and THEN we can build real operating systems and platforms, whichs are worths of researching.