"The Linux Scheduler: a Decade of Wasted Cores"

[mallardest]

Hi, I found this paper online and I thought that some of you might find it interesting.

http://www.ece.ubc.ca/~sasha/papers/eur ... inal29.pdf

13% higher latency for kernel make, and a 14- 23% decrease in TPC-H throughput for a widely used commercial database

[Solar]

The paper completely aside, it's the Torvalds quote that cracked me up.

And you have to realize that there are not very many things that have aged as well as the scheduler. Which is just another proof that scheduling is easy.

-- Linus Torvalds, 2001

That guy is such a smug {bleep} that I really don't understand why certain people revere him so much. He made things happen, granted. Bill Gates did that as well... Linus might (might!) even be a competent developer. But that's where I draw the line regarding that bloke...

[mallardest]

The paper completely aside, it's the Torvalds quote that cracked me up.

And you have to realize that there are not very many things that have aged as well as the scheduler. Which is just another proof that scheduling is easy.

-- Linus Torvalds, 2001

That guy is such a smug {bleep} that I really don't understand why certain people revere him so much. He made things happen, granted. Bill Gates did that as well... Linus might (might!) even be a competent developer. But that's where I draw the line regarding that bloke...

I think he's being sarcastic in that quote because the Linux scheduler is changed all the time.

Also there's this quote too:

I'm an egotistical bastard, and I name all my projects after myself. First 'Linux', now 'Git'

[Schol-R-LEA]

I'm an egotistical bastard, and I name all my projects after myself. First 'Linux', now 'Git'

Unfortunately, that joke doesn't work for Merkins unless they know a bunch of Brit humor; over here, 'git' is a regional pronunciation of 'get', not a synonym for a-hole.

[b.zaar]

That guy is such a smug {bleep} that I really don't understand why certain people revere him so much. He made things happen, granted. Bill Gates did that as well... Linus might (might!) even be a competent developer. But that's where I draw the line regarding that bloke...

I think he has a personality disorder, something like narcissism, which isn't something people choose to be or even recognise themselves that they have a disorder. I'm not a fan of him either but I'll always watch a video interview with him as he made a successful software project and there's always something to be learnt from others. I don't admire Steve Jobs or Bill Gates either but I watch their interviews too.

The guys I really admire tend to hide behind the scenes and focus only on the software that they need for their own (or clients) use. Unix was invented in house without financial incentive, similar with CP/M, the Xerox UI and the computer mouse.

As for what he's done, he cloned some software and did it better than the other guys. Games do the same thing, angry birds == crush the castle and candy crush == bejewelled.

[jojo]

I've always gotten the impression, after watching many an interview and what-have-you, that people take the guy way too seriously. He's obviously a relatively humble and jokey person who, like the rest of us (and look at this particular board as a rather egregious example at most times), has some strong opinions about certain things.

But if there's one constant I've noticed about him, it's that he's always witty and self-deprecating and almost constantly jokes about how often his own strong opinions change.

I think the real thing here is that this is pretty much the way most of us technical nerds are when it comes to our opinions, and most of our opinions are silly and religious and we hold onto them because that's what we do. And we like picking on Torvalds because he's an easy icon who represents everything we can project onto a person about our own place in our weird little culture, and it's nice to have a lightning rod.

I mean, the guy has made some bad decisions re: linux in the past, sure, maybe, I don't really know. But come on, guys, we're on a forum that's full of people working on always unfinished pipedream projects with a respectable contingent of relative crackpots in its ranks. And I count myself among that group, and am semi-proud to identify as such. But cast not the first stone and maybe chill a bit.

[b.zaar]

I've always gotten the impression, after watching many an interview and what-have-you, that people take the guy way too seriously. He's obviously a relatively humble and jokey person who, like the rest of us (and look at this particular board as a rather egregious example at most times), has some strong opinions about certain things.

But if there's one constant I've noticed about him, it's that he's always witty and self-deprecating and almost constantly jokes about how often his own strong opinions change.

There is a difference between strong opinions and plain insult: https://en.wikiquote.org/wiki/Linus_Tor ... rnal_links.

People on this board do have strong opinions that differ but I rarely see straight up insult.

I'm a bastard. I have absolutely no clue why people can ever think otherwise. Yet they do. People think I'm a nice guy, and the fact is that I'm a scheming, conniving bastard who doesn't care for any hurt feelings or lost hours of work, if it just results in what I consider to be a better system. And I'm not just saying that. I'm really not a very nice person. I can say "I don't care" with a straight face, and really mean it.

Torvalds, Linus (2000-09-06). Message to linux-kernel mailing list. Retrieved on 2007-05-28.

And I believe him.

If he was working for a commercial company there's a good chance he would have been fired like Steve Jobs was years ago. I don't think Kernighan, Richie and Wozniak would have been able to work this way and still be part of the business.

[TverrBjelke]

Nerd showing a paper about Linux Scheduler ... and inevitably Nerd-Folks slide into ranting about Linus? Yeah!

I did a (brief) overview about this paper... so lets see...

Spoiler (TL;DR)

“Nobody actually creates perfect code the first time
around, except me. But there’s only one of me.”
Linus Torvalds, 2007 [44] (from page 5 of this paper)

I actually LIKE this insane and sarcastic troll of a guy (and am quite glad that I do not work near him)

Now what is the paper really about... new hardware had turned the once already solved scheduler problem
into a complex nightmare and its implementations full of bugs...
So they have crafted new tools to detect invariant violations and proposed bugfixes

Well, AND they could not resist to spice it all by citing Linus endless reservoire of sayings

################################

scheduler must maintain the following, simple, invariant:
make sure that ready threads are scheduled on available
cores.

Our recent experience with the Linux scheduler revealed
that the pressure to work around the challenging properties
of modern hardware, such as non-uniform memory access
latencies (NUMA), high costs of cache coherency and synchronization,
and diverging CPU and memory latencies, resulted in a scheduler with an incredibly complex implementation.

The bugs we described resulted from developers want-
ing to put more and more optimizations into the scheduler,
whose purpose was mostly to cater to complexity of modern
hardware.

This occurs even though the scheduler is not explicitly configured to save
power by purposefully leaving cores unused so they can be brought into a
low-power state.

We provide fixes to these bugs, and observe substantial performance improvements.

[onlyonemac]

If he was working for a commercial company there's a good chance he would have been fired like Steve Jobs was years ago. I don't think Kernighan, Richie and Wozniak would have been able to work this way and still be part of the business.

...which just proves why open source software can be such a successful development model. You can have many people pooling together their development knowledge, skills, and experience without having to get along with each other on a personal level.

[Solar]

...which just proves why open source software can be such a successful development model. You can have many people pooling together their development knowledge, skills, and experience without having to get along with each other on a personal level.

Err... do you have actual experience being employed in software engineering?

Teams aren't put together "because you get along". You aren't hired because you "get along" with other team members, and unless you are also lacking in technical skills, you aren't fired for "not getting along" with others either (extreme cases notwithstanding).

But you tend to "get along" in the corporate field because you are spending 8+ hours in the same office every working day, and don't want to make your own life hell. So you swallow your pride from time to time, and compromise.

Enter Open Source development, where communication is usually done via the internet and without a clear hierachy of superiors to appeal to if things go sour... yeah. Doesn't take much imagination to picture how great this will work if you don't "get along".

----

That being said, I remember the first day at a company, being presented my new colleagues. I don't remember the exact words anymore, but the general message was, "meet Mr. D.; he's from an external company. Your first assignment is to make him redundant as soon as possible so we can terminate his contract."

Oh joy...

[onlyonemac]

...which just proves why open source software can be such a successful development model. You can have many people pooling together their development knowledge, skills, and experience without having to get along with each other on a personal level.

Err... do you have actual experience being employed in software engineering?

Teams aren't put together "because you get along". You aren't hired because you "get along" with other team members, and unless you are also lacking in technical skills, you aren't fired for "not getting along" with others either (extreme cases notwithstanding).

But you tend to "get along" in the corporate field because you are spending 8+ hours in the same office every working day, and don't want to make your own life hell. So you swallow your pride from time to time, and compromise.

Enter Open Source development, where communication is usually done via the internet and without a clear hierachy of superiors to appeal to if things go sour... yeah. Doesn't take much imagination to picture how great this will work if you don't "get along".

----

That being said, I remember the first day at a company, being presented my new colleagues. I don't remember the exact words anymore, but the general message was, "meet Mr. D.; he's from an external company. Your first assignment is to make him redundant as soon as possible so we can terminate his contract."

Oh joy...

Perhaps you should direct your post at b.zaar, because he's the one that suggested that people would get fired if they didn't get along with the rest of the team; I was merely building on what he had said.

[JoeEagar]

I think that's a pretty common viewpoint among younger workers. Remember, those of us under 35 did not benefit from our governments eviserating their economies with inflation just to get us employed. When one comes of age in a slack labor market one learns to care about teamwork, as otherwise one is liable to be replaced.

[Solar]

Perhaps you should direct your post at b.zaar...

Yea... I rambled a bit all over the place, didn't I?

Old age.

[rdos]

At least the paper shows the complexity of it. Also, it's not a problem that can be solved with a huge amount of complexity really, because at least the low-level scheduler doing the real-time scheduling should be as fast as possible, and thus simple.

[willedwards]

We presume that all the other OS have decades of wasted cores too.

This paper shows a way to find out and quantify it. I hope other OSes get tested and fixed too.