SPICE: lots of theoretical wankery that may someday be an OS

[commodorejohn]

It's a quiet Saturday evening and I'm waiting for the laundry to be done, so I may as well register on an Internet forum and introduce myself by way of a long, rambling post on how I'd like to design an operating system!

I started thinking about OS development about three years ago, when a few key developments took place in my computing lifestyle. First off, in November of 2011, I finally gave up on the idea of switching to Linux. Man, I tried to drink the Flavor-Aid, but after a miserable three-month full-switchover attempt capping off a good seven years of periodic screwing around with a whole bunch of distros and running into the same problems (ridiculously complex yet unbelievably primitive system architecture, the complete lack of a unified, redistributable binary format necessitating both the use of poorly-maintained repositories and building from source in order to get a full complement of useful software, and astonishingly poor, inconsistent UI owing to a graphics system that doesn't so much separate Mechanism and Policy as lock them in separate cells in separate dungeons in separate fortresses on separate continents and then summary-execute Policy for good measure, and to a developer culture that worships the command line as though it were handed down by God Almighty) on every single one of them, I reached the point where I realized that Linux is just fundamentally broken on an architectural and cultural level, and is never, ever going to be a pleasant, usable desktop operating system.

I was so disgusted with that mess that I very nearly swore off the idea of alternative operating systems altogether, even though that would leave me to try and resign myself to the increasingly retarded UI of post-XP Windows for the rest of my life. I may have gone through with that anyway, if it weren't for the fact that I'd also been reading through Andy Hertzfeld's folklore.org, a collaborative history of the development of the Macintosh. I grew up on 68k Macs, and it was both a nostalgia trip and a revelation - I knew the general textbook blurb of "Mac OS was influenced by projects from Xerox PARC," but actually reading about what those influences were and how they got there, in anecdotes about and by Mac Team member and actual PARC alumnus Bruce Horn, was astonishing. It led me to look into the details of SmallTalk and the Alto, and threw into sharp relief just how much and in just how many ways Unix/Linux, OSX, and even my OS-of-choice Windows NT/XP are a giant step backwards from the actual work that was being done at PARC in the early- to mid-'70s, or even from the simplified versions of those ideas that were built into classic Mac OS.

The final influence took a lot longer to manifest, but it was at this time that I started looking into Haiku. It seemed like a very sensible, clean design for an operating system, and I was convinced that once it finally "got there" with regards to hardware support and application software, it would be basically my ideal OS. Unfortunately, three years later, it not only hasn't gotten there, it's made some problematic strides backward, pulling in terrible Linux concepts like package management and repositories and sidelining the actual development work to do it. It now has a useless package manager, but it still divides things confusingly down the lines of which versions of gcc are supported, and it still doesn't have a decent wi-fi client or working power management on multicore systems - Progress! What a shame.

From that, I took away some crucial lessons: A. if you want something done right, you gotta do it yourself, B. getting tied down to architectural details is a sucker's game, and C. Unix/Linux-worship is poison, pure and simple. So at this point, I started thinking about what my ideal computing environment would be like, and how to go about implementing it. I was spurred on in this by exposure (via my overly-zealous supervisor "upgrading" all our workstations) to Windows 8, solidifying my certainty that if this is the future of Windows, I don't want any part of it. On the theory that I might be able to focus more on this project if I discuss it with other people, I figure I'll share my thoughts here - but I'm not making an announcement thread just yet, since I don't have so much as a single line of code written. This is still all just ideas at this point. For now, I'm calling it SPICE (Sensible, Pleasant Interpreted Computing Environment,) which nicely sums up my design goals.

But the laundry is now done, and it's late, so further yammering will follow as I get the time.

[iansjack]

This is still just ideas at this point.

I read your long post, but I couldn't actually discern any ideas in it. Do you have some sort of outline of what you are planning and how it is going to solve the problems that you see in Unix/Windows, etc.?

I would make one suggestion - choose a name that isn't already used for a well known piece of software. It's a bit like calling it "Blender".

[shikhin]

Hi,

[...] pulling in terrible Linux concepts like package management [...]

What's wrong and so terrible with package management? What's your alternative?

[...] it still doesn't have a decent wi-fi client or working power management on multicore systems - Progress! What a shame. [...]

I hope you do realize how complex either of the task is. Further, if you feel that apart from those, the project has/had a decent trajectory, there's always the alternative to contribute those missing features yourself.

A. if you want something done right, you gotta do it yourself

Given how terribly subjective "right" is, yes.

All in all, I agree with iansjack that you should follow up with more on your solutions.

Regards,
Shikhin

[embryo]

From that, I took away some crucial lessons: A. if you want something done right, you gotta do it yourself, B. getting tied down to architectural details is a sucker's game, and C. Unix/Linux-worship is poison, pure and simple.

How can you get any OS without being tied down to architectural details?

May be you mean there should be simple architecture? Yes, there should be. And in fact there are such things, even already developed and implemented. Have you studied such things?

But I can understand your question as a cry about need of some systematic introduction into the area of OS architecture. If it goes about some links to useful resources on the subject then I prefer to point to my approach, which is (hopefully) simple and has been done right form the ground zero. But, of course, there also are plenty of alternate OS architecture descriptions. Have you read about them?

And now I want to ask other osdevers - may be it worth to introduce an architecture section to the osdev wiki? There is the "Design Considerations" section, but it lacks a global vision and describes just some separate components instead of a unified view of a general OS architecture.

[Combuster]

And now I want to ask other osdevers - may be it worth to introduce an architecture section to the osdev wiki? There is the "Design Considerations" section, but it lacks a global vision and describes just some separate components instead of a unified view of a general OS architecture.

By all means, do to the wiki what you think is helpful. Questions can always be asked later. If you really need a committee, there's a separate (slow) forum for that.

[commodorejohn]

I read your long post, but I couldn't actually discern any ideas in it. Do you have some sort of outline of what you are planning and how it is going to solve the problems that you see in Unix/Windows, etc.?

Yep. Like I said, it was late and I needed to get to bed. I'll be blathering more as I get the time.

I would make one suggestion - choose a name that isn't already used for a well known piece of software. It's a bit like calling it "Blender".

Yeah, I'll get to that eventually. It's just a name, it can be changed.

What's wrong and so terrible with package management? What's your alternative?

What's wrong with it is that, the way I see it, if you need a program just to handle the installation of application software, your setup is too complicated. Take Linux as an example; installing an application can involve placing files in half the branches in the filesystem (more if it has to install additional libraries, which it basically always does) and editing any number of config files, and uninstallation is equally elaborate. That's just absurd.

Compare it with classic Mac OS, where an application could store all of its data in the resource fork of its executable, and all local configuration data in the specific system preferences folder. Installation could be as simple as copying the file onto the hard drive, or even just running it straight off the distribution disk, and uninstallation was just deleting the application from wherever you'd put it, and then removing the prefs file from the preferences folder. Dead simple, and zero need for anything like a package manager. (Granted, some Mac software did come with an installer, but it was mostly for cases where the user might not want every optional feature of the software, and uninstallation was still quite simple.)

How can you get any OS without being tied down to architectural details?

May be you mean there should be simple architecture? Yes, there should be. And in fact there are such things, even already developed and implemented. Have you studied such things?

Yes, I have studied such things, as was sort of implied when I mentioned I'd been looking into this for the past three years. As for not being tied down to architectural details, what I mean is that a nominally "portable" OS which requires re-compilation of software over things like compiler/binary format is not really that portable. My answer to the problem is in the "interpreted" part of that silly acronym up there - use a VM.

[iansjack]

I wonder if you, perhaps, misunderstand the purpose of a package management system. It is not merely there to install software. (You can easily do that on any Linux system without using the package management facilities.) Perhaps more importantly:

1. It provides a centralised source of software. No more searching several web sites to find software; it is all there in one central location neatly catalogued and categorised.

2. It keeps a record of what software you have installed on your computer, and facilitates its removal.

3. It serves as an easy method of ensuring that every piece of software installed on the computer is updated to the latest levels. That is particularly important in these days of security concerns.

How do you propose looking after these issues (in particular point 3) without some way of managing the packages installed on the computer - what you might term a "package management system". It is easy to criticise (not that I think your criticism is valid), but you don't provide any alternative suggestions.

IMO the ports system pioneered by BSD and adopted by some Linux distributions, and now various other Package Managers, was a great step forward in the maintenance of stable, secure operating systems. I don't think that anyone would accuse the Classic Mac OS of living up to either of those adjectives. It was such a poor OS that the Macintosh didn't come anywhere near mainstream status until they abandoned that ancient clunky mess and replaced it with a Unix-based OS.

[Brendan]

I wonder if you, perhaps, misunderstand the purpose of a package management system. It is not merely there to install software. (You can easily do that on any Linux system without using the package management facilities.) Perhaps more importantly:

1. It provides a centralised source of software. No more searching several web sites to find software; it is all there in one central location neatly catalogued and categorised.

2. It keeps a record of what software you have installed on your computer, and facilitates its removal.

3. It serves as an easy method of ensuring that every piece of software installed on the computer is updated to the latest levels. That is particularly important in these days of security concerns.

I agree - the purpose of package management is good. I suspect commodorejohn is reacting to the way it's implemented in Linux distributions, which (in my opinion) is a direct consequence of the problems caused by the "Bazaar" culture.

In general; the best way to design something may be:

• Determine all stakeholders (e.g. for a library, the stakeholders would be people who would be using the library)
• Consult with stakeholders to determine the full set of requirements
• Design a solution to suit the full set of requirements (e.g. for a library, this might be a draft specification for the library's interface).
• Enter a "Request For Comments" phase, where feedback from anyone (including stakeholders) is taken into account. The end result of this phase would be a formal specification.
• After all of the above (and not before or in parallel); begin implementing software that complies with the formal specification (e.g. start writing the library).

The "Linux way" is more like this:

• Don't do anything resembling proper design, and hack together something to suit your current project while ignoring everyone else's requirements and you own future project's requirements.
• Laugh while 10 other people use the same "lack of proper design" method to create 10 slightly different alternatives for almost exactly the same purpose
• When you realise that your initial "lack of proper design" has created something less than ideal; change things without warning anyone beforehand or telling anyone after, and without any regard to backward compatibility.

The "Linux way" is great for flexibility - it gives developers the most freedom to do anything they like any way they like with the least amount of design/consultation delay. However; when it gets down to package management and maintaining dependencies you end up with something like an explosion in a sewerage treatment facility - e.g. 20 different "graphics libraries" with 20 different versions of each library and no official "standard graphics interface"; where each version of each application depends on any of the 400 different possibilities. The end result is bloated and painful nightmare where for 10 applications you have to install 15 different "graphics libraries" that all do mostly the same thing, and everything would break every 2 months if "repository maintainers" weren't slaving away 100 hours per day in the background attempting to hide all the breakage (and sometimes succeeding).

Note: I'm not just talking about libraries here - it's all dependencies (e.g. scripting languages, compiled languages, utilities/tools, daemons, etc).


Cheers,

Brendan

[AndrewAPrice]

I agree with the OP that package management is a mess on many operating systems.

I love the idea of applications being self contained. You delete it - poof. No side effects, configuration files, user files, or others.

In Linux, I think it would be a mess cleaning up /etc, /usr/share/, /usr/lib/ etc without a package manager. On Windows I place some trust on the uninstalled cleaning up user directories and the registry.

On my operating system I'm planning for programs to simply be directories. Configuration files must stay in that directory - even if they're user-specific, they can create sub-directories. If you delete a program's directory - poof - it's uninstalled. For an application to leave its directory it needs to request explicit permission from the user.

I will have two root level directories:
/Programs
/Libraries

Programs don't have to go into /Programs (e.g. During development) but my launcher will scan /Programs (and/or other directories you configure it to). Simple and clean.

[iansjack]

I can't see either of those replies as relating to package management. They are about the fact that Linux may use different libraries for programs and stores files relating to a program in several locations. But what is that to do with the package manager, other than the fact that it makes it easier to deal with those design flaws?

I just don't understand how anyone believes that managing the packages installed on a computer is anything but a good thing.

[Brendan]

Hi,

I just don't understand how anyone believes that managing the packages installed on a computer is anything but a good thing.

I don't think anyone is saying that the idea of package management is bad in theory. I do think most people are saying it's been implemented very badly in practice.

For a simple example; why can't I install an application on a USB flash stick, then plug that USB flash stick into any computer I like whenever I feel like using that application (without reinstalling the application because it's already been installed)? An OS could be capable of handling that, and the "package manager" could still do things like update the application on the USB flash stick when a new version is released.


Cheers,

Brendan

[commodorejohn]

I wonder if you, perhaps, misunderstand the purpose of a package management system. It is not merely there to install software. (You can easily do that on any Linux system without using the package management facilities.)

You really can't, at least not reliably. Since there's no universal binary format, installing Linux software without a package manager depends entirely on how well-put-together the makefiles that build a program from source and install it are, which ranges from "actually quite nice" to "compiles on the developer's system and that's it."

Perhaps more importantly:

1. It provides a centralised source of software. No more searching several web sites to find software; it is all there in one central location neatly catalogued and categorised.

2. It keeps a record of what software you have installed on your computer, and facilitates its removal.

3. It serves as an easy method of ensuring that every piece of software installed on the computer is updated to the latest levels. That is particularly important in these days of security concerns.

How do you propose looking after these issues (in particular point 3) without some way of managing the packages installed on the computer - what you might term a "package management system". It is easy to criticise (not that I think your criticism is valid), but you don't provide any alternative suggestions.

Well, for starters, 1. is one of those things that sounds very noble in theory but in practice leads to a lot of petty politics (see the various distros that kowtow to the FSF zealots by lumping all the Stallman-incorrect stuff like firmware blobs into a "non-free" repository that the user has to specifically enable, even though it may be useful or necessary for the end user's system.) Besides which, there's nothing fundamentally unreasonable about a developer hosting their programs on their own site, and it's really not that hard to find such things - this isn't the internet of 1998 where new sites had to be manually added to web directories until the handful of search crawlers finally indexed them a month later, and a given site couldn't necessarily be counted on to be around for more than half a year without vanishing into the ether.

As for 2., that is, in my opinion, a problem that only arises from having an underlying system that's too complicated. A system where applications simply reside in one place, in as few files as possible, makes it simple and natural both to find out what programs you have installed and to delete programs you no longer need. It's not until you get into needlessly complex systems that those tasks become something you need a specific program to do.

3. is a valid point, at least (though I'm of the opinion that if software other than a web browser or other network application is even able to be a security risk, the developer is doing it wrong.) But there are other ways to work that without requiring a central repository and package manager. Plenty of Windows programs check for updates themselves from their own distribution hub and offer to install them; an OS could even provide a service specifically to do that, without needing a full-fledged package manager for it.

I don't think that anyone would accuse the Classic Mac OS of living up to either of those adjectives. It was such a poor OS that the Macintosh didn't come anywhere near mainstream status until they abandoned that ancient clunky mess and replaced it with a Unix-based OS.

No, it really wasn't. Classic Mac OS was basically the most advanced thing in the personal-computer market until the Amiga came out (and even that was mostly just as advanced in different ways,) no less stable than any of its competition (certainly no less stable than 16-bit Windows!) and hit an excellent balance of simplicity and flexibility. The reason it failed to gain major market share was because of Apple's exclusivity and pricing and generally poor market savvy, not because it was a bad product - and the reason it was replaced with a Unix-based OS was because it had a few stumbling blocks that would've required significant re-engineering to get past and Steve Jobs was eager to put his stamp back on Apple when he returned from NeXT and didn't want to pay Jean-Louis Gassée for BeOS, not because it was written in the prophecy of Unix Manifest Destiny.

The "Linux way" is more like this:

• Don't do anything resembling proper design, and hack together something to suit your current project while ignoring everyone else's requirements and you own future project's requirements.
• Laugh while 10 other people use the same "lack of proper design" method to create 10 slightly different alternatives for almost exactly the same purpose
• When you realise that your initial "lack of proper design" has created something less than ideal; change things without warning anyone beforehand or telling anyone after, and without any regard to backward compatibility.

The "Linux way" is great for flexibility - it gives developers the most freedom to do anything they like any way they like with the least amount of design/consultation delay. However; when it gets down to package management and maintaining dependencies you end up with something like an explosion in a sewerage treatment facility - e.g. 20 different "graphics libraries" with 20 different versions of each library and no official "standard graphics interface"; where each version of each application depends on any of the 400 different possibilities. The end result is bloated and painful nightmare where for 10 applications you have to install 15 different "graphics libraries" that all do mostly the same thing, and everything would break every 2 months if "repository maintainers" weren't slaving away 100 hours per day in the background attempting to hide all the breakage (and sometimes succeeding).

This.

[Owen]

I agree with the OP that package management is a mess on many operating systems.

I love the idea of applications being self contained. You delete it - poof. No side effects, configuration files, user files, or others.

In Linux, I think it would be a mess cleaning up /etc, /usr/share/, /usr/lib/ etc without a package manager. On Windows I place some trust on the uninstalled cleaning up user directories and the registry.

On my operating system I'm planning for programs to simply be directories. Configuration files must stay in that directory - even if they're user-specific, they can create sub-directories. If you delete a program's directory - poof - it's uninstalled. For an application to leave its directory it needs to request explicit permission from the user.

I will have two root level directories:
/Programs
/Libraries

Programs don't have to go into /Programs (e.g. During development) but my launcher will scan /Programs (and/or other directories you configure it to). Simple and clean.

If my config data is inside the package, how do I upgrade the package without losing it?

Why do the programs have to be inside /Programs for the OS to find them? Why can't the OS just set up file system notifications so that it detects that a file called "Firefox.app" now exists in, say, ~, and automatically make it a candidate for opening http URLs? (Note: this is exactly how OS X works!)

--

I'm quite a fan of many of the concepts behind 0install. I'm not /quite/ sure how I'd personally like to implement it. One thing I'd like is that when you download a package, the system would install it silently in the background (any actions which require user permission or administrator action would be postponed until the user attempted to launch the application). When you insert a USB stick containing applications, a similar behaviour should happen. One important point is that the list of installed packages should be a function of what you've downloaded - not some package manager internal database (any such database should be considered a cache)

[iansjack]

3. is a valid point, at least (though I'm of the opinion that if software other than a web browser or other network application is even able to be a security risk, the developer is doing it wrong.) But there are other ways to work that without requiring a central repository and package manager. Plenty of Windows programs check for updates themselves from their own distribution hub and offer to install them; an OS could even provide a service specifically to do that, without needing a full-fledged package manager for it.

The opinion that the Windows way of keeping programs updated - where each program is responsible for checking the Internet for updates and runs its own unique server process to do so - is better than a centrally maintained database is certainly interesting. But it's not something that I can agree with. I'm all for the Unix philosophy that each program does just basically just one thing and does that as well as it can. So the job of managing the packages installed on a computer is best done (IMO) by a dedicated program, and its associated database, and is certainly not something that should rely upon the user's memory or his searching through the file system.

In the same way he should not have to visit a number of developers' web sites to discover what software is available. That reminds me of the way that Foyles bookshop in London used to arrange their books (and perhaps still do) ordered by publisher rather than subject matter. Understandable as a conceit of the book trade, but hardly user friendly.

[commodorejohn]

The opinion that the Windows way of keeping programs updated - where each program is responsible for checking the Internet for updates and runs its own unique server process to do so - is better than a centrally maintained database is certainly interesting. But it's not something that I can agree with. I'm all for the Unix philosophy that each program does just basically just one thing and does that as well as it can. So the job of managing the packages installed on a computer is best done (IMO) by a dedicated program, and its associated database, and is certainly not something that should rely upon the user's memory or his searching through the file system.

I didn't specifically mean to hold up the Windows approach as the ideal - just that there's more than one way to do it, and certainly that there are ways to do it that don't require a full-fledged package manager.

In the same way he should not have to visit a number of developers' web sites to discover what software is available.

Again, I don't think this is at all unreasonable. When I want to find the best pizzeria in my area, I don't labor under the illusion that I can accomplish that by going to Wal-Mart and checking the frozen foods section - I go out, visit different pizza places, get a feel for the atmosphere and the priorities of the people making it, and of course try the pizza itself. A developer's website can tell you much, much more about a piece of software and the philosophy behind it than the little blurb in the repository; whereas the Wal-Mart approach to software (the repository) leaves the process largely in the hands of maintainers who may or may not have the same priorities as either the developer or the end user. It's like the Linux community has convinced itself that it's not really a walled garden if it's the community building the walls.