Semi-Open-Source OS Development

[CoryG]

I've been developing since I was 8 (approaching two decades, sadly) and have learned a lot about every portion of the field since then. However there are so many places that can stand to be improved that are common in all or most operating systems I would like to get a team together to work on a completely new type of operating system with the innovations seen feasible by all team members. The areas for improvement cover pretty much everything, and its just impractical and slow to implement these as bandages ie VFS's or thread synchronization apps or security protocols as modules of linux or bsd - and as I've learned its beyond me purely due to time to do them all myself.

A few of the biggest points (not all core OS level, but most are and the ones that aren't are intrinsically linked to development style) are below, any contributor's ideas would also be welcome of course - I've had a lot of stress from archaic design practices, but I'm sure I'm not the only one
________________________________________________________________

Security:
Exisitng security measures, even in the best operating systems, typically revolve around band aids on top of band aids, resulting in surface level security - attackers are expected from specific entry points and those entry points are protected against. Security implemented in a more strict fashion (better implementation than r/w/e permissions on files, inherent cryptography within files, network compatable cryptographic measures and checks, etc) would protect against any feasible attack, the perfectly secure system could be imaged, run in a sandbox, and be completely impervious to attack while running.
(this code has already been written, and is one of the points of why I believe band aid security is horrible - written at an application or kernel mod level on an existing system is too cumbersome)

Threading:
Race conditions shouldn't be possible outside of OS development, yet every large application I've worked on I find myself writing my own thread handling engine and having to work through this - it can be implemented through the use of some threading libraries, but they suffer overhead due to the deviation from the core OS thread handling calls. Granted this code is often trivial (at least by this point), but it is time consuming none the less - unless you simply give up and use an external library that is typically 99% right at absolute best and carries the aforementioned overhead.

File Systems:
Developers at an application and web level are moving more towards relational databases (which themselves have tremendous room for improvement, though I'll be releasing a POSIX database shortly that makes a huge leap in this area - and would like to port it in as the core FS). Core issues with existing filesystems that often hinders development of more advanced systems is concurrency and corruptibility - there is no reason for this if you treat the root filesystem as a database itself - all half decent databases essentially replicate FS functionality anyway simply to function in a multiuser state.

Networking:
Feasibly this is the only component that has to conform to existing standards in any degree, and in many ways is implemented through libraries anyway past a driver level - I think the concept of plug and play can be implemented better, but being a protocol level attribute of the database, this is something that would be implemented between the OS and app levels of the system.

Administration:
Computer security works well with Mac for instance, acting as a CA to digitally sign drivers after they are tested, and should they ever get the implementations that secure the CA certs right, it may work for Win32/64 systems well. However, system administrators need methods to secure and create exceptions within their networks, and the distribution of applications can be done without the significant overhead of addon applications (this is inherent in the security implementation described earlier). When a user sits down at a fresh machine on a workplace network it often requires a significant investment of time to get the machine's role properly adjusted, be it through network downloads or direct installation - the network however offers the flexibility for better systems (such as local CA customized roles that can be applied to a machine on the fly).

Rapid Application Development
.net does this very well, its about the only respectable thing left about Win32/64 systems, and the early versions of Vista proved it doesn't work when you attempt it on an OS level as a library, but mutable applications have significant possibilities - more of this can be described later if your interested (in the database I am preparing to release there are mutable calculation binaries that can be implemented in much the same way in a more thorough sense - though not having developed this as a code manipulation and compiler tool my opinions of it are just that, and are likely not worked out to such an extent as the previous topics).
________________________________________________________________

All these concepts are open to further expansion and customization by contributors of the project and I would like to have the OS licensed in a way where only contributors have access to source, with the rights to use the last version they made contributions to for personal use, with the final OS being sold as a proprietary system with all contributors getting a share of profits equal to their contributions - the system for this is also open to discussion. I am willing to host the project site and development portals, secure source mirrors would be welcome from contributors as well.

If your interested in this project please feel free to contact me directly - I'll check this thread for commentary as well of course. My personal site is http://cory.geesaman.com/ and the company I will be releasing the DB under is at http://www.novorum.com/

[rachel]

What all would be included in your vision that would be of any interest to the general public? Not that I look down on something made with developers in mind but developers have little use for coding on an OS that no one uses.

[CoryG]

What all would be included in your vision that would be of any interest to the general public? Not that I look down on something made with developers in mind but developers have little use for coding on an OS that no one uses.

Look at MS as an example, if you make development faster than on any other platform developers will use it for prototyping, if it is also usable it will be used as a result. If it secure and supports features you typically pay thousands for as addon application suites natively it will be worth using for businesses, and if you can throw together code quickly for random ideas it will be useful for personal use. Of course performance also comes into play, but that's easy enough if we control the primary compiler for the system. As an aside, coding on an OS no one uses seems especially suited for the type of people viewing this thread, or so I thought

[rachel]

" Security implemented in a more strict fashion (better implementation than r/w/e permissions on files"

What are your methods for achieving that? I've always been content with the security that permissions separations provides, so anything superior to it would be of interest to me.

[Combuster]

No good design without a proper defense. Lets see if you can handle some criticism:

Security
Use languages with array bound checking and strict pointer management to fix one major cause of security leaks. (i.e. Java), ACLs already exist, as does the whole load of os based security plugins which only help when that kind of errors exist. Most other security issues are still caused by bad configuration (and thus, human error on the sysop side)
Not worth writing an OS for #1

Threading
You have to choose between portability (and use a lib) or not (and use native calls). Making a new OS doesn't help
Not worth writing an OS for #2

File Systems
If you need a relational database, hook up PostgreSQL and have it use the filesystem and the necessary system calls to force flushing to disk when that's deemed necessary.
Not worth writing an OS for #3

Networking
Your point here?

Administration
Remote desktop, secure shell, setup tools that are controlled by some form of authority to guarantee consistency. hmm, linux with package manager?
Not worth writing an OS for #4

Rapid Application Development
Eclipse with VE plugin, Java studio, or as mentioned, .net... If you want a RAD tool, make that instead
Not worth writing an OS for #5



Conclusion:
Go make a linux distro instead.

[rachel]

I think the question of what makes an OS worth re-recreating is very apt. I can't say I agree or disagree about the worthiness of the specific points mentioned individually, but I definitely see the need for a larger, more innovative draw than just the minor tweaks suggested.

It would take, imo, something that is going to make one lift their eyebrows upon suggestion in order to be worth doing. Could it still run everything I currently use without the need to swap between platforms or use a vm? Does it waste less resources and run faster? Can I modify every aspect of the interface easily and safely? Can I make it run on my toaster like linux?
-things of that nature.

Otherwise perhaps what you have in mind is better suited to work alongside current operating systems and perhaps one that bridges the gaps between platforms via an independent and cross compatible db format would be preferable to a completely new os design. In fact, such a thing strikes me as pretty eyebrow-raising itself.

[CoryG]

Security
Use languages with array bound checking and strict pointer management. (i.e. Java)
Not worth writing an OS for #1

Security is something that isn't even considered to a decent degree in modern operating systems, if you have direct access to a machine you can plug in your own kernel, overwrite core libraries, outright read the data, and do whatever else (even within more secure systems such as BSD). I would personally like to take on the security side of the OS because it was my first route of attack when trying to make my own from scratch a few years ago, and spent ~18 months writing the libraries required to make a truely secured OS (all-be-it as a bsd kernel module and thereby not 100% secure to a decent hacker).

Threading
You have to choose between portability (and use a lib) or not (and use native calls). Making a new OS doesn't help
Not worth writing an OS for #2

This is another portion of an OS I would love to tackle, I've written at least a dozen fairly bulky applications that required entirely different custom threading solutions from one another, and believe I can make a generic implementation that would equate to a few minutes implementation for each end application designed for the system. A key point in my desire to make an OS is faster RAD - I spend all my time coding, and as a result hate the more mundane portions of code.

File Systems
If you need a relational database, hook up PostgreSQL and have it use the filesystem and the necessary system calls to force flushing to disk when that's deemed necessary.
Not worth writing an OS for #3

Its not the just the need for a DB that makes replacing the filesystem with one worthwhile. You can perform faster searches based on meta information, have a more dynamic filesystem, and support multi-user or multiple thread access in a much more friendly way if you design the filesystem with those points in mind. Aside from the ability to queue your data R/W operations in the most efficient manner system-wide. If you have to install a database its only because the operating system lacks those key functionality requirements and a very small ANN thrown into the system itself can sync everything running in a much more efficient way (I've used this concept in a database I am currently finishing off and would like to integrate this into the operating system as well - with some modifications for security and table divisioning it would make for a much more powerful system for users and developers).

Networking
Your point here?

General rambling because I got going on a bit of a spree citing poor usability points that would be fun to correct

Administration
Remote desktop, secure shell, setup tools that are controlled by some form of authority to guarantee consistency. hmm, linux with package manager?
Not worth writing an OS for #4

For end user applications sure, but these are all aspects of the same issue, band aids to patch problems because the system itself is deficient. A properly constructed operating system wouldn't need SSH or remote desktop and certainly wouldn't have independent setup tools outside of personal computing. I'm not saying this isn't quite a change so I understand if it's misunderstood, but it goes back to security. Security should never be compromised, remote desktop is typically operated over an insecure channel, secure shell is simply not friendly to an end user, but even if you make remote desktop secure (as there are secure solutions) the fact an insecure solution is possible negates the security of the system itself.

Rapid Application Development
Eclipse with VE plugin, Java studio, or as mentioned, .net... If you want a RAD tool, make that instead
Not worth writing an OS for #5

RAD isn't the end goal, its just something that comes naturally with everything else implemented to the fullest extent. Though the issue with Java and .net is the inherent speed drop, this can likely be avoided by making the operating system designed for it. The only real perk of Java over .net is the portability, I'm not out for a RAD tool, but a RAD tool that is fast. If you base this a bit more around template modification code where the templates are maintained within the OS you can have applications that are built rapidly and are kept secure and optimized over time as the operating system is upgraded.

Conclusion:
Go make a linux distro instead.

Personally I would choose BSD over Linux, for two main reasons. BSD is far more secure without significant hacking, and there are so many flavors of Linux that a new distro would be lost. There's plenty to be said for POSIX compliance for the sake of having existing applications included, but that itself would go against the philosophy of this operating system design because you couldn't have the operating system keep the various code branches up to date in their present form. I think POSIX compliance would be a definite plus for driver support, at least in the initial version - but you need the ideological change in order to avoid applications built on archaic design principles.

[Troy Martin]

When I saw this post I thought "oh my god not a hobbyist version of M$ shared source...."

Now I have nothing to say. Combuster basically summed it all up.

[offtopic]Oogh, post-caffiene crash "hangover"...[/offtopic]

[JohnnyTheDon]

Security is something that isn't even considered to a decent degree in modern operating systems, if you have direct access to a machine you can plug in your own kernel, overwrite core libraries, outright read the data, and do whatever else (even within more secure systems such as BSD). I would personally like to take on the security side of the OS because it was my first route of attack when trying to make my own from scratch a few years ago, and spent ~18 months writing the libraries required to make a truely secured OS (all-be-it as a bsd kernel module and thereby not 100% secure to a decent hacker).

If you have direct access to a machine, you're done. Nothing you do in software can prevent tampering with hardware, like booting from a usb key or removing the hard drive and writing a virus on it. The only things that would help you (somewhat) in this case are hardware addons like TPM, which IMHO aren't worth it because they limit the software you can run on your system.

This is another portion of an OS I would love to tackle, I've written at least a dozen fairly bulky applications that required entirely different custom threading solutions from one another, and believe I can make a generic implementation that would equate to a few minutes implementation for each end application designed for the system. A key point in my desire to make an OS is faster RAD - I spend all my time coding, and as a result hate the more mundane portions of code.

Like combuster said, libraries will work fine. You can make a generic implementation in a library that wraps the native implementation.

For end user applications sure, but these are all aspects of the same issue, band aids to patch problems because the system itself is deficient. A properly constructed operating system wouldn't need SSH or remote desktop and certainly wouldn't have independent setup tools outside of personal computing. I'm not saying this isn't quite a change so I understand if it's misunderstood, but it goes back to security. Security should never be compromised, remote desktop is typically operated over an insecure channel, secure shell is simply not friendly to an end user, but even if you make remote desktop secure (as there are secure solutions) the fact an insecure solution is possible negates the security of the system itself.

So how would a properly constructed os make ssh and remote desktop obselete? If you mean inserting similar tools into the OS, you're not really changing anything. There are open source programs for both ssh and remote desktop that one could simply package with an OS. And you can run remote desktop, just like any other network application, with encryption even if it doesn't inherently support it(ssh tunnels, anyone?).

RAD isn't the end goal, its just something that comes naturally with everything else implemented to the fullest extent. Though the issue with Java and .net is the inherent speed drop, this can likely be avoided by making the operating system designed for it. The only real perk of Java over .net is the portability, I'm not out for a RAD tool, but a RAD tool that is fast. If you base this a bit more around template modification code where the templates are maintained within the OS you can have applications that are built rapidly and are kept secure and optimized over time as the operating system is upgraded.

The inherent speed drop isn't that big of a deal in most cases, especially if you use good programming practice (ie. foreach insted of for). What speed drop there is cannot be avoided by changing the operating system, since the only way you can safely remove bounds and type checks is to run the application inside hardware protection (which is already done by every OS).

[ru2aqare]

Security is something that isn't even considered to a decent degree in modern operating systems, if you have direct access to a machine you can plug in your own kernel, overwrite core libraries, outright read the data, and do whatever else (even within more secure systems such as BSD). I would personally like to take on the security side of the OS because it was my first route of attack when trying to make my own from scratch a few years ago, and spent ~18 months writing the libraries required to make a truely secured OS (all-be-it as a bsd kernel module and thereby not 100% secure to a decent hacker).

I think that is because the machine design is inherently insecure (in the sense of chain-of-trust). If you have access to the BIOS and/or RAM, you can hijack the boot process and do whatever you want with the machine. You can add security to your OS, but when your OS can be patched when it boots, you can't do much about it... This insecure design can't easily be fixed, and the alternatives are the "trusted computing" (I may not be correct here) and an Xbox360 or game console like secure design.

Edit: JohnnyTheDon beat me to it.

[CoryG]

If you have direct access to a machine, you're done. Nothing you do in software can prevent tampering with hardware, like booting from a usb key or removing the hard drive and writing a virus on it. The only things that would help you (somewhat) in this case are hardware addons like TPM, which IMHO aren't worth it because they limit the software you can run on your system.

Not true, if the cryptography is properly implemented within the kernel the user would require a password to successfully write a virus to the hard drive. You could feasibly alter the loader if you wrote the kernel in two stages (past the boot loader), though if you ensure the critical blocks loaded by the second stage kernel immediately follow it the likeliness of someone successfully creating a replacement first stage kernel are pretty slim (since either way you would at least need the password to alter any data or the second stage kernel).

Like combuster said, libraries will work fine. You can make a generic implementation in a library that wraps the native implementation.

The threading could be done as a library, but then you don't gain the speed enhancements of a low level ANN that ties into all the basic processes in the system.

So how would a properly constructed os make ssh and remote desktop obselete? If you mean inserting similar tools into the OS, you're not really changing anything. There are open source programs for both ssh and remote desktop that one could simply package with an OS. And you can run remote desktop, just like any other network application, with encryption even if it doesn't inherently support it(ssh tunnels, anyone?).

While "properly constructed" is entirely objective in this case, remote desktop and ssh wouldn't be needed from other machines running the same OS if you build the remote management into applications so that app developers only have to surface what features can be remotely modified. This comes in mostly within the RAD portion of the previous descriptions. Ideally, to remain secure after writing the core security features, no network or device interface could be accessible without passing through the encryption layer if it surfaced any system resources or potential executables (with a clear distinction of executable and data blocks within the file system itself - with transfer into the executable layer only possible with the appropriate network or local authentication present.

The inherent speed drop isn't that big of a deal in most cases, especially if you use good programming practice (ie. foreach insted of for). What speed drop there is cannot be avoided by changing the operating system, since the only way you can safely remove bounds and type checks is to run the application inside hardware protection (which is already done by every OS).

I agree, its not a big deal in most cases, I only use .net for prototyping or when real number crunching isn't a factor - though when it is I always go with a POSIX based daemon - because Windows alone is too much overhead for anything serious. Ideally the operating system would have the ability to go into a GUI or console mode based on use, with the ability to switch between the two relatively seemlessly (xorg is typically a pain to do such a thing with, at least in my experience - so I've always run either Windows or BSD (with xorg only going for the POSIX dev machines). Though OS controlled code at every level does open the door for more advanced file system buffering, network management, thread utilization etc - whether your writing a more serious app or just RAD GUI stuff/prototyping.

[CoryG]

I think that is because the machine design is inherently insecure (in the sense of chain-of-trust). If you have access to the BIOS and/or RAM, you can hijack the boot process and do whatever you want with the machine. You can add security to your OS, but when your OS can be patched when it boots, you can't do much about it... This insecure design can't easily be fixed, and the alternatives are the "trusted computing" (I may not be correct here) and an Xbox360 or game console like secure design.

Edit: JohnnyTheDon beat me to it.

A work around for a kernel highjacking attack would be in the 2 stage kernel (3 if you count the boot loader):
bootloader->kernel stage 1(requires password to decrypt stage 2 and run)->kernel stage 2(verifies signature of stage 1, dies if verification fails) if the stage 2 kernel and all data are encrypted it would then be secure for a single user. for a multi-user environment an added stage would be required to handle the password management correctly before the second stage kernel.

[ru2aqare]

A work around for a kernel highjacking attack would be in the 2 stage kernel (3 if you count the boot loader):
bootloader->kernel stage 1(requires password to decrypt stage 2 and run)->kernel stage 2(verifies signature of stage 1, dies if verification fails) if the stage 2 kernel and all data are encrypted it would then be secure for a single user. for a multi-user environment an added stage would be required to handle the password management correctly before the second stage kernel.

BIOS/firmware/... patches bootloader to call back when it finishes loading stage1. Then it patches stage1 to call back when it has decrypted stage2. Then it patches the signature verification function in stage2, and executes stage2, installs backdoor, whatever. Of course, the BIOS/firmware/... needs to know what bytes to patch over, but nevertheless it can be done.

Also you could attack from the "input password" part - run the entire OS in a virtual machine, trapping all I/O access to the keyboard/smartcard/etc and voilá, you have the password.

[JohnnyTheDon]

Not true, if the cryptography is properly implemented within the kernel the user would require a password to successfully write a virus to the hard drive. You could feasibly alter the loader if you wrote the kernel in two stages (past the boot loader), though if you ensure the critical blocks loaded by the second stage kernel immediately follow it the likeliness of someone successfully creating a replacement first stage kernel are pretty slim (since either way you would at least need the password to alter any data or the second stage kernel).

That sounds like Windows' BitLocker and encrypted filesystems. And it has a lot of vulnerabilities, such as inserting malicous code in the decryptor (which can't be encrypted) or pulling the decryption key from memory during reboot. In any case its already been implemented on multiple operating systems.

While "properly constructed" is entirely objective in this case, remote desktop and ssh wouldn't be needed from other machines running the same OS if you build the remote management into applications so that app developers only have to surface what features can be remotely modified. This comes in mostly within the RAD portion of the previous descriptions. Ideally, to remain secure after writing the core security features, no network or device interface could be accessible without passing through the encryption layer if it surfaced any system resources or potential executables (with a clear distinction of executable and data blocks within the file system itself - with transfer into the executable layer only possible with the appropriate network or local authentication present.

But in some cases, like if you're on the go and you want to access your desktop at home, this won't be enough. It also means that programmers have to implement remote access in their application themselves if they want it to work. This is the exact opposite of your idea of spending less time coding repetivive things. Its much easier to just allow encrypted remote desktop, which will work just fine.

It seems that most of the things you are suggesting are a) already implemented or b) not implemented because they're not particulary useful.

[CoryG]

BIOS/firmware/... patches bootloader to call back when it finishes loading stage1. Then it patches stage1 to call back when it has decrypted stage2. Then it patches the signature verification function in stage2, and executes stage2, installs backdoor, whatever. Of course, the BIOS/firmware/... needs to know what bytes to patch over, but nevertheless it can be done.

Also you could attack from the "input password" part - run the entire OS in a virtual machine, trapping all I/O access to the keyboard/smartcard/etc and voilá, you have the password.

I've got an idea of how to handle this, and will make a single user open source boot loader patch to cover it shortly - I'll post a link to code on these forums once completed so that it can be reviewed by those interested.