Scripting language loading kernel/OS

[sparkymat]

[My system programming is a bit rusty and some (or all) of the things below may seem stupid.]

I am considering developing an OS (microkernel-based, though that part comes in a bit later), who's loader (ELF loader) would have a preprocessor attached, which would accept as input, a high-level language file to load (for example, Python or Ruby) and match it against the corresponding binary (or compile it to a binary if not already preset or outdated) and load the ELF from that binary. Possibly, the kernel will maintain an ELF repository, with a hash table of code files to binary files mapping.

Coupling this with a versioned file-system, I can think of various useful (as well as not-so-useful) features:
- Updates/upgrades would be a set of .diff files as the various processes (both system and user) would be essentially script files which are compiled on the fly the first time they are run
- Versioned file system means your entire update/upgrade history is revisioned
- OS development in a (very) high level language

I do understand that the major concern would be the performance bottlenecks of such a system. I am not saying that this is feasible (in the short run) as a desktop-replacement OS. It would be nice to have a toy OS which one can build/extend/use, which is mostly written (and maintained) in a high level language.

Does such a system seem feasible and practical?

All comments and thoughts welcome.

[NickJohnson]

Having a full Python and Perl compiler in your kernel would make it *extremely* massive. Compilers aren't that great for either language (or other very high level languages) already, because they are meant to be interpreted. I doubt you're going to best all of the compiler writers in the world on your first try. And I don't quite see the advantage to doing this. You would need a lot of space to keep the entire userland in the kernel heap, and dynamically compiling everything, not to mention within the kernel itself, would be complex and unsecure.

There are no problems using high level languages all the way down within userspace, but the interpreters and kernel really should be written in C/C++ anyway, and kept separate. The other issue is that very high level languages are just not intended for low level systems programming, so it would be tricky (though not impossible, as many have proven) to adapt them. But once you unbind the system and the language, it's a userland decision, so you can just go and write a usual kind of kernel.

Edit:
However, having a version controlled filesystem is a lot more practical, and you would get all of the benefits you listed. Not many people have done such a thing, so it would be a somewhat pioneering concept, and I don't know what the tangible advantages would really be. I've considered building a filesystem that is based on git, but I'm still far from implementation of my OS' filesystem either way. What you shouldn't do is trap yourself having only version controlled filesystems, in case the concept isn't good. Keep things flexible, and you'll have room to experiment.

[sparkymat]

I was considering the same approach; of having the kernel, script compiler and VM in C/C++, with the rest of the userland in Ruby/Python. This, coupled with a versioned filesystem, would satisfy what I am looking to achieve. Additionally, the versioning filesystem needn't be mandatory, providing rollbacks and revision histories IF available. The primary focus would be to still have userland in Ruby/Python.

[linuxfood]

True not many people have done the versioned filesystem but... I believe VAXes had a versioned filesystem, so it's not really "pioneering". The question is, can you do it right?

My notion is that the Reason not many versioned filesystems exist is because the "user interface" semantics are hard to get right.
Consider a fairly standard case:
1. user creates a file
2. user edits file
3. user edits file again
4. user "rm"'s the file. <-- what do you do here? revert to the previous version? IIRC that's what VAXes did.
That could potentially be rather confusing.

[sparkymat]

I would have assumed that an 'rm <filename>' would remove the file (apparently) from the filesystem, but still be available from the revision history of that folder. Obviously, a lot more thought is needed in this direction though - storage overheads, performance overheads, etc.

Btw, regarding the OS, I came across this -> http://code.google.com/p/cleese/ . It seems there's an effort to get the Python VM (bytecode interpreter) to sit directly on metal. This achieves something similar to what I am looking for, except that its in Python (whereas, I am looking for a Ruby OS ). So, I would assume what I could try is to port the Ruby VM onto metal (though how much effort is required for this is something I have yet to estimate).

I just started experimenting and so far have a "kernel" that boots and prints Ruby Operating System. So, I guess there's a long way to go.

[linuxfood]

Diskspace is cheap. If someone is going to run your operating system because of the versioning filesystem, you can assume that they've got plenty of disk (not a reason to be sloppy, but should not be a primary concern).
Also, there are filesystems out there that do snapshots (user controlled versioning), transparent compression, encryption, checksumming with real world usable performance (ZFS is an example, btrfs is an up-and-coming).

[quanganht]

Give GRUB2 a try.

[sparkymat]

I'm sorry if its obvious, but I didn't quite get why you mentioned GRUB2. I am using GRUB (1) to load the "kernel". Is that what you are referring to as well?

[pcmattman]

Give GRUB2 a try.

Did you actually read the thread?

It would be nice to have a toy OS which one can build/extend/use, which is mostly written (and maintained) in a high level language.

I personally like the idea of having a basic C kernel which provides just enough functionality to run the Python interpreter, and have your shell and (most) applications in Python. It *should* be possible to have all of userspace running in such a scripting language. It sounds like you are building a little bit on that idea.

If I understand correctly, it would appear you intend to have a loader which loads Python/Ruby/<your language here> scripts, compiles them to ELF, and then runs them. I don't know if the performance benefit from compilation would be any greater than that of directly interpreting the scripts, unless the compilation was done on a high-use program. For instance, you'd probably want to compile something like "ls", but something like "du" probably wouldn't need to be compiled. Note I use these as examples. You could have "listfiles" and "diskusage" instead

So what you might do is have a specific line in each script (perhaps similar to the shebang) which specifies whether the script is to be compiled or not. This would allow the writer to have the freedom to choose based on their own prediction of the usage of their software whether to be compiled at runtime or not. Whatever happens, I definitely agree with the "ELF Repository" idea - you're going to want to (persistently) map paths to the compiled binaries or else there's no benefit at all in compilation.

I may have totally misunderstood your idea, but if I haven't I hope I've given at least a little food for thought!

[sparkymat]

Did you actually read the thread?

Well, I did. I guess I need to read up on GRUB2, and what it does more than GRUB.

I personally like the idea of having a basic C kernel which provides just enough functionality to run the Python interpreter, and have your shell and (most) applications in Python. It *should* be possible to have all of userspace running in such a scripting language. It sounds like you are building a little bit on that idea.

This does sound like the sensible thing to do. My personal choice is the Minix kernel, though I'd like to hear the pros and cons as opposed to, say, the FreeBSD or the Linux kernel.

If I understand correctly, it would appear you intend to have a loader which loads Python/Ruby/<your language here> scripts, compiles them to ELF, and then runs them. I don't know if the performance benefit from compilation would be any greater than that of directly interpreting the scripts, unless the compilation was done on a high-use program. For instance, you'd probably want to compile something like "ls", but something like "du" probably wouldn't need to be compiled. Note I use these as examples. You could have "listfiles" and "diskusage" instead . So what you might do is have a specific line in each script (perhaps similar to the shebang) which specifies whether the script is to be compiled or not. This would allow the writer to have the freedom to choose based on their own prediction of the usage of their software whether to be compiled at runtime or not. Whatever happens, I definitely agree with the "ELF Repository" idea - you're going to want to (persistently) map paths to the compiled binaries or else there's no benefit at all in compilation.
I may have totally misunderstood your idea, but if I haven't I hope I've given at least a little food for thought!

Yeah, lotsa food for thought. I guess I need to spend some time thinking over all these.

[pcmattman]

Well, I did. I guess I need to read up on GRUB2, and what it does more than GRUB.

My apologies, I was responding to quanganht's post, not yours. I should've made that clearer.

[sparkymat]

My apologies, I was responding to quanganht's post, not yours. I should've made that clearer.

No issues. I was wondering if I missed something.

[earlz]

I personally like the idea of a versioned filesystem...

I think it could even be done over a regular filesystem with a little magic. Like have your system calls and such for files write files as myfilename:r1 or whatever. Next modification it is written as myfilename:r2 etc etc..
though, that would have definite performance penalties, but it could render you with a filesystem(involving links and such) that is readable from other systems not aware of versioning.

[AndrewAPrice]

If it's a micro kernel, don't stick the Python/Ruby compiler in the kernel. Stick it in your application loader, and just have the bare minimum (the loader/compiler) already in as binary. Then your loader can be as complex as you want (multiple languages) without affecting the kernel size. Or better yet, the compiler can be a separate program, and when your shell goes to execute a program it can tell it's python source and send it to the python compiler, which pipes the resulting binary to your loader.

[sparkymat]

Perhaps this can be achieved as a coreboot payload?