OSDev.org

Rather than hijack someone else's discussion, I decided to post my OS ideas here, and link to it. I don't feel like I'm far enough along for an official announcement quite yet, but I wanted to get some feedback from you guys. (Constructive feedback only, please.)

I've spent a lot of time working on my first OS, and I've put a lot of thought into OS support for Classes and Objects, as they were at the top of my list of OS "features" I wanted to support. However, because there were so many disadvantages to this approach, I decided that perhaps the best approach would be to make them optional. My current approach is to implement this functionality in "layers", to allow the developer (currently, me) to choose what level of control/support he (I) would like. For instance:

Layer 0 : Primitive Data (byte, short, integer, long, string(aka byte[]), label, reference (aka address of label), etc.)
Layer 1 : CPU / Platform Byte Code (MOV, JMP, CMP, BIOS constants (aka int 10h), etc.)
Layer 2 : Platform Functions (DrawString, ReadBlock, Shutdown, Reboot, etc.)
Layer 3 : Platform Classes (Audio, Video, System, String, Integer, etc.)
Layer 4 : Application Workflow / Cross-Platform Method Calls

The plan is that all of these layers use the same basic language (currently, XML, XSD, and XSLT), and you can easily write most of your code in, say Layer 3, but throw in a few lines of Layer 1 code if you want to control exactly what the processor does (like asm blocks in C). In fact, Layer 3 code essentially breaks down into Layer 2 code, which is then broken down into Layer 1, which is broken down into Layer 0, which is simply written out as a byte stream to a file or device. Layer 4 code, however, will likely be completely separate, since it will only allow method calls, and is essentially a workflow graph. It will probably either be interpreted, or broken down into Level 3 code at run-time (JIT) or install time (NGEN).

I'm still making some design decisions, but here are some ideas that I'm trying to include by the time I get to Layer 4:

Static Classes - Essentially, interface definitions
Dynamic Objects - Similar to JavaScript, all members can be added/removed.
Subtypes - Similar to Templates/Generics, all objects can have types and subtypes. For instance, Enumerable of Strings.
Enumerable Results - Similar to LINQ, all methods return collections, even if they are empty, or only contain one object. (No null reference exceptions)
Set Operations - Similar to JQuery, Enumerables inherit methods from their subtypes, and method calls are called for each item in the collection.
Eliminate Language Keywords - Essentially, you can only call static or instance methods.
OS Managed Objects / Memory - Combined memory manager and object garbage collection.

I reached Level 3 code on my 32-bit version of my OS, but I decided to start a new 64-bit version, and take the opportunity to rethink some things that I wasn't happy with, like the file system and the memory manager. I'll post some screen shots at some point to show how it all works.

I started this project because I couldn't find anything like it out there, but I fully expect someone else to beat me to it, since I'm only working on it in my spare time, and I have very little of that these days.

Still, let me know what you guys think, and if anyone has any interest in seeing this concept further developed.

SpyderTL wrote:I started this project because I couldn't find anything like it out there, but I fully expect someone else to beat me to it, since I'm only working on it in my spare time, and I have very little of that these days.

Yeah, you were beaten to it. See the Smalltalk-80 info (and demo video) here: http://forum.osdev.org/viewtopic.php?f= ... 93&start=0

In your defence, you're not alone. Not a lot of new ideas in this game so I suppose we're probably all "beaten to it" with all of our great ideas.

If Smalltalk were an OS, then yes, it's very similar. Actually, while reading about Smalltalk, I found this OS that looks pretty similar, although it doesn't look like it's been updated in a few years...

http://squeaknos.blogspot.com/

My description drifts a bit between the OS and the programming "language" that I'm using to develop it. They should probably be treated separately.

The OS (at least, the 32-bit version that is the most complete) has a command line interface shell similar to jEmbryOS (which is one of the projects that will probably be done way before I am). I actually modeled it after the Immediate Window in Visual Studio. You can see a list of all of the static classes, and the static methods within those classes, and execute them from the command line. Any object returned from the method is converted to a string (.ToString), and displayed on the screen. You can chain these together to drill down into the members:
Unfortunately, this shell currently makes up the majority of the OS, along will all of the included classes that it can access. In the next (64-bit) version, a lot of this will be broken up into separate classes.

The language, on the other hand, is essentially a bunch of XSD/XSLT pairs which "break down" an XML file into smaller (lower level) pieces until it is nothing but "Level 0" elements (byte, short, int, string, label, etc). The XSD defines the available elements that you can use in your XML file (similar to a .h file), and the XSLT transforms them into lower level XML files (similar to inline functions in a .c file). (This design was actually not my idea -- it was suggested to me by a co-worker during a discussion about how code is just data. It is something else that I looked for, but could not find, so I wrote my own.)

Using this approach, I've been able to "compile" several x86 OS images (16-bit, 32-bit, and now a 64-bit boot loader, floppy disk image, CD ISO, Hard Disk, Flash, Pxe image), a Commodore 64 cartridge image, an NES cartridge image, SNES, N64, GameBoy, GBA, ATARI cartridge image, a 16-bit Windows .EXE, a 16-bit DOS .COM program, a Java .class file (almost), and a Raspberry Pi boot image that turns on an LED. I've got platform XSD/XSLT files for Intel, ARM, MOS, JRE, CLI, MIPS, Motorola, and Z80 processors, and a bunch of PCI devices, video controllers, USB controllers, etc.

This site, and the people on it have been invaluable over the past few years. Thanks to all of the Wiki contributors and forum users for all of the help.

SpyderTL wrote:Layer 0 : Primitive Data (byte, short, integer, long, string(aka byte[]), label, reference (aka address of label), etc.)
Layer 1 : CPU / Platform Byte Code (MOV, JMP, CMP, BIOS constants (aka int 10h), etc.)
Layer 2 : Platform Functions (DrawString, ReadBlock, Shutdown, Reboot, etc.)
Layer 3 : Platform Classes (Audio, Video, System, String, Integer, etc.)
Layer 4 : Application Workflow / Cross-Platform Method Calls

How do you implement a transition from level 0 to the level 1? There should be some language constructs, that can describe the transition. But as you have pointed, you use XSLT as a language environment, so it is unclear for me, how exactly it is possible to create some XSLT code and translate it to the level 1. At least it seems there should be a lot of templates for every assembly instruction. And does XSLT have capability of writing raw bytes? My XSLT knowledge is not such deep to write an OS in it

SpyderTL wrote:here are some ideas that I'm trying to include by the time I get to Layer 4:

Static Classes - Essentially, interface definitions
Dynamic Objects - Similar to JavaScript, all members can be added/removed.

The Class notion is a bit alienated in XSLT, doesn't it? At least the common practice is not to think about templates as about classes. So, the "object oriented" system in XSLT looks very strange to me. But may be it is because of lack of XSLT knowledge.

SpyderTL wrote:Still, let me know what you guys think, and if anyone has any interest in seeing this concept further developed.

As I understand, any concept should be developed if there is something promising in it. May be it is because I am not an XSLT fan, but I can't see the promise yet. The kind of a promise can be some very interesting generalisation, like everything is a primitive data and all the other is produced from the simple types. Or it can be some useful area of application for a new system with many users or developers. Or it can be something else. But combination of XSLT and objects with an OS as a result is something not very trivial to understand.

SpyderTL wrote:The OS (at least, the 32-bit version that is the most complete) has a command line interface shell similar to jEmbryOS

Nice to see some interest here

SpyderTL wrote:Using this approach, I've been able to "compile" several x86 OS images (16-bit, 32-bit, and now a 64-bit boot loader, floppy disk image, CD ISO, Hard Disk, Flash, Pxe image), a Commodore 64 cartridge image, an NES cartridge image, SNES, N64, GameBoy, GBA, ATARI cartridge image, a 16-bit Windows .EXE, a 16-bit DOS .COM program, a Java .class file (almost), and a Raspberry Pi boot image that turns on an LED. I've got platform XSD/XSLT files for Intel, ARM, MOS, JRE, CLI, MIPS, Motorola, and Z80 processors, and a bunch of PCI devices, video controllers, USB controllers, etc.

Wow! Do you really have all the names above in your hand? How it is possible to have so many standards covered and still not to have functionally very rich OS? Again, I haven't understand something really important.

SpyderTL wrote:If Smalltalk were an OS, then yes, it's very similar. Actually, while reading about Smalltalk, I found this OS that looks pretty similar, although it doesn't look like it's been updated in a few years...

http://squeaknos.blogspot.com/

If you look at the Blue Book ( stephane.ducasse.free.fr/FreeBooks/BlueBook/Bluebook.pdf‎ ) you'll see that Smalltalk-80 is an OS. It's got a process scheduler with semaphores, hardware interrupt handling, device drivers, a timer, GUI, etc. They're all written in Smalltalk. The Xerox machines were microcoded to execute the bytecodes and maybe do some BitBLT (IIRC). It seems it can all be done in ~6000 lines of code. Later implementations turned Smalltalk-80 into an application.

bwat wrote:Smalltalk-80 is an OS. It's got a process scheduler with semaphores, hardware interrupt handling, device drivers, a timer, GUI, etc. They're all written in Smalltalk. The Xerox machines were microcoded to execute the bytecodes and maybe do some BitBLT (IIRC).

It seems such approach scales very badly. I mean there should be some translation layer into target system instruction set, but not the bytecode execution environment. The bytecode execution environment prevents the system from being run by different hardware, this is very limiting factor.

embryo wrote:

bwat wrote:Smalltalk-80 is an OS. It's got a process scheduler with semaphores, hardware interrupt handling, device drivers, a timer, GUI, etc. They're all written in Smalltalk. The Xerox machines were microcoded to execute the bytecodes and maybe do some BitBLT (IIRC).

It seems such approach scales very badly. I mean there should be some translation layer into target system instruction set, but not the bytecode execution environment. The bytecode execution environment prevents the system from being run by different hardware, this is very limiting factor.

The microcode executes the bytecodes, they are machine instructions. You build the language, then the hardware.

bwat wrote:The microcode executes the bytecodes, they are machine instructions. You build the language, then the hardware.

It's great, but such high level design leads to the nowhere. So, there is no such hardware any more. May be in the 80-s the design was relatively good, but as for now - there should be compiler first, and the hardware will be second, while language is irrelevant.

But again about XSLT.

I had some thoughts and now I see the picture a bit better. As I understand there is a tightly related set of means, including XSLT, XSD and XML. XSD defines data structures, XSLT defines operations and XML is the data carrier. But it is not classes still, because class is supposed to have data and methods in one place. However, the combination of XSLT and XSD provides us with some close analogy to the OOP class. Now I can see how an object graph can be implemented and processed with such an "almost OOP" system.

But it still seems impossible to me to organize all the XML files in some manageable project with the OS as an output. At least it will be very great bunch of XML files and there should be some means to manage all of them. Also, the XML (and XSLT in particular) is not as compact as C or Java or many other languages - it is a problem too, it hides essential parts from a quick view or requires very powerful management application (XML IDE). It is very interesting if there is such an IDE and how it manages proposed combination of XSD+XSLT+XML.

Hey, SpyderTL, where are you?

I hope it's not me to blame for your escape? And I hope you do not give up your idea because of my comments

embryo wrote:Hey, SpyderTL, where are you?

I hope it's not me to blame for your escape? And I hope you do not give up your idea because of my comments

Nope. Just been busy for the past few days. Finishing up a critical project at work, and two kids take up a bulk of my time. And the 360 is eating up the rest...

Yes I really have all of those platforms listed above defined in XSD. Actually, what I do is start with an XML file that basically has name value pairs for each instruction name and byte code, or memory address, or constant value (like BIOS interrupt call values).

Then I have 2 XSLT files, one which transforms that file into an XSD that defines each of those values as an element in a namespace, and the other which transforms that same XML file into another XSLT file that will convert those elements into their corresponding values (usually hex values).

XSLT can't transform directly to binary format, so I wrote a C# class that will read the "level 0" XML file and write out the bytes to a file or device. This is the piece that I could not find, and had to write my own.

Then it's just a matter of creating higher level XSD/XSLT pairs that contain those level 1 elements to perform specific functions (write characters to screen, read blocks from a floppy drive, etc.). The XSD file defines the function names as elements, and the XSLT transforms those elements into inline functions (blocks of lower level elements), which are then broken down themselves.

Right now, each project had its own compiler, due to having to dress up the output file (i.e. Setting the last two bytes in a boot loader, calculating checksums for n64 cartridges, etc.) But I'm planning on moving this functionality out to XML as well, as soon as I figure out how.

Using XSLT to do your compilation has some interesting advantages. For instance, I can include a significant amount of logic in XSLT, so that I can do things like emit two completely different versions of a function based on how many parameters are passed (i.e. How many attributes are included in the function element.)

There are some disadvantages as well. You have to take great care that you put your CPU instructions in the right order, and you have to know how many bytes each instruction requires. Forget one little element, and you instantly have completely random behavior.

Still, it is rewarding when it all finally works, and there is no better way to learn the hardware from the inside out.

I'll post some example code for you guys to look at shortly.

Thanks for the feedback, guys.

As for the 'classes', I have a XSD definition for classes that allow you to define the name, the namespace (XML style), the base class namespace, and any fields and/or methods in that class. Then I have an XSLT file that puts it all into a structure that the OS expects, and the OS allows you to type a static class name and a method name to execute, and any methods to call on the returned object, all chained together with a dot, C# style.
I don't have any support for passing arguments to functions, yet. But it's on my list. All of this has been written for 32-bit x86, but I just started over on a 64-bit version, and I'm rethinking all of my structures and file system stuff at the moment.

I may post up a bootable ISO file as well, if anyone is interested.

All of the "layers" described above are all "compiled" down to layer 0 at build time, except for layer 4, which I've barely even started on. Layer 4 is basically a virtual machine layer that only allows you to define a high level workflow by creating classes with methods that contain calls to other class methods. I'm still working out the details, but I'm already realizing that XML may not be the best approach for that.

The actual program itself is written in XML, and contains elements from all of the namespaces from all of the various XSD files that have been created so far. It can contain any elements from layers 0, 1 and 2. (To use layer 3 elements, you really need an OS.)

For example, this code would display "Hello World!" on the screen and halt the processor: I've left out the namespaces, but con: is a collection of 16-bit x86 BIOS INT 10h console methods, and cpu: is a list of 16-bit x86 CPU instructions.

WriteCharacters generates more cpu instructions to write individual characters to the screen (as opposed to copying them as a string). DisableInterrupts is the CLI instruction, and WaitForInterrupts is the HLT instruction. The memory address location of the code (i.e. org 0x7c00) is defined outside of the program file, but since there are no address references in this code, you could put it in a boot loader, or in a .com file, and it would (probably) work.

The WriteCharacters function above demonstrates one of the advantages of using XSLT as a "compiler". In this case, it loops through the text in the WriteCharacters element, and writes out x86 cpu instructions for each character. So passing a 2 character string generates a lot less code than it would if you were to pass 10 characters, or 100 characters. I have another function, called con:WriteString that is used the same way, but it generates cpu instructions that copy all of the characters to the screen in a loop, so that the code that is output will always be the same size.

Another advantage is that you can map your namespace to whatever prefix makes sense to you. So you could use <x86:WaitForInterrupt/> if you wanted, or <console:WriteCharacters/>.

It is also trivial to rename the elements, or add new elements that make more sense to you. For example, you could create your own set of XML/XSD/XSLT files that had elements like <x86:cli/> and <x86:hlt/>, and you could use those instead. As long as the values are correct in the XSLT file, it would work exactly the same. You could even mix and match them if you wanted. As you may have noticed, I've named my cpu instructions in a way that makes sense to me. I'm using visual studio for all of my XML editing, so I get intellisense, and autocomplete, and documentation tooltips for free, since visual studio is smart enough to read the XSD and display any comments that are attached to the elements defined in the XSD file. This makes life much easier. (Imagine NASM with intellisense and autocomplete and popup documentation...)