Object Oriented OS

[embryo]

As a matter of fact, I have one huge example that shows how to make a simple operating system with objects and a shell. The shell even has Intellisense and Autocomplete.

You have done a great job, of course, but it can be a bit better

[embryo]

Just let me know what you need.

Let it be a really good object oriented OS in XML

[embryo]

Wait a sec. You would have run into this same problem with the embryo assembler. How did you name your java/native platform instruction methods? X86.MOV(source, destination)?

Actually it looks like this:

Code: Select all

mov.x8(AL,i(EBX));

Where x8 is a method, 8 is number of bits consumed by the instruction, mov, AL and EBX are fields, the i is another method and returns an indirect addressing data structure.

In assembly it looks like this:

Code: Select all

mov byte AL,[EBX]

[embryo]

The first version of my IDE will be similar to a traditional IDE (e.g. user enters text and sees text, without necessarily knowing or caring that the IDE is converting to/from tokens while they're typing), but probably with a little "source code style" configuration option (so different people can set their IDE to "native style" or "C-like style" or "no braces Python-like style" or whatever they want). Then I'll add features to extract information from the source code and generate diagrams (e.g. dependency diagrams, etc) to make it easier for people to "see the forest amongst the trees" and to aid navigating through the project. After that I'd probably add a "drag and drop" style to it (mostly for children and/or touchpad support).

It looks like a parsing "on the fly" and looking for a set of predefined words to create some internal representation. But there already are many tools with such capabilities. There is customizable coloring of entered text, custom fonts and even some custom cursor behavior. The only problem here is - how to use the structure information about the user input. And it is mostly the problem of visualization. Efficiency is not a problem, considering your close attention to it. But what will be an efficient structure used for? It is a problem. Because user interaction domain is complex and very large. May be you need to read something about it? Just to get used to the well defined terms of the domain, to get acknowledged with ideas from the domain and so on.

Ideally, I want people to be able to manipulate source code in some sort of 3D environment; possibly where the CPU/s are represented as machines that follow a track, and things like branches, loops and function calls are just track layout, and expressions/statements are devices placed on the track.

But what is the goal of such manipulation? If you state it then you can divide it in some implementation steps. But more probably - you will find that it should be divided in a lot of subgoals.

[Brendan]

Hi,

The second point is about machine friendly efficiency. I agree, that shorter data formats are better suited for such purpose. Also I agree, that a parser friendly format leads to a greater performance. And I agree, that if we have the user friendliness as a main goal in mind when we develop some system, then the benefits above should stop us when we try to think about developer problems. But we have one problem here, as it was mentioned in the osdev.wiki, you shouldn't do it unless you have many lives to spend. It is really serious problem. May be it is possible to organize our society in a user friendly manner and really stop thinking about complexity and time spent, but, unfortunately, the society is in no way close to such imaginary form. It encourages speed and greed, but not the waste of time for something not promising quick return on investment. It means while we have no another society - most of us should work quickly and forget about long term goals of quality and beauty. But yes, it contradicts with the osdeving "for fun"...

If we're only recycling the status quo and not doing anything different (because that's harder and takes longer); then it's a waste of time bothering to write software in the first place (note: this includes writing software for the purpose of gaining knowledge, as the knowledge gained would have no practical use). We must do something different or there's no point. "Different" can mean better or worse, but making things worse than pre-existing stuff isn't an attractive goal.

With this in mind, let's define 2 important words:

• Success: Producing something that is better than existing stuff for at least one purpose (not necessarily "better for all purposes")
  Failure: Failing to producing something that is "better for any purpose"; including:
  • Failing to produce something that works
  • Producing something that does work, but is not better than existing stuff

And now the point four - it's just a matter of a format standard acceptation, if it has a great user base, then there will be a lot of tools. Are the tools efficient or not - is the problem I have mentioned above.

Note that my definition of success (above) does not in any way depend on adoption - adoption is just a side-effect of success and there are no guarantees. However, if something works and is better then it's more likely that people will adopt it, and if something either doesn't work or isn't better then there's no reason for people to adopt it.

If you are the person creating the tools and only care about how long it takes (and hate the potential users of your tools so much that you don't care how much they suffer), then XML isn't "worse" at all.

You've got it - most of the people care about "how long it takes" (what is the cost of).

Yes, but that only leads to failure.


Cheers,

Brendan

[SpyderTL]

For an example, for your own language, you've got a fixed number of valid identifiers (e.g. "CopySIAddressToAXAndIncrementSI"). If you enumerated these identifiers it's extremely likely that you could encode it as a 16-bit integer instead, which would avoid the need for string comparisons and probably reduce file sizes by an order of magnitude.

(I'm actually really enjoying this conversation...)

I actually am working in the opposite direction (basically, one XML tag per opcode byte). At this level, I'm creating identifiers for each of the platform specific opcodes. Taking my identifiers and turning them back into numbers would be redundant.

Now, with that being said, something that I could do, and have considered doing, is to write a new layer on top of this layer that is platform specific, but easeir to use (one tag per instruction). I would need to find a balance between verbosity/readability and complexity/functionality.

Readable (simple to write code, difficult to implement in XSLT):

Code: Select all

<copy><ax/><bx/></copy>

Functional (simple to implement compiler in XSLT, but would require lots of tags):

Code: Select all

<copyAxToBx/>

Compatible (cross-platform, but would require converting in your head)

Code: Select all

<copy fromRegister=1 toRegister=2/>

I guess, technically, I could do all three of these and let the user choose, since they would all eventually end up compiled down to the opcode specific version above.

[Brendan]

Hi,

The first version of my IDE will be similar to a traditional IDE (e.g. user enters text and sees text, without necessarily knowing or caring that the IDE is converting to/from tokens while they're typing), but probably with a little "source code style" configuration option (so different people can set their IDE to "native style" or "C-like style" or "no braces Python-like style" or whatever they want). Then I'll add features to extract information from the source code and generate diagrams (e.g. dependency diagrams, etc) to make it easier for people to "see the forest amongst the trees" and to aid navigating through the project. After that I'd probably add a "drag and drop" style to it (mostly for children and/or touchpad support).

It looks like a parsing "on the fly" and looking for a set of predefined words to create some internal representation. But there already are many tools with such capabilities. There is customizable coloring of entered text, custom fonts and even some custom cursor behavior.

Yes, the traditional approach is for the IDE to parse text when it enters the IDE (either typed by the user or loaded from file system) so that the IDE can do "fancy features"; and then that work is discarded so that the same text has to be "re-parsed" every single time the file is loaded (by the IDE or a compiler or anything else). Mostly all I'm doing is reducing "work done then discarded" - changing the file format used for source code (from plain text to tokens) doesn't change anything other than efficiency.

The only problem here is - how to use the structure information about the user input. And it is mostly the problem of visualization. Efficiency is not a problem, considering your close attention to it. But what will be an efficient structure used for? It is a problem. Because user interaction domain is complex and very large. May be you need to read something about it? Just to get used to the well defined terms of the domain, to get acknowledged with ideas from the domain and so on.

Ideally, I want people to be able to manipulate source code in some sort of 3D environment; possibly where the CPU/s are represented as machines that follow a track, and things like branches, loops and function calls are just track layout, and expressions/statements are devices placed on the track.

But what is the goal of such manipulation? If you state it then you can divide it in some implementation steps. But more probably - you will find that it should be divided in a lot of subgoals.

I refuse to believe that text is the most ideal user interface for programming possible. Let's look at a few other fields. For electronics, circuits are described via. diagrams and not plain text. For architecture, CAD/CAM, mechanics it's the same - a graphical representation not text. For statistics we have charts and graphs. For geography we use diagrams/maps. Even for software there's been many attempts at representing code in a graphical way (UML diagrams, flow charts, Nassi–Shneiderman diagrams, etc). The reason for all of this is obvious - humans understand information easier when it's presented in graphical form.

This makes me "strongly suspect" that there's a better way of representing and interacting with (writing) code than the text-based editors we're all familiar with. The only problem I see is that the ideal user interface for programming hasn't been discovered yet.

Basically; I don't know what ideal user interface for programming is, I only know it hasn't been discovered yet and it's not text.

One part of my project's goals is to make it easier for someone (maybe a future me) to discover the ideal user interface for programming.

The largest thing preventing the discovery of the ideal user interface for programming is people's imagination. It's extremely difficult for people (including me) to imagine anything that they haven't seen before; and when an IDE reads text and writes text it's natural to assume the IDE is a glorified editor of text so most people don't even try to imagine anything better. Storing source code as "not text" helps to break the "IDE is a glorified text editor" assumption; and may be a significant step towards enabling the discovery of the ideal user interface for programming. The idea of a 3D environment where CPU/s are represented as machines that follow a track might sound awkward (and to be honest, to me it sounds more like a children's game than something a professional programmer would want); but that's just a way to encourage people to try to imagine something they haven't seen before (another potentially significant step towards enabling the discovery of the ideal user interface for programming).


Cheers,

Brendan

[Brendan]

Hi,

For an example, for your own language, you've got a fixed number of valid identifiers (e.g. "CopySIAddressToAXAndIncrementSI"). If you enumerated these identifiers it's extremely likely that you could encode it as a 16-bit integer instead, which would avoid the need for string comparisons and probably reduce file sizes by an order of magnitude.

(I'm actually really enjoying this conversation...)

I actually am working in the opposite direction (basically, one XML tag per opcode byte). At this level, I'm creating identifiers for each of the platform specific opcodes. Taking my identifiers and turning them back into numbers would be redundant.

Now, with that being said, something that I could do, and have considered doing, is to write a new layer on top of this layer that is platform specific, but easeir to use (one tag per instruction). I would need to find a balance between verbosity/readability and complexity/functionality.

Readable (simple to write code, difficult to implement in XSLT):

Code: Select all

<copy><ax/><bx/></copy>

Functional (simple to implement compiler in XSLT, but would require lots of tags):

Code: Select all

<copyAxToBx/>

Compatible (cross-platform, but would require converting in your head)

Code: Select all

<copy fromRegister=1 toRegister=2/>

I guess, technically, I could do all three of these and let the user choose, since they would all eventually end up compiled down to the opcode specific version above.

You're letting "the tail wag the dog", if that makes sense.

Instead of designing a good programming language syntax (and then finding a way to implement tools that support the programming language syntax) you are doing the opposite - selecting the way you want to implement it, and letting the severe limitations of that "way you want to implement it" destroy the design of the programming language syntax.

Note that the most natural syntax for copying a register to another (for people who aren't assembly language programmers) is extremely obvious:

Code: Select all

ax = bx

Cheers,

Brendan

[SpyderTL]

I'm guessing you haven't actually written any assembly language before.

Good guess.

Using ASM was my original plan. But I assumed that the "tools" that were available nowadays were more, um, mature than they were 20 years ago. That turned out to be wrong. Basically I would be developing in NotePad.

That's when I decided to go my own route.

Do you honestly think that (e.g.) auto-complete or tool-tips are helpful for something like "lodsb" for any assembly language programmer that isn't a complete beginner?

Not for the "core" instructions, like MOV and LODSB, but there are a lot more instructions out there that could use some tooltips to help explain what it does and how it is used. Let's try an example... without google or intel manuals, can you tell me how to use the XLATB instruction? Or which instruction to use if I want to convert a short (16-bit) to an int (32-bit)? Or what happens if you set SP to 1 and then PUSHF?
Or how about writing ASM for an ARM9 without looking at the documentation? I can do that, and I have autocomplete, intellisense and tooltip documentation, and I didn't have to write my own IDE.

I would consider myself an expert C# developer. I've been using it as a business development language since 2002, and developed in Java before that. The naming conventions used in both C# and Java are well organized, and easily readable, at least for me. But I still use autocomplete, intellisense and I still read the tooltips that pop up as I'm searching for a specific method. That's because there are thousands of methods in the base libraries, and I couldn't possibly remember everything about all of them.

Autocomplete and Intellisense make less sense if you are programming in ASM, but I consider that a disadvantage to using ASM, and not an advantage.

Seriously; why not post an example of your "XML assembly" code somewhere (like the #asm IRC channel on freenode, or alt.lang.asm newsgroup, or the FASM forums), and ask actual assembly language programmers to comment on the syntax? I'm sure that you'll get important feedback.

(I thought that's what this whole thread was about... )

The same reason that I don't post ASM code and ask C# developers what they think. Bias.

All of the code is up on http://ozone.codeplex.com. Feel free to take a look and make any (positive) suggestions you want. Just saying it's "stupid" or "will never work" will be ignored.

Yes; so instead of doing it right (and implementing a suitable editor), they were lazy.

Go ahead and give it a try. Download MESS, and write a non-text editor IDE for an Altair 8800. (Intel processor, up to 8KB of memory!)
Unless you are "too lazy"

That may have been due to the fact that you were limited to about 2-4KB of memory...

The reverse actually - when there's severe limits on RAM and CPU speed you want to do everything you can to avoid wasting RAM and wasting CPU time.

Yeah, like loading an IDE into memory. Writing keystrokes to a file takes a lot less memory than a program that converts text to bytecodes in real time.

However, I only have preliminary ideas for the IDE's user interface. The first version of my IDE will be similar to a traditional IDE (e.g. user enters text and sees text, without necessarily knowing or caring that the IDE is converting to/from tokens while they're typing), but probably with a little "source code style" configuration option (so different people can set their IDE to "native style" or "C-like style" or "no braces Python-like style" or whatever they want). Then I'll add features to extract information from the source code and generate diagrams (e.g. dependency diagrams, etc) to make it easier for people to "see the forest amongst the trees" and to aid navigating through the project. After that I'd probably add a "drag and drop" style to it (mostly for children and/or touchpad support).

Beyond that, I don't know. Ideally, I want people to be able to manipulate source code in some sort of 3D environment; possibly where the CPU/s are represented as machines that follow a track, and things like branches, loops and function calls are just track layout, and expressions/statements are devices placed on the track.

Awesome. Let me know when it's done.

In the meantime, I'll be working on my OS.

I know that sounded snarky, but, seriously, that would be awesome. There was a PSOne game called Carnage Heart. http://en.wikipedia.org/wiki/Carnage_Heart

In it, you built your robot, and programmed it using a visual programming language where you plugged in different "chips" into a giant grid of slots. The "processor" would start at the top left slot on the grid (0,0), and each chip would do it's function, and then it would redirect the cpu next instruction either to the chip directly to the right, or to the chip directly beneath it, based on some logic (if statement branch). As the cpu ran, you could see the processor "move" from chip to chip. I've always remembered that, because it's one of the first true visual programming languages that I'd ever seen.

Who cares if builds take 500 ms longer?

Um, what? You realise 500 ms is half a second?

For my current project; a script builds my own "build utility", then it checks everything and regenerates my any web pages that changed (docs, etc), builds any utilities that changed, uses those utilities to compile anything that needs to be recompiled, and generates a full backup (and manages my backup directory); and this typically takes less than 10 ms. Absolute worst case (everything fully rebuilt entirely from scratch) currently takes 820 ms.

And how many times, per second, do you build?

[SpyderTL]

No. The plan is to write temporary/crappy tools that work on Linux; then use those tools to write a (minimal) OS, then write native compilers and the IDE for that OS.

Of course I do care about the quality of the end result, because the end result is the entire point of it all. This means that if I'm not happy with something (e.g. I think I can do better) I stop what I'm doing and redesign it. So far everything has been redesigned at least twice since I started designing the languages and toolchain.

Same here. Eventually, I would like to be able to be able to run my own OS, and compile my own C#/Java-like code.

Ironically, I think you and I and embryo are all going to end up with exactly the same end result. We just took 3 completely different paths to get there. (And I fully expect to be the last to arrive...)

I would suggest that we all work together, on one project, but something tells me that would be a disaster!

[SpyderTL]

The problem with typing XML is having to close tags, and your tags seem very verbose. With other languages you can simply type ],}, or ) and move on - making a specialized editor irrelevant

So Visual Studio is irrelevant for writing C#? Should I just be using Notepad.exe, and calling CSC.exe to compile?

BTW, VS2013 supports JSON Schemas.

I... did not know that.

[SpyderTL]

Wow...

then, How do you compile it? The code

Code: Select all

<program>
    <con:WriteCharacters>abc</con:WriteCharacters>
    <cpu:cli />
    <cpu:hlt />
</program>

must be compiled like this, isn't it?

Code: Select all

[bits 16]
[org 0x7c00]
mov ax, 0xb800
mov es, ax
xor di, di
cld
mov ax, 'a' | (0x07 << 8)
stosw
mov ax, 'b' | (0x07 << 8)
stosw
mov ax, 'c' | (0x07 << 8)
stosw
cli
hlt

Do you use your own "compiler"? Or is there any easy and simple way to do this?

Basically, yes. (mov al, 'c', actually).

But I'm bypassing the ASM step. The xml above is recursively converted into "simpler", lower-level xml tags, until finally it's nothing but:

Code: Select all

<hex>a0</hex>
<label id="start"/>
<hex>00</hex>
<hex>ff</hex>
<string>c</string>
<byte>1</byte>
<addressOf ref="start" type="Relative8"/>
...

Luckily, all of this happens in the background, so you don't actually see any of this XML, unless you look at the temp files that are produced.

Once the code is "converted" down to this level, the compiler simply reads the <hex> tags, and writes them out to a file.

[SpyderTL]

If you're spending most of your development time figuring out syntax, then I'd blame the language.

I'd just like to point out, XML syntax is pretty simple. Only 6 symbols (<:="/>), and one keyword (xmlns).

Once you've mastered those, you can use namespaces to organize all of your instructions, commands, functions, data, and constants that you need.

Tack on another 20 or so "keywords" for your base XML to Binary language (byte, short, int, label, etc).

Then all you need is a few thousand "keywords" for your CPU instruction set...

See? It's simple!

[SpyderTL]

Here we talk about efforts that are required to simplify the machine code and represent it in some form with higher abstraction level. Your approach maps machine code directly to some XML tags, but it is very low abstraction level. And what is the importance of the level of abstraction you can see in any contemporary high level language.

In order to cover all of the bases, I need several "layers" of code, each with its own language. The bottom layer is a direct one-to-one mapping from cpu instructions to xml elements. Above that, you get functions and variables, and above that you get classes and methods. (A little something for everyone).

I think the XML works well, by itself, for the low level stuff, but for the high level stuff, I think I'm going to need something that's easier to code. I haven't really gotten that far, yet.

Maybe I can clarify what I actually did in a way that focuses on the advantages of this approach... and separates it from the traditional approaches you would normally use.

There are a few developers (maniacs) out there who believe that Assembly is too high level. There are a lot of developers who think that Assembly is too low level, takes too long to code, requires a significant amount of reference documentation, and lacks any modern tools to provide assistance to new developers.

But if you want to write a boot loader, or if you want the absolute smallest and fastest program (at run time), you really have no alternative. Assembly has no competition at this level.

But it definitely has its faults. Each platform has its own "flavor" of assembler, meaning an expert in x86 assembly could be a beginner in ARM assembly. There is no inherent concept of functions, code blocks, namespaces (to resolve naming conflicts), scope (also to resolve naming conflicts) or variables.

Since it has no real competition, none of this really matters. But what if it did? What if there were a modern alternative to Assembler, that had some (or ideally, all) of these features, generated the same small, fast results as Assembly does, and could be used to write an Operating System from scratch, and could cross-compile to any platform, from any platform? (including VMs like Java and .NET)

It's a lofty goal, but I think that I have, at least partially, achieved it. Could it use some improvement? Definitely. Is it the "best" solution? Possibly not, but it does cover all of the bases above, and it uses tools that already exist, and formats that are well-known and well supported.

I'm always looking for ideas for improvement, or even new approaches that I haven't thought of. But "just use ASM" isn't a new idea, so you guys are going to have to do better than that.

[SpyderTL]

But I see a lot of C# files in your project. Are all of them needed just to translate "ABC" to bytes 65, 66, 67?

The C# code is there to do what I can't do easily in XSLT. Reading the XSLT files, transforming the XML files, and writing out the bytes to disk is obvious. Other things include calculating checksums for copy protection on game consoles and writing data intensive structures, like file system table entries for bootable CD images. (These are all things that have nothing to do with the actual OS code, itself, so I don't feel too bad about cheating and writing them in C#)

Also, I don't feel like running all of the XSLT's by hand, every time I make a change to a single file.

But you have a lot of tools for it. Just let a tool to do it's job.

The tools only translate one file at a time, which would take forever if you made a change to one file that impacted 20 other files. The C# code detects which files have changed, and which files are impacted and need to be recompiled, and recompiles them all at once.