Object Oriented OS

[embryo]

That's because it's not always a simple name to hex value translation.

Well, then why not to translate entire operation like MOV AX, BX in all it's variations like MOV AX,[BX]; MOV [AX],123; and so on?

But I can understand a beginner way of thinking - there actually are many instructions even in Intel's manual under MOV instruction definition, all of them are named uniformly, but use different opcodes and other parts of translated instruction. And it seems you have decided to split uniformity of names into something more close to the actual machine code. It, of course, simplifies your translation job, but it also cuts all possible ties with the legacy syntax. And how important the syntax is you can see from the comments above. For example - you can write identical programs using pascal (delphy) or C, but the syntax difference made the pascal an obvious looser. And this example still misses the scale of change you are demanding from others.

In order for the C# code to convert these files into a byte stream, everything must be converted into elements from the Programs/Program.xsd schema. Those are the only tags that the C# code understands.

But why not to have plain text result in hexadecimal form and then to convert it into raw bytes using C# or anything else? Then you can have all build stages in XML, except the last and simplest - hex to byte translation.

[embryo]

If I'm writing a boot loader, or a console application, I don't really care that the instruction ends up being converted to a zero, just like I don't care if I write

Code: Select all

ADD AX, 1

that it ends up spitting out a zero. But it is nice to know that I can look it up, if needed.

Do you mean the translation part is of no importance to the programmer? Maybe. But at least it is a good idea not to leave there the "value" attribute, without it the code will be more efficient and less obscure.

[embryo]

Which is easier to understand?

Code: Select all

lodsb

or

Code: Select all

<cpu:CopySIAddressToAXAndIncrementSI/>

For a beginner without any assembly background it is a bit viable. But for an any more experienced programmer it is a pain.

But the beginner approach is flawed in your case. If we have to describe the actual meaning of the instruction it is much better to describe all parts of instruction job. And such description is relatively long. Then the command can look like:

Code: Select all

<cpu:CopyByteFromAddressInSIToALAndIncrementSI/>

And it is not the best descriptive instruction name. But such long names are really inconvenient. I have used a bit of such names for some methods, but I still doubt if it is better to place the description in the method name instead of the method documentation. In my case it was a situation, when it is not convenient to look for any documentation and reading method name is a bit quicker than reading intellisense's text. But in your case we have an instructions that are used to often to read about them every time you are going to use them. The difference is - how often you are using something. If the usage is regular then there is no need in auto-documenting of the code (by means of method name or instruction element name). But if the usage is occasional or infrequent, then long method or element name can help a bit.

[Brendan]

Hi,

Which is easier to understand?

Code: Select all

lodsb

or

Code: Select all

<cpu:CopySIAddressToAXAndIncrementSI/>

For a beginner without any assembly background it is a bit viable.

Before a beginner can use an instruction they have to know it exists and what it does (at least enough to know they want that instruction), and they find out it exists and all the details (including which segment register/s it uses and how the direction flag effects it) from Intel's manual. Then they take the knowledge they learned from the manual (including the "lodsb" mnemonic) and try to apply it in practice and would end up confused.

But the beginner approach is flawed in your case. If we have to describe the actual meaning of the instruction it is much better to describe all parts of instruction job. And such description is relatively long. Then the command can look like:

Code: Select all

<cpu:CopyByteFromAddressInSIToALAndIncrementSI/>

For an accurate descriptive name it should be:

Code: Select all

<cpu:CopyByteFromAddressInDS:SIOrDS:ESIDependingOnAddressingModeToALWithoutEffectingFLAGSAndThenEitherIncrementSIOrDecrementSIDependingOnDirectionFlagAndRepeatUntilCXorECXisZeroIfREPPrefixPresentThenIncreaseEitherIPOrEPIBySizeOfInstructionUnlessTheresASegmentTypeProblemOrSegmentLimitViolationOrPageFaultAndAnExceptionHappensInstead/>

Cheers,

Brendan

[AndrewAPrice]

That depends. Do you already know x86 Assembly?

Let me put it another way. Which is easier to understand?

Code: Select all

lodsb

or

Code: Select all

<cpu:CopySIAddressToAXAndIncrementSI/>

Why not just write an assembler where the instruction is called CopySIAddressToAXAndIncrementSI instead of lodsb? They have to learn XML, and learn assembly.

So they can see descriptive name of the instruction - they still need to learn assembly. Registers, stacks, the control flow of assembly programs.

When working in a new language, you can pick up working knowledge of the syntax fairly quickly, but most of your time trying to do something useful will be spent figuring out how the standard library works. An IDE that lets you see the symbols/procedures/types exported by another file is useful for this.

The same IDE that references the Intel manual when you hover your mouse over an instruction would be a much more useful tool, than say, forcing them to type and remember 30+ characters (case sensitive and all) for something that only requires 5.

It's not that it's a bad idea, if you're goal is an XML-based language, and you have an IDE that closes tags, autocompletes long and verbose names, to the fact that on my keyboard I can type:

Code: Select all

mov[tab][ax][tab]bx[enter]

And it auto-generates:

Code: Select all

<cpu:Move>[enter]
[tab]<Source>[enter]
[tab][tab]<Cpu:MemoryAddress>[enter]
[tab][tab][tab]<Cpu:Register>ax</Cpu:Register>[enter]
[tab][tab]</Cpu:MemoryAddress>[enter]
[tab]</Source>[enter]
[tab]<Destination>[enter]
[tab][tab]<Cpu:Register>bx</Cpu:Register>[enter]
[tab]</Destination>[enter]
</cpu:Move>[enter]

12 keystrokes vs 189 keystrokes (or 241 if you count the 52 times I had to press the shift key.)

s-expression:

Code: Select all

(move (ataddr ax) bx)[enter]

21 keystrokes (or 25 if you count the 4 times I pressed shift.)

Also, you need to think about for code compactness? It's much easy to comprehend code when you can see as much of it as possible on screen (without being overly messy). In many companies, this:

Code: Select all

switch (item_hit) {
  case "coin":     points += 1;    break;
  case "enemy":    points += 10;   show_message("You hit " + item.name + "!"); break;
  case "teammate": points -= 10;   show_message("You hit your own teammate!"); break;
  case "flag":     capture_flag(); break;
}

is considered 'better practice' than this:

Code: Select all

switch (item_hit)
{
 case "coin":
  points += 1;
  break;
 case "enemy":
  points += 10;
  show_message("You hit " + item.name + "!");
  break;
 case "teammate":
  points -= 10;
  show_message("You hit your own teammate!");
  break;
 case "flag":
  capture_flag();
  break;
}

Simply because the former is much more legible - it's easier to read because a) it looks like a table, with the switch statements on the left, code on the right, and b) you can fit much more on your screen - maybe a switch statement with 50 lines of code could fit completely on screen. 100 lines of neat code is often better than 500 lines of code of the same code with everything on it's own line, since we're able to visualize the shorter code much better - that is, along as it's not messy. E.g.:

Code: Select all

switch(i){case "c":p+=1;break;case "e":p+=10;m("You hit "+i.n+"!");break;
case "t":p-=10;m("You hit your own teammate!");break;case "f":f();break;}

That definitely isn't legible. You can go extreme in both ways - it's all about finding a happy medium - the unreadable 2 lines, vs the neat 6 line table, or the verbose 17 lines. The verbose version takes up nearly 3 times as much screen space as the neat table. That results in much more scrolling, and much less code you can see at once.

Extreme case, I work with people that love to write:

Code: Select all

SELECT
  a.abc abc,
  b.def def
  MIN
    (
      a.ghi
    )
    jkl
  FROM
    hij a,
    lmn b
  WHERE
    (
      a.opq
      =
      b.rst
    )
    AND
    (
      wxy
      =
      'z'
    )

A lot of tools even generate code like that.

I feel that an XML language, while it may help adoption of someone who is familiar with XML and wants to write a small tool, will be very cumbersome for large complex programs, namely because it will be very verbose, taking up maybe 10+ lines (and 20x the keystrokes) for something that may take 1 line in another language.

[SpyderTL]

Something like "<cpu:CopySIAddressToAXAndIncrementSI/>" is an extreme risk to human life - many people will die from the dehydration caused by uncontrollable vomiting.

Stop this frakking madness and start using short, memorable (it's called a mnemonic, you know) and recognizable identifiers. Don't invent ridiculously long, not memorable and not easily recognisable ones just because you are too lazy to properly learn your tool before using it.

Wow!

I think I may have struck a nerve...

I don't have to read mnemonics (except thise crazy SIMD ones)

I think you just proved my point.

[SpyderTL]

It means you have an opportunity to regenerate such files in two ways - one, which would follow long command description pattern, and second, when a close to the standard assembly syntax is used.

Yep. But you still have the problem that MOV will actually be converted into one of about 20 different instructions (single byte and multi-byte) depending on the situation. Your XSLT would have to be awfully smart in order to pick the right one. I'd rather just specify exactly what I want the CPU to do, as a developer.

[SpyderTL]

Well, then why not to translate entire operation like MOV AX, BX in all it's variations like MOV AX,[BX]; MOV [AX],123; and so on?

That's essentially what I did. I just had to name them differently so that you could pick the specific "MOV" function that you want.

...And it seems you have decided to split uniformity of names into something more close to the actual machine code.

That is precisely what I did.

It, of course, simplifies your translation job, but it also cuts all possible ties with the legacy syntax. And how important the syntax is you can see from the comments above. For example - you can write identical programs using pascal (delphy) or C, but the syntax difference made the pascal an obvious looser. And this example still misses the scale of change you are demanding from others.

I didn't set out (5 years ago... whew) to write something "like ASM". (Okay, well maybe I did. But that was a long time ago. )

But what I ultimately wanted was a completely new way of looking at code. I proposed that all code (instructions, commands, functions, RPC calls, workflows, etc.) was actually just sequenced data, and therefore, should be well suited to be stored in a data-friendly format. I didn't want to put the "data" in a database, so that pretty much leaves CSV files, tab delimited, XML and JSON. I chose XML due to its widespread compatibility and support from various tools and platforms.

I'm pretty happy with what I've accomplished, so far -- as an academic exercise, if nothing else. I've learned a ton about ASM, x86, gaming consoles, networking and virtual machines.

And, by storing all of the spec sheet "table" data as XML, and publishing it as open source, others may benefit by "converting" that XML data to other formats, like language specific enums, or even as a basis for new virtual machines.

But why not to have plain text result in hexadecimal form and then to convert it into raw bytes using C# or anything else? Then you can have all build stages in XML, except the last and simplest - hex to byte translation.

That is precisely what I'm doing. But rather than manually transforming the "high-level" XML files, by hand, I'm letting the compiler do that for me (The C# code). So it converts all of the non-program.xsd XML into program.xsd compatible XML, and then it writes that XML out to disk as a binary file. It basically works like if you were to run a C compiler, and you were to force it to always inline every function call.

[SpyderTL]

Do you mean the translation part is of no importance to the programmer? Maybe. But at least it is a good idea not to leave there the "value" attribute, without it the code will be more efficient and less obscure.

The conversion from "high-level" XML to "low-level" XML (program.xsd compatible XML), and the conversion from the "low-level" XML to a binary file all happens at the same time, when you run the compiler. You never see the intermediate (temporary) XML, unless you happen to be looking at the temp folder as the compiler is running.

[SpyderTL]

For a beginner without any assembly background it is a bit viable. But for an any more experienced programmer it is a pain.

I think the same could be said of any programming language. If you already know a language, it's easier to keep using that language than to switch to another language. (It's also easier for us to speak and post messages in English, than it would be if we were all to switch to a new language that no one has ever seen before. But that doesn't necessarily mean that English is the easiest language to learn and/or use.)

But the beginner approach is flawed in your case. If we have to describe the actual meaning of the instruction it is much better to describe all parts of instruction job. And such description is relatively long. Then the command can look like:

Code: Select all

<cpu:CopyByteFromAddressInSIToALAndIncrementSI/>

And it is not the best descriptive instruction name. But such long names are really inconvenient. I have used a bit of such names for some methods, but I still doubt if it is better to place the description in the method name instead of the method documentation. In my case it was a situation, when it is not convenient to look for any documentation and reading method name is a bit quicker than reading intellisense's text. But in your case we have an instructions that are used to often to read about them every time you are going to use them. The difference is - how often you are using something. If the usage is regular then there is no need in auto-documenting of the code (by means of method name or instruction element name). But if the usage is occasional or infrequent, then long method or element name can help a bit.

I agree that some of the names could be improved. As a matter of fact, some of them have been "improved" over time. For example, if you look at the 8086 platform XSD, you'll see that I named the HLT instruction "HaltProcessing". However, after I had used it for a while, and realized that "Halt" was sort of a misnomer, I renamed it later (in the 80386 platform) to "WaitForInterrupt", because it made more sense to me, and seemed to better represent what was going on. Luckily, changing an instruction name is fairly trivial, since I'm the only one currently working on the project, I can just do a solution-wide find/replace.

I've also considered changing the "ReturnToNearCaller" instruction name to something like "PullIPFromStack", because it more accurately describes what is happening. I'm still on the fence on that one, though. Also, for all stack operations, I've replaced the naming pair from "Push/Pop" to "Push/Pull", just because it makes more sense to me.

Remember, I'm going back in time, and revisiting all of the accepted history of computer software development, and looking for ways that I can use modern tools to solve the problems that developers had 50 years ago. I'm proceeding on the assumption that, if they had known then what we know now, that things would have turned out very differently. And "because that's the way it's always been done" is not an acceptable answer.

[SpyderTL]

Before a beginner can use an instruction they have to know it exists and what it does (at least enough to know they want that instruction), and they find out it exists and all the details (including which segment register/s it uses and how the direction flag effects it) from Intel's manual. Then they take the knowledge they learned from the manual (including the "lodsb" mnemonic) and try to apply it in practice and would end up confused.

This was definitely true 15 years ago. But I think things have changed a lot since then. Modern development is a lot more organized and documented.

Compare writing code in Assembly, Basic, C, C++ to writing code in Java or C#. Nowadays, you have intellisense, pop-up menus, balloon comments, Google, StackOverflow, etc. And all of the built-in functionality is neatly organized into namespaces and classes, compared to just having a ton of built in functions with no organization and no documentation (no metadata). This allows you to quickly learn and use a new language with little or no need for books or tutorials, or training sessions. I'm just trying to bring the same level of tooling to the 1970's era development platforms.

For an accurate descriptive name it should be:

Code: Select all

<cpu:CopyByteFromAddressInDS:SIOrDS:ESIDependingOnAddressingModeToALWithoutEffectingFLAGSAndThen.../>

Congratulations, you win the verbosity award!

[SpyderTL]

Why not just write an assembler where the instruction is called CopySIAddressToAXAndIncrementSI instead of lodsb?

Autocomplete, Intellisense, Documentation, F12 - View Definition, Inline Functions... Plus, for all intents and purposes, the whole build chain is open source. You can see exactly what is going on at every level. Not the case with NASM.

When working in a new language, you can pick up working knowledge of the syntax fairly quickly, but most of your time trying to do something useful will be spent figuring out how the standard library works. An IDE that lets you see the symbols/procedures/types exported by another file is useful for this.

That's why I chose XML. I have my choice of dozens of IDE's, and I don't have to write my own.

The same IDE that references the Intel manual when you hover your mouse over an instruction would be a much more useful tool, than say, forcing them to type and remember 30+ characters (case sensitive and all) for something that only requires 5.

Normally, I would agree. But in this case, I don't consider ASM a "better" development language than, say, C#, just because it uses commands with fewer characters.

It's not that it's a bad idea, if you're goal is an XML-based language, and you have an IDE that closes tags, autocompletes long and verbose names, to the fact that on my keyboard I can type:

Code: Select all

mov[tab][ax][tab]bx[enter]

And it auto-generates:

Code: Select all

<cpu:Move>[enter]
[tab]<Source>[enter]
[tab][tab]<Cpu:MemoryAddress>[enter]
[tab][tab][tab]<Cpu:Register>ax</Cpu:Register>[enter]
[tab][tab]</Cpu:MemoryAddress>[enter]
[tab]</Source>[enter]
[tab]<Destination>[enter]
[tab][tab]<Cpu:Register>bx</Cpu:Register>[enter]
[tab]</Destination>[enter]
</cpu:Move>[enter]

12 keystrokes vs 189 keystrokes (or 241 if you count the 52 times I had to press the shift key.)

s-expression:

Code: Select all

(move (ataddr ax) bx)[enter]

21 keystrokes (or 25 if you count the 4 times I pressed shift.)

I'm not sure if you looked at the code at all, but we are specifically talking, at the moment, about the difference between:

Code: Select all

MOV AX, BX

and:

Code: Select all

<cpu:CopyRegisterToOperand/>
<op:AX-BXRegister/>

Including the Shift key, and using autocomplete, in Visual Studio, that's about 28 keystrokes, versus 11 in ASM. I think that's a pretty fair trade-off for all of the other features you get (for free).

Also, you need to think about for code compactness? It's much easy to comprehend code when you can see as much of it as possible on screen (without being overly messy). In many companies, this:

Code: Select all

switch (item_hit) {
  case "coin":     points += 1;    break;
  case "enemy":    points += 10;   show_message("You hit " + item.name + "!"); break;
  case "teammate": points -= 10;   show_message("You hit your own teammate!"); break;
  case "flag":     capture_flag(); break;
}

is considered 'better practice' than this:

Code: Select all

switch (item_hit)
{
 case "coin":
  points += 1;
  break;
 case "enemy":
  points += 10;
  show_message("You hit " + item.name + "!");
  break;
 case "teammate":
  points -= 10;
  show_message("You hit your own teammate!");
  break;
 case "flag":
  capture_flag();
  break;
}

Simply because the former is much more legible

Companies don't "choose" a language based on how compact or what the code looks like on the screen. If they did, we'd all still be developing in COBOL.

I feel that an XML language, while it may help adoption of someone who is familiar with XML and wants to write a small tool, will be very cumbersome for large complex programs, namely because it will be very verbose, taking up maybe 10+ lines (and 20x the keystrokes) for something that may take 1 line in another language.

Yes, I admit that XML is about the most verbose option I can think of. Fortunately, code size is way down on my list of priorities, at least for this low level development. For higher-level, I'd much rather write code in, say C#, and then convert it into XML. But there's really no good C#-ish option, if you are trying to write a boot loader. The XML above is about as close as I can get.

I do, at times, feel like a cave man, banging rocks together, and trying to build a working time machine. But, that's part of the fun. I think I can honestly say that I am, very likely, THE only person on the planet who has ever tried this approach.

[Griwes]

I don't have to read mnemonics (except these crazy SIMD ones)

I think you just proved my point.

If you kept the context of that statement, you'd realize how dumb assuming I'm proving your point is.

What I said is, basically: short mnemonics are good, because one can learn how they look like (aside from how they are spelled) and not *read* them, but recognize them by their shape. Your "mnemonics" can get ridiculously long, which makes remembering shapes of them difficult - and they are written in CamelCase, which makes it virtually impossible to remember the shapes of single words composing them.

And the SIMD part just meant that in their case, I have to actually read the instruction; if you made, say, "PUNPCKHBW" spell "UnpackHighOrderWordDataElements", you would gain exactly nothing (and actually it'd probably take me longer to read the whole name once I've been using a particular instruction set for a while).

As people said before - optimizing for the newbie case is extremely dumb, because it tends to hurt productivity of people who actually need (or want) to be productive, not learn while using their tools (while they should've learned them beforehand). Look at the current state of our "glorious" first subforum. Tons of questions that could've been avoided if people decided to learn all they really need before starting writing an "OS" or a "kernel" (which are really just a bunch of tutorial code duct taped with garbage code they produced, because they decided to write a bootloader without knowing assembly or a "C kernel" without even knowing C).

[Bender]

Yep. But you still have the problem that MOV will actually be converted into one of about 20 different instructions (single byte and multi-byte) depending on the situation. Your XSLT would have to be awfully smart in order to pick the right one. I'd rather just specify exactly what I want the CPU to do, as a developer.

Code: Select all

mov ax, bx
mov ax,  word[bx]
mov ax, 20
mov ax, word [20]
mov word [20], ax
.......

No idea what the problem is to differentiate between this.

The x86 instruction set is huge, plus the "verbosity" of <XML/> and those assembler "sentences" will make it worse. Every assembly programmer can't live without the Intel Software Developer Manuals. When those "beginners" will refer to the Intel Manual (or any other source which gives information about x86 instructions) they'll become confused because someone "renamed" all the instructions as it will help in the learning process.
Assembler is human-readable machine code, not English, let's keep it that way.

SpiderTL, please don't get offended by my reply. If you think you can make assembly easier this way, go for it. Those above lines are totally my opinion and I've no rights to force your thoughts to be similar too.

If you're writing an IDE why not use tooltips? When the programmer will hover the cursor over an instruction, a small tooltip will appear showing what equivalent of that instruction will be in your <XML/> language.

[SpyderTL]

Yep. But you still have the problem that MOV will actually be converted into one of about 20 different instructions (single byte and multi-byte) depending on the situation. Your XSLT would have to be awfully smart in order to pick the right one. I'd rather just specify exactly what I want the CPU to do, as a developer.

Code: Select all

mov ax, bx
mov ax,  word[bx]
mov ax, 20
mov ax, word [20]
mov word [20], ax
.......

No idea what the problem is to differentiate between this.

The problem is that I have a list of opcodes that the CPU understands, and I need to give each of them a different name (or I have to write some seriously smart XSLT to figure out which opcode to use based on the operand size and address mode, etc.)

The x86 instruction set is huge, plus the "verbosity" of <XML/> and those assembler "sentences" will make it worse. Every assembly programmer can't live without the Intel Software Developer Manuals. When those "beginners" will refer to the Intel Manual (or any other source which gives information about x86 instructions) they'll become confused because someone "renamed" all the instructions as it will help in the learning process.
Assembler is human-readable machine code, not English, let's keep it that way.

Like I said above, I didn't set out to "rewrite" Assembler in XML. I wanted a way to write low-level code, but I wanted a modern tool set that would actually help me develop software, like C# (Visual Studio) and Java (Eclipse) do. I could not find any tools that would let me write ASM code and that provided any sort of modern development assistance (intellisense, context sensitive help, etc.) So I wrote my own.

SpiderTL, please don't get offended by my reply.

If you're writing an IDE why not use tooltips? When the programmer will hover the cursor over an instruction, a small tooltip will appear showing what equivalent of that instruction will be in your <XML/> language.

I'm not writing an IDE. There are plenty out there that are very good at editing XML files. If I can use Visual Studio, or Eclipse, or XCode, or XmlPad, and they already have intellisense, autocomplete, tooltips and can transform XML files using XSLT, why not just use one of them?

The whole XML idea is based on the idea that code is actually data, and XML happens to be really good at storing data. And XSLT just happens to be really good at "transforming" a single XML node into one or more different XML nodes, which just happens to be really useful when you want to "compile" a program into a bunch of byte codes. So, essentially what I did was give every cpu instruction (byte code) a name, and created an XLST file that would replace any references to that name with the underlying byte code. Once I got that working, then I started creating more XSLT files that would replace a function name with the cpu instructions for that function, and so on, until I ended up with a very simple operating system.