OSDev.org

embryo wrote:

SpyderTL wrote:

embryo wrote:For a beginner without any assembly background it is a bit viable. But for an any more experienced programmer it is a pain.

I think the same could be said of any programming language.

No. There is no useful language without examples and documentation. It means you are denying all the documentation and examples (in assembly) for a low level developer in XML.

I'm working on the documentation. My real-life job that pays all of the bills cuts into my OS dev time.

And I have a bunch of example code. It's in CodePlex. As a matter of fact, I have one huge example that shows how to make a simple operating system with objects and a shell. The shell even has Intellisense and Autocomplete.

embryo wrote:

SpyderTL wrote:I could not find any tools that would let me write ASM code and that provided any sort of modern development assistance (intellisense, context sensitive help, etc.) So I wrote my own.

It's a bit of advertising for me, but you can use jEmbryoAssembler project to get all those "Autocomplete, Intellisense, Documentation, View Definition, Inline Functions...". But it is in Java. However, you can rewrite it in C#, it's really easy. And one more point - the names there are pure assembly without any complains about some complexity of translation.

Great! Where were you 5 years ago? You could have saved me a lot of trouble.

Seriously, though, if we had had this conversation 5 years ago, I probably would not have bothered going down the XML road at all. I'm still glad I did, cause I've learned a ton. But you are farther along than I am, and I've already offered to help out wherever possible. Just let me know what you need.

Brendan wrote:Let's look at all the use cases - for any kind of source file format:

• A compiler has to read it and check it (e.g. first few steps of compiling). For this case XML is worse than plain text, and plain text is worse than other alternatives.
• It has to be stored on disk, transferred between computers, etc. For this case XML is worse than plain text, and plain text is worse than other alternatives.
• It has to be displayed to humans (who don't read "binary bytes", regardless of whether those binary bytes represent characters or not), and entered/edited by humans; typically in a kind of "feedback loop" (e.g. press some keys, see the changes, repeat until it looks like what you want). For this case XML is worse than plain text, and plain text is worse than other alternatives.
• It needs to be read and understood by miscellaneous tools (e.g. source level debugger, possibly tools to extract documentation from the source, revision control systems, etc). For these cases XML is worse than plain text, and plain text is worse than other alternatives.

I don't suppose you would mind sharing one or two of these "alternatives"?

MessiahAndrw wrote:Working with big data and people that have never used XML before, XML is a) very hard to get your head around if you've never encountered with it before, and b) takes up a lot of storage (and compression is often out of the question because throughput performance is a requirement.)

I've often found myself in situations having to walk people through the XML syntax and layout (namespaces, tags, properties, children) and explain how XSL and XSD files work.

I was showing my 6 year old daughter how to write HTML to display text and change the color. She didn't seem to have too much trouble.

But seriously, coming up with a computer programming language that is a) powerful enough to write an operating system, and b) user friendly enough to where someone who has never seen it before can understand it is going to be pretty difficult to pull off. Give it a shot, though.

MessiahAndrw wrote:
vs:

I can't help but notice that one of those has Intellisense and Tooltips, and the other does not.

MessiahAndrw wrote:Also, I haven't found a good XML IDE yet.

Have you found a good JSON IDE yet? Or YAML?

Hi,

SpyderTL wrote:

Brendan wrote:Let's look at all the use cases - for any kind of source file format:

• A compiler has to read it and check it (e.g. first few steps of compiling). For this case XML is worse than plain text, and plain text is worse than other alternatives.
• It has to be stored on disk, transferred between computers, etc. For this case XML is worse than plain text, and plain text is worse than other alternatives.
• It has to be displayed to humans (who don't read "binary bytes", regardless of whether those binary bytes represent characters or not), and entered/edited by humans; typically in a kind of "feedback loop" (e.g. press some keys, see the changes, repeat until it looks like what you want). For this case XML is worse than plain text, and plain text is worse than other alternatives.
• It needs to be read and understood by miscellaneous tools (e.g. source level debugger, possibly tools to extract documentation from the source, revision control systems, etc). For these cases XML is worse than plain text, and plain text is worse than other alternatives.

I don't suppose you would mind sharing one or two of these "alternatives"?

There's a virtually unlimited number of alternatives.

For an example, for your own language, you've got a fixed number of valid identifiers (e.g. "CopySIAddressToAXAndIncrementSI"). If you enumerated these identifiers it's extremely likely that you could encode it as a 16-bit integer instead, which would avoid the need for string comparisons and probably reduce file sizes by an order of magnitude.

For another example, for my own source file format, lines of source code are stored as tokens. Each token has a "token ID" and an (optional) "token data or length" value that are combined to form a 26-bit integer, and then that 26-bit integer is stored as a (variable length) sequence of 1 to 4 bytes (where the most frequently used tokens use values that are encoded in the least number of bytes). For "token IDs" within a certain range, this is followed by "length" raw bytes of data (mostly for things like string literals and comments).

Note that this applies to every possible use of XML and plain text (and isn't limited to programming language source code at all). There is *always* a more efficient "raw binary" representation that is superior.


Cheers,

Brendan

Brendan wrote:For another example, for my own source file format, lines of source code are stored as tokens. Each token has a "token ID" and an (optional) "token data or length" value that are combined to form a 26-bit integer, and then that 26-bit integer is stored as a (variable length) sequence of 1 to 4 bytes (where the most frequently used tokens use values that are encoded in the least number of bytes). For "token IDs" within a certain range, this is followed by "length" raw bytes of data (mostly for things like string literals and comments).

Note that this applies to every possible use of XML and plain text (and isn't limited to programming language source code at all). There is *always* a more efficient "raw binary" representation that is superior.

So you wrote an IDE that reads and writes these codes?

Wait a sec. You would have run into this same problem with the embryo assembler. How did you name your java/native platform instruction methods? X86.MOV(source, destination)?

Hi,

SpyderTL wrote:

Brendan wrote:Intellisense, autocomplete and tooltips are only minor conveniences;

Whoa! That's quite a statement.

I'm guessing you haven't actually written any assembly language before. Do you honestly think that (e.g.) auto-complete or tool-tips are helpful for something like "lodsb" for any assembly language programmer that isn't a complete beginner?

SpyderTL wrote:

Brendan wrote:while hideously unusable syntax is an extreme inconvenience. The advantages don't justify the disadvantages in the slightest.

I'm, obviously, going to go ahead and, sort of disagree with you there...

<sentence><word>You</word><word>can</word><word>think</word><word>what</word><word>you</word><word>like</word><exclamation/></sentence>

Seriously; why not post an example of your "XML assembly" code somewhere (like the #asm IRC channel on freenode, or alt.lang.asm newsgroup, or the FASM forums), and ask actual assembly language programmers to comment on the syntax? I'm sure that you'll get important feedback.

SpyderTL wrote:

Brendan wrote:By storing source code in a binary form (e.g. as "tokens") you don't need a scanner and can improve the efficiency of both IDE and compiler. In addition, (at least in my experience) this reduces source code file sizes to about 80% of the "plain text" equivelent.

Yes, byte code is physically smaller than ASM text files. And .NET IL is physically smaller than C# text files. Binary files are smaller than text files. But no one uses a keyboard to type in binary files. You need a significant user interface in order to "translate" the binary data into something that humans can easily manipulate. Significant user interfaces were pretty difficult to come by in the 50's.

Yes; so instead of doing it right (and implementing a suitable editor), they were lazy.

Note that this isn't technically accurate. For example, in the late 1970s and early 1980s there plenty of systems that encoded text into tokens as you entered it and converted tokens back into text for display (mostly for both performance and space reasons).

SpyderTL wrote:

Brendan wrote:Basically, plain text is bad. In 1960 someone was too lazy to define a proper source file format and provide a suitable IDE

That may have been due to the fact that you were limited to about 2-4KB of memory...

The reverse actually - when there's severe limits on RAM and CPU speed you want to do everything you can to avoid wasting RAM and wasting CPU time.

SpyderTL wrote:

Brendan wrote: - it amazes me that programmers can't do the same for their own tools.

Same here. That's kind of why I decided to go back and revisit all of those design decisions. But I'll admit that I've tried to come up with a non-text development strategy, and it turns out that it's not quite as easy as you would think. Did you have something specific in mind as a user interface?

The user interface is just an interface - it doesn't effect the source file format, and it's the source file format that I'm mostly talking about.

However, I only have preliminary ideas for the IDE's user interface. The first version of my IDE will be similar to a traditional IDE (e.g. user enters text and sees text, without necessarily knowing or caring that the IDE is converting to/from tokens while they're typing), but probably with a little "source code style" configuration option (so different people can set their IDE to "native style" or "C-like style" or "no braces Python-like style" or whatever they want). Then I'll add features to extract information from the source code and generate diagrams (e.g. dependency diagrams, etc) to make it easier for people to "see the forest amongst the trees" and to aid navigating through the project. After that I'd probably add a "drag and drop" style to it (mostly for children and/or touchpad support).

Beyond that, I don't know. Ideally, I want people to be able to manipulate source code in some sort of 3D environment; possibly where the CPU/s are represented as machines that follow a track, and things like branches, loops and function calls are just track layout, and expressions/statements are devices placed on the track.

SpyderTL wrote:

Brendan wrote:It's less human readable for people

... than what? 3 character instruction mnemonics?

Yes.

To improve "<cpu:CopySIAddressToAXAndIncrementSI/>" you could maybe use an easier to remember and easier to type acronym (like "cpSI2AXincSI"); then realise that the "cpu:" part is worthless hassle, and that encasing it in angle brackets is for your convenience not the end users and should be removed. The end result of all these improvements would be "cpSI2AXincSI", but then an even shorter mnemonic like "lodsb" would be even better (even if you ignore the fact that "lodsb" happens to match all the existing documentation everywhere, or that all 80x86 assembly language programmers are already familiar with it). If you actually need a huge wanky paragraph, then add it as a tool-tip.

SpyderTL wrote:

Brendan wrote:, and less efficient (more expensive for scanning and parsing, and larger file sizes) for computers.

Yeah, at build time. Who cares if builds take 500 ms longer?

Um, what? You realise 500 ms is half a second?

For my current project; a script builds my own "build utility", then it checks everything and regenerates my any web pages that changed (docs, etc), builds any utilities that changed, uses those utilities to compile anything that needs to be recompiled, and generates a full backup (and manages my backup directory); and this typically takes less than 10 ms. Absolute worst case (everything fully rebuilt entirely from scratch) currently takes 820 ms.

This is important. It means I can press F12 (a keyboard shortcut to build everything) and have everything done very quickly with no silly distractions breaking my train of thought. It means that my "edit, build, edit, build, edit, build, test" cycle is as productive as possible.

SpyderTL wrote:

Brendan wrote:The only "benefit" that XML has is that it allows the developer to sacrifice quality to reduce development time. Basically, XML/XSLT are tools that allow lazy/incompetent people to create crap.

Developers are perfectly capable of writing crap code in any language that I'm aware of. Preventing that at the language level would be pretty impressive.

Yes, but the opposite isn't true - it's impossible to write good code in some languages. Basically, it doesn't matter how good you are as a programmer, if you use XML/XSLT the end result will be crap.

SpyderTL wrote:I've already listed numerous "benefits" to using XML in this thread. You'll have to go back and find them... But suffice to say that your statement that XML only has one benefit (reduce development time) has not convinced me.

Don't worry - I understand that you wouldn't want to create a concise list showing that all of the "benefits" are for the developer and not for the end user.


Cheers,

Brendan

Hi,

SpyderTL wrote:

Brendan wrote:For another example, for my own source file format, lines of source code are stored as tokens. Each token has a "token ID" and an (optional) "token data or length" value that are combined to form a 26-bit integer, and then that 26-bit integer is stored as a (variable length) sequence of 1 to 4 bytes (where the most frequently used tokens use values that are encoded in the least number of bytes). For "token IDs" within a certain range, this is followed by "length" raw bytes of data (mostly for things like string literals and comments).

Note that this applies to every possible use of XML and plain text (and isn't limited to programming language source code at all). There is *always* a more efficient "raw binary" representation that is superior.

So you wrote an IDE that reads and writes these codes?

No. The plan is to write temporary/crappy tools that work on Linux; then use those tools to write a (minimal) OS, then write native compilers and the IDE for that OS.

Of course I do care about the quality of the end result, because the end result is the entire point of it all. This means that if I'm not happy with something (e.g. I think I can do better) I stop what I'm doing and redesign it. So far everything has been redesigned at least twice since I started designing the languages and toolchain.


Cheers,

Brendan

SpyderTL wrote:But seriously, coming up with a computer programming language that is a) powerful enough to write an operating system, and b) user friendly enough to where someone who has never seen it before can understand it is going to be pretty difficult to pull off. Give it a shot, though.

The difficulty of operating system development is not learning the programming language (once you get beyond the bootloader there are tutorials on the Wiki for using Ada, C, C++, D, FreeBasic, and Pascal), but designing the system, the algorithms, the data structures - solving the problems of process management, scheduling, device IO, memory management, security, file access, implementing a shell.

Do you think those algorithms will be easier to understand in your language, compared to say, implementing it in C or FreeBasic?

This brings me to:

SpyderTL wrote:I was showing my 6 year old daughter how to write HTML to display text and change the color. She didn't seem to have too much trouble.

I'm sure your daughter could figure out what "textbox.foregroundColor = Color.Red" does in C#, but can she write she write a complex application in C#?

Likewise, she may understand a few blocks of XML, but that doesn't mean she can wrap her head around the complexities of a large system like an OS, even if it is written in an XML.

SpyderTL wrote:

MessiahAndrw wrote:Also, I haven't found a good XML IDE yet.

Have you found a good JSON IDE yet? Or YAML?

The problem with typing XML is having to close tags, and your tags seem very verbose. With other languages you can simply type ],}, or ) and move on - making a specialized editor irrelevant, and XML editors that close tags for you are frustrating because you still need to move the cursor a lot to jump over the closing tag if you want to type code.

BTW, VS2013 supports JSON Schemas.

Hi,

MessiahAndrw wrote:

SpyderTL wrote:I was showing my 6 year old daughter how to write HTML to display text and change the color. She didn't seem to have too much trouble.

I'm sure your daughter could figure out what "textbox.foregroundColor = Color.Red" does in C#, but can she write she write a complex application in C#?

Likewise, she may understand a few blocks of XML, but that doesn't mean she can wrap her head around the complexities of a large system like an OS, even if it is written in an XML.

For small children, you really should be looking at something like Etoys:




Cheers,

Brendan

Brendan wrote:I wasn't really talking about visualisation.

You have defined following points:
1. Validation and compilation speed.
2. Storage/transfer speed/efficiency.
3. Display to humans.
4. Tool friendliness (format recognition).

The third point is about visualization. But there are 3 more points.

The first point is about taking user input and processing it in some way. It's another part of the visualization problem - the visualization predefines user actions and forms action recognition processing. But the processing part, of course, is still here and still important. However, if most of the processing is predefined by the visualization approach, then it is legal to speak about visualization first.

The second point is about machine friendly efficiency. I agree, that shorter data formats are better suited for such purpose. Also I agree, that a parser friendly format leads to a greater performance. And I agree, that if we have the user friendliness as a main goal in mind when we develop some system, then the benefits above should stop us when we try to think about developer problems. But we have one problem here, as it was mentioned in the osdev.wiki, you shouldn't do it unless you have many lives to spend. It is really serious problem. May be it is possible to organize our society in a user friendly manner and really stop thinking about complexity and time spent, but, unfortunately, the society is in no way close to such imaginary form. It encourages speed and greed, but not the waste of time for something not promising quick return on investment. It means while we have no another society - most of us should work quickly and forget about long term goals of quality and beauty. But yes, it contradicts with the osdeving "for fun"...

At least, it is better to understand the people, who is looking for speed.

And now the point four - it's just a matter of a format standard acceptation, if it has a great user base, then there will be a lot of tools. Are the tools efficient or not - is the problem I have mentioned above.

Brendan wrote:If you are the person creating the tools and only care about how long it takes (and hate the potential users of your tools so much that you don't care how much they suffer), then XML isn't "worse" at all.

You've got it - most of the people care about "how long it takes" (what is the cost of).

SpyderTL wrote:So, don't create a new programming language, because Assembly is so well documented?

OK, we are talking about new language. But then we should list it's advantages. What the advantage list can look like? I hope it's not about intellisense only?

SpyderTL wrote:

embryo wrote:So, you trade development speed for quality. Do you want to produce low quality OS?

...
Quality has nothing to do with the tools you use. Or, more specifically, good tools don't necessarily produce good products.

Here we talk about efforts that are required to simplify the machine code and represent it in some form with higher abstraction level. Your approach maps machine code directly to some XML tags, but it is very low abstraction level. And what is the importance of the level of abstraction you can see in any contemporary high level language.

But I'll be happy with explanation like this - I just do not want to spend my time on the tricky XSLT.

SpyderTL wrote:I assure you, if I could figure out a way to get rid of C# altogether, I would. But I don't know of any other way to write out bytes from XML.

But I see a lot of C# files in your project. Are all of them needed just to translate "ABC" to bytes 65, 66, 67?

SpyderTL wrote:Also, I don't feel like running all of the XSLT's by hand, every time I make a change to a single file.

But you have a lot of tools for it. Just let a tool to do it's job.

And, of course, it's just a suggestion and if there is no time to implement it, then I'm out of luck