Which language is best for Operating System Development?

Question about which tools to use, bugs, the best way to implement a function, etc should go here. Don't forget to see if your question is answered in the wiki first! When in doubt post here.
User avatar
Solar
Member
Member
Posts: 7615
Joined: Thu Nov 16, 2006 12:01 pm
Location: Germany
Contact:

Re: Which language is best for Operating System Development?

Post by Solar »

rdos wrote:
Solar wrote:I'd really like to see a non-trivial, non-biased piece of your code benchmarked against compiled C/C++. The proof is in the eating, you know...
Not easy to do. First, most compilers would lose on walk-over as they cannot generate segmented code, and secondly, those that could would lose because they cannot handle minimizing segment register reloads, and optimize segment register usage. [...] And comparing flat-memory model code with segmented code is not valid.
That is what I meant with "non-biased". You are basing your claims of your ASM code being superior firmly on the memory model... which conveniently avoids the issue of real-life benchmarks against modern compilers.

You also avoid several side-subjects like amount and quality of available documentation, amount and quality of available developers, self-documentation and maintainability of code, functionality implemented per hour of effort, ...

I don't want to draw offensive conclusions from this, because I do respect your achievements with RDOS, and ASM is a viable design decision, especially in the embedded field you've been targeting with RDOS. But you do see where your position regarding the superiority of ASM in general has a vulnerable flank or two?

That's why I cannot let statements like "ASM is best, of course" pass unchallenged.
Every good solution is obvious once you've found it.
User avatar
Combuster
Member
Member
Posts: 9301
Joined: Wed Oct 18, 2006 3:45 am
Libera.chat IRC: [com]buster
Location: On the balcony, where I can actually keep 1½m distance
Contact:

Re: Which language is best for Operating System Development?

Post by Combuster »

You too, young padawan, must learn that if you see no disadvantages in a subject, you lack reason or knowledge.
"Certainly avoid yourself. He is a newbie and might not realize it. You'll hate his code deeply a few years down the road." - Sortie
[ My OS ] [ VDisk/SFS ]
cxzuk
Member
Member
Posts: 164
Joined: Mon Dec 21, 2009 6:03 pm

Re: Which language is best for Operating System Development?

Post by cxzuk »

Solar wrote:ridiculous" section, like JavaScript or XML.
Erm! My Kernel is written in XML!

Well.. i say written in, XML is just the abstraction tree when compared to a normal traditional compiler/language.

It is not ridiculous at all, and the choice was very justified after a good few months of trying other languages. :)

Why XML? - I wanted a setup that i had -complete- control. Calling Convention, Optimisations, etc. I didnt want to be limited by a languages assumption on something. Most people would use ASM, which i started to use. But ASM has some very clear problems that i wanted to avoid.

ASM code can easily grow out of control, Abstraction is limited. ASM can easily have bugs due to such low instructions and repeatative nature (Syntactic sugar?). And finally the biggest problem for me was that what im typing is not a clear description (annotation) of what I want it to do.

Highly structured, and alot of already existing tools, XML was a great choice. Super flexable, and being machine-readable I can completely automate optimisations (like normal compilers), but as I have a summary of possible bugs as im typing :) e.g. bounds checking (Overflow, Carry).

I have an XSLT sheet which changes the XML into ASM, optimisations are kept in seperate XSLT files. It was really simple to do, and has saved alot of time. I may invest some time into a language to go onto of it, or borrow something like srcML (C to XML).

Mike Brown
User avatar
qw
Member
Member
Posts: 792
Joined: Mon Jan 26, 2009 2:48 am

Re: Which language is best for Operating System Development?

Post by qw »

cxzuk wrote:Erm! My Kernel is written in XML!
Sure makes me curious!
User avatar
JamesM
Member
Member
Posts: 2935
Joined: Tue Jul 10, 2007 5:27 am
Location: York, United Kingdom
Contact:

Re: Which language is best for Operating System Development?

Post by JamesM »

rdos wrote:Of course. Vectorising or not, C-compilers simple cannot handle global optimizations (like automatic register allocation between functions) very well, and therefore will suck at function-call performance. And even if they have an automatic register allocation scheme, these are bound to be less than optimal, requiring shuffling things between memory/stack/registers. I clearly can see the difference (in performance) between my old call interface that used stack for parameters, and needed an intermediate step, with the new fine-tuned interface that tells the compiler exactly which registers to pass parameters in. Additionally, C compilers also suck when there is more than one out-parameter, as these need to be passed as pointers. With an assembly-interface, they would be passed in registers.
Function inlining and inter-prodedural optimisations are well known and well understood. It gets more difficult when you get to inter-compilation-unit optimisations but LTCG (as exhibited by LLVM and GCC) has this sorted too.

Calling convention optimisation is certainly possible on an intra-compilation-unit basis (especially when calling a function with internal linkage with callsites that are only in the current function).
Another issue is that optimization works best on long procedures, but it is generally good programming practise to keep procedures short, which leads to less local optimizations.
Here we come to the crux of the matter - You will not, in the general case, outdo a compiler because you must write maintainable ASM code. You need to be able to read it, you need to be able to change it. There are many optimisations that just make no logical sense - no sane programmer would think of doing something in that particular way (a good example is the C library expert at work found he could shave a few instructions off a highly critical c library integer math function by XORing the two operands first - something that supercompilation had told him that was completely counterintuitive).

Compilers don't need to worry about producing short procedures. They will inline aggressively, especially in hot loops, to reduce the number of calls and optimise register usage.

There are certain cases where a human will beat a compiler. These usually involve very hot loops of just a few instructions - codecs (and microbenchmarks, ugh) and the like. This is not the general case.

Compilers can deal with exception handling better. They can use zero-cost exception semantics to generate unwind tables which keep error-handling code out of the hot path. Human assembler programmers cannot realistically do this - it would require too much effort to maintain.

I've reached the crux here: Assembly has its place, as have higher level languages. Assembly has more of a niche than HLLs, quite understandably - you won't find me bashing assembly as something that is completely pointless (unlike you vs. HLLs). But seriously, saying that assembly is the only way forward is just bloody idiotic.

I honestly don't know who your employer was, but how you managed to convince them that writing a payment system with a from-scratch OS using untestable, unverifiable and error prone assembler language was a wise idea, I have no idea. And I hope to hell that your company never gets audited.
rdos
Member
Member
Posts: 3306
Joined: Wed Oct 01, 2008 1:55 pm

Re: Which language is best for Operating System Development?

Post by rdos »

Solar wrote:I don't want to draw offensive conclusions from this, because I do respect your achievements with RDOS, and ASM is a viable design decision, especially in the embedded field you've been targeting with RDOS. But you do see where your position regarding the superiority of ASM in general has a vulnerable flank or two?
I know it has, and that is why I'm working on the C/C++ kernel header-files, as well as making the API 32-bit. I've just finalized the porting of all kernel API functions to be usable from a 32-bit compact memory model, which is a big step in the direction of using HLLs in device-drivers. I'll soon restart the work of providing the clib, which I had to abort before due to how the kernel API looked like. Then I'll use Intel's ACPI code as a first test suite for the new possibilities. That certainly does not mean that I will rewrite anything critical like scheduler, interrupt system or similar to use C. But I do intend to provide C++ base-classes for common device-classes (for instance file systems, audio-drivers, network-drivers, printer-drivers) which would make some projects more feasible in the future.
rdos
Member
Member
Posts: 3306
Joined: Wed Oct 01, 2008 1:55 pm

Re: Which language is best for Operating System Development?

Post by rdos »

JamesM wrote:Function inlining and inter-prodedural optimisations are well known and well understood. It gets more difficult when you get to inter-compilation-unit optimisations but LTCG (as exhibited by LLVM and GCC) has this sorted too.

Calling convention optimisation is certainly possible on an intra-compilation-unit basis (especially when calling a function with internal linkage with callsites that are only in the current function).
I don't see how that solves any of the optimization issues I brought up before. When I write HLL code, I use C++, and I at least have a base-class (with a lot of virtual methods), and derived classes. Most methods are short, typically less than 10 lines of code. The compiler cannot inline base-class methods, nor can it normally inline methods in the class itself because they are virtual. A typical optimizer cannot do much on code that on average contains 10 lines of code per method. Your optimizations might be very effective on large numeric projects, but I doubt they are effective on largely event-driven code. And an OS-kernel (at least mine do) looks more like an event-driven application than a large numeric application.
JamesM wrote:Compilers don't need to worry about producing short procedures. They will inline aggressively, especially in hot loops, to reduce the number of calls and optimise register usage.
You misunderstood. I meant it was good SOFTWARE coding practise to use many small procedures rather than a few large. This might be more important for assembly, but it is just as desirable for HLLs in order to produce code that can be maintained.
JamesM wrote:Compilers can deal with exception handling better. They can use zero-cost exception semantics to generate unwind tables which keep error-handling code out of the hot path. Human assembler programmers cannot realistically do this - it would require too much effort to maintain.
CY flag is a pretty good solution for error-handling in assembly. It doesn't lead to hard-to-maintain code, and it is quite readable if done in a consistent manner.

And centralized error-handling has no place in standalone applications that cannot rely on end-users to click-down error message boxes. This is why Windows and Linux are such poor solutions for those applications. Error handling in RDOS, as well as in our payment application, is localized. Errors are handled & logged where they occur. Faults, for instance, rather than starting an exception chain, either goes to the debugger, or are written to CF-disc, and after the system has restarted, are retrieved & logged. Faults thus are handled by the OS, and requires no intervention from the application. Faults in the OS kernel are handled in the same way (no blue screens, no panic screens).
User avatar
Solar
Member
Member
Posts: 7615
Joined: Thu Nov 16, 2006 12:01 pm
Location: Germany
Contact:

Re: Which language is best for Operating System Development?

Post by Solar »

rdos wrote:When I write HLL code, I use C++, and I at least have a base-class (with a lot of virtual methods), and derived classes.
When I write C++, I have derived classes overriding virtual methods when I need them, because other concepts would result in less efficient and / or maintainable code.

Please, do compare your smart assembly against smart C++, not against run-off-the-mill code-monkey OOP mush. C++ isn't Java or C#, where everything has to be derived from Object and you need a class just to put your main() method in.
A typical optimizer cannot do much on code that on average contains 10 lines of code per method. Your optimizations might be very effective on large numeric projects, but I doubt they are effective on largely event-driven code. And an OS-kernel (at least mine do) looks more like an event-driven application than a large numeric application.
Hearsay and assumptions. There is only one rule to optimizing: Measure. Optimize. Measure.
And centralized error-handling has no place in standalone applications that cannot rely on end-users to click-down error message boxes.
What has this to do with anything? Sorry, but where did you get the idea that exception handling has anything to do with "click-down error message boxes" or "blue screens"?
Every good solution is obvious once you've found it.
User avatar
JamesM
Member
Member
Posts: 2935
Joined: Tue Jul 10, 2007 5:27 am
Location: York, United Kingdom
Contact:

Re: Which language is best for Operating System Development?

Post by JamesM »

You misunderstood. I meant it was good SOFTWARE coding practise to use many small procedures rather than a few large. This might be more important for assembly, but it is just as desirable for HLLs in order to produce code that can be maintained.
Actually you misunderstood me - what I meant was that while the programmer can write many, small procedures in his HLL of choice (C, for example), the compiler don't have to worry about maintainability of the resulting assembler so can generate long, monolithic functions (by inlining) if that is more efficient.

You as an assembler-level programmer would not do this as it would be completely unmaintainable. That was my point.
cxzuk
Member
Member
Posts: 164
Joined: Mon Dec 21, 2009 6:03 pm

Re: Which language is best for Operating System Development?

Post by cxzuk »

Hobbes wrote:
cxzuk wrote:Erm! My Kernel is written in XML!
Sure makes me curious!
What would you like to know? :) My Operating System is not very orthodox, and i really wanted to have a go at every aspect to learn, experiment etc.

I use XML basically as an Abstract Syntax Tree (http://en.wikipedia.org/wiki/Abstract_syntax_tree).
I dont believe there is a language thats best for Operating System Development. So i cut the language out completely and just write an AST directly. Gives me complete control over the assembly produced :)

Mike Brown.
User avatar
qw
Member
Member
Posts: 792
Joined: Mon Jan 26, 2009 2:48 am

Re: Which language is best for Operating System Development?

Post by qw »

cxzuk wrote:What would you like to know?
I was wondering what your XML and XSLT scripts look like. Thinking of it, I realized that XML is very fit to represent an Abstract Syntax Tree because it is a tree by nature.

Just for the fun of it, I translated the example on Wikipedia to XML:

Code: Select all

<statement-sequence>
  <while>
    <!-- condition -->
    <compare op="!=">
      <variable name="b" />
      <constant value="0" />
    </compare>

    <!-- body -->
    <branch>
      <!-- condition -->
      <compare op=">">
        <variable name="a" />
        <variable name="b" />
      </compare>

      <!-- if-body -->
      <assign>
        <variable name="a" />
        <bin-op op="-">
          <variable name="a" />
          <variable name="b" />
        </bin-op>
      </assign>

      <!-- else-body -->
      <assign>
        <variable name="b" />
        <bin-op op="-">
          <variable name="b" />
          <variable name="a" />
        </bin-op>
      </assign>
    </branch>
  </while>

  <return>
    <variable name="a" />
  </return>
</statement-sequence>
User avatar
Combuster
Member
Member
Posts: 9301
Joined: Wed Oct 18, 2006 3:45 am
Libera.chat IRC: [com]buster
Location: On the balcony, where I can actually keep 1½m distance
Contact:

Re: Which language is best for Operating System Development?

Post by Combuster »

You should not have asked :wink:
"Certainly avoid yourself. He is a newbie and might not realize it. You'll hate his code deeply a few years down the road." - Sortie
[ My OS ] [ VDisk/SFS ]
User avatar
qw
Member
Member
Posts: 792
Joined: Mon Jan 26, 2009 2:48 am

Re: Which language is best for Operating System Development?

Post by qw »

Combuster wrote:You should not have asked :wink:
I admit that the same code in C is a lot shorter, more readable and easier to maintain, but a kernel in XML is fascinating. I am very curious what the XSLT scripts look like.
User avatar
Rusky
Member
Member
Posts: 792
Joined: Wed Jan 06, 2010 7:07 pm

Re: Which language is best for Operating System Development?

Post by Rusky »

That looks awfully close to Lisp macros. :) If there were a byte code/machine code library in Lisp (maybe LLVM bindings?) you could do something like that much less verbosely, although at that point you may as well just use a regular Lisp compiler.
h0bby1
Member
Member
Posts: 240
Joined: Wed Aug 21, 2013 7:08 am

Re: Which language is best for Operating System Development?

Post by h0bby1 »

rdos wrote:
JamesM wrote:
rdos wrote:Assembly is best, of course. Some less important and complex drivers could be in C/C++, but nothing critical should be.
Yes, because you're better than an optimizing and auto-vectorising compiler.

****.
Of course. Vectorising or not, C-compilers simple cannot handle global optimizations (like automatic register allocation between functions) very well, and therefore will suck at function-call performance. And even if they have an automatic register allocation scheme, these are bound to be less than optimal, requiring shuffling things between memory/stack/registers. I clearly can see the difference (in performance) between my old call interface that used stack for parameters, and needed an intermediate step, with the new fine-tuned interface that tells the compiler exactly which registers to pass parameters in. Additionally, C compilers also suck when there is more than one out-parameter, as these need to be passed as pointers. With an assembly-interface, they would be passed in registers.

Not to speak of the fact that C compilers really suck at handling segmented memory-models.

EDIT: Forgot to mention that C cannot handle using the CY flag to signal return status. This is clearly superior to using a whole return-register for status (especially when combined with ordinary results which then needs to be passed by reference).

These problems are built-into C, and thus cannot be solved with any kind of optimizations available.
compiler nowdays have options to do 'global program optimization', and can optimize the code between function call, even between different files, and you can also mannage to pass all parameters as a structure pointer, it still can polute the cache and add slight overhead to the code, but it's not dramatic either to push a pointer on the stack, and the number of registers is also limited on 32 bit x86 architecture, i think i have read that there is a new calling convention for x64 that make use of registers to pass parameters in a standard way because x64 have more registers, but then yay for recursive function =)

for return status, you could just make a macro to make the return that setup some bit flag in the return value to indicate the status and store the return value in lower bits, it would not allow for a full 32 bit return type, it would not work if it has to return a pointer, unless it is done as in most hardware to require a certain alignement from pointer and use the lower bits as flags, or could use a global variable to store some status even if that's rather ugly, or defining some kind of execution context in which function can store some informations, anyway it would really matter if there is lot of function call to be made, and in that case some other way can be thought off to pass parameters and handle status

there is a good series of book that cover the whole topic very in depth, it's called 'how to optimize software in c++', there are also version about C and assembler, they are very well made and explain the whole problematic

the way i see it, to really optimize asm nowdays is very different than it used to be at the time of DOS before the pentium architecture, because now, there are pipelined execution, and to really optimize the asm code correctly, you'd need to know how the instruction will be broke down into micro opcode and how that would fit into the different pipelines, there is the whole thing of throughput/latency to mannage, to see what kind of instruction will create latency with the next one, and to really understand the logic, it need to understand how it's broke down into micro opcode and handled by the different piepline, which can be very dependent on the type/brand/generation of cpu, and very hard to optimize manually for each architecture

instruction ordering, and the way branching is done, and good handling of cacheline could have more impact on the performance than traditional way of 'linear' optimisation as counting clock cycle instruction/instruction, and regarding the whole number of different cpu architecture around, between intel/amd, and the whole different generation of cpus, it would make it very hard to have generic asm code that are optimized for all architectures

when you look at what can be done with intel compiler and kernel math lib and openMP, with whole code path detected at runtime and optimized specifically for the specific cpu it is executed on, it make assembler speed optimization pretty useless, and with the intrinsec functions ( <xmm.h>), even sse and simd can be handled in pure C , with good management of inlining and global program optimization, it can give very good result

even at a scale as large as an OS, global optimization can be better achieved with good data organisation, good scheduling, and better handling of whole lot of feature and asynchronous things at high level rather than on saving a stack access to make a function call, i mean there is no little profits as they say, but asm can also prevent to have a good large scale mangement of all ressource in a way that is easy to debug and maintain, and it can prevent to implement as well high level algorythm that can do more to improve global performance than the little thing you can gain optimizing function calls, and there can be many way to optimize code to avoid to have to pass many parameters many times to a function within a loop
Last edited by h0bby1 on Sat Aug 31, 2013 1:05 am, edited 1 time in total.
Locked