Rant: Random thoughts on Direction, Purpose, and other Fun T

[soap]

Progress. Kinda. Nine months - and many complete rewrites - after I started I have a [very] primitive yet working operating system. Yay. Go me. Parades, tickertape, etc.

Now I've stopped looking at what I'm implementing, and started looking at *how* I'm implementing it...

See, my OS uses a homebrew IPC design, dedicated to specifying and calling objects. I've spent a lot of time implementing, testing, and redesigning 'till it's reached a point where I'm happy with it.

At the moment, I'm writing lots of wrapper code so my C programs can access and define objects in this system. This is ugly and slow to write. I was thinking it would make more sense to switch to an object-orientated language (C++). Then I took that a step further - wouldn't it be nice if I had a language with native support for manipulating objects in my system? While I was at it, I started building up a wishlist of features I've seen and liked in various programming languages that I'd like to add to mine.

I formed an incremental plan of attack.Write an assembler. Design and write a Forth interpreter. Add a compiler. Then attempt to rewrite and extend the Forth implementation to support the syntax and all the features of my dream language.

Uh oh. Focus shift.

Suddenly I've gone from designing and implementing an operating system to designing and implementing a programming language for my operating system.

This is not necessarily a bad thing.

There's lots of advice on writing operating systems, much of it conflicting. One thing everyone seems to agree on is this: don't write your own tools.

That's good advice. I understand the reasoning behind it. Writing an operating system is hard - you shouldn't make the task even harder by forcing yourself to design, implement, and debug a custom built set of tools on top of writing an OS. If you discover an error, or things crash, where is the problem? In your program or in your tools?

Thing is, I understand all this but don't really care. The people giving that advice are assuming a goal - writing an operating system - and then dispensing wisdom on the best and fastest way to reach that goal. Make sense... however, they're assuming the end product is the only reason to write an operating system.

I'm more interested in the process than the product - having fun and experimenting rather than producing yet another operating system. If I was to abandon my OS right now - in it's current rather crappy and barely usable state - I'd still consider the project a success. I've learned a lot about OS theory and internals, I've picked up a new programming language in the process (x86 assembler) and I've had a lot of fun while doing so.

Writing a programming language is, to me, just as interesting and worthy a goal as writing an operating system. I'm certainly not in any hurry, so I'm happy to divert my time and effort in to what I see as a promising line of enquiry. Especially if, as I suspect, I'll have fun along the way and it'll pay off later when I eventually work my way back to OS development.

To me, my ability to completely change tack illustrates an important downside to collaborative development:

In a group environment where team members are depending on one another, you need to be very careful about the design and implementation. You have to set explicit goals and then stick with them or you'll end up hindering your teammates. If you're flying solo, you're much more free to experiment and have fun.

[Solar]

In a group environment where team members are depending on one another, you need to be very careful about the design and implementation. You have to set explicit goals and then stick with them or you'll end up hindering your teammates.

ACK.

If you're flying solo, you're much more free to experiment and have fun.

ACK, but I'd really miss the friends I've made in the process.

[Pype.Clicker]

i'm certainly not the kind of guy to tell you not to use your own tools... Get a quick look at Clicker CVS: you have kernel/ and tools/ ... Get a deeper look at tools/ what do you see ?

• assistants to automatic makefiles generations, dependencies detection and stuff like these -- am i re-making my own IDE ?
• file format converters, extracting informations and reformatting things like .COFF or .ELF into something more easy to get loaded by the kernel.
• text preprocessors, most of which are based on PERL which rewrite some of the ugly things needed for my interface-based communication system on the fly (you say "method xyz", it builds struct xyz_message and static __inline__ xyz() {} )
• and even a compiler (yes, i have one) which extracts some commands out of your code and generate importation/exportation structures needed for modular programming.

And i'm not even mentionning the large amount of macros that extends my C program with list management (lCar, lCons, lCdr ... do you recognize them ), error management (error, errlog, errsign and others and even #define catch(x) ... ) and even inheritance (#define classXyz_inherit() and #define objXyz_extends())

Now just get a look at *how* it's done. I'm using existing tools to make my tools: language and libraries that have built-in text manipulation like perl and flex and my compiler/code generatorz generates C code for GCC, not assembly code ...

So i made my own tools, but not *from scratch* ...

Imho, most "new languages" could be outputting C code rather than assembly, and would automatically gain from C compiler's optimizer and multiplatform support ...

[Jamethiel]

I formed an incremental plan of attack.Write an assembler. Design and write a Forth interpreter. Add a compiler. Then attempt to rewrite and extend the Forth implementation to support the syntax and all the features of my dream language.

Been there, done that.

The order I did things in was to start writing bits of the Forth runtime in NASM, extending it until I had a working interpreter, then extended that until I had a working compiler. After that I added a couple tools for doing self-hosted development and wrote the assembler in Forth. By now the system is mostly written in Forth with a few ASM routines written in the Forth assembler, with the exception of the bootsector which has been patched so that it no longer exactly resembles the original source.

If you know what you're doing and work on it regularly, you can have it all working in a week or so. If not, it can take months.

My implementation is pretty bad, but you can have a copy if you want. There should be a reasonably up-to-date copy at http://www.dridus.com/~nyef/angel/forthimage.img.gz. Just gunzip it and it's a bootable floppy image. It may even work in BOCHS. It's a non-ANS ITC Forth implementation. Once you have it booted type 'edit2', and use PgUp and PgDn to look through the source.

Uh oh. Focus shift.

Suddenly I've gone from designing and implementing an operating system to designing and implementing a programming language for my operating system.

This is not necessarily a bad thing.

As far as I'm concerned, it's a good thing. I'm a big fan of dedicated-language systems.

There's lots of advice on writing operating systems, much of it conflicting. One thing everyone seems to agree on is this: don't write your own tools.

I haven't quite seen this, but I have noticed a distressing assumption that everyone should use GCC and the rest of the GNU toolchain.

I'm more interested in the process than the product - having fun and experimenting rather than producing yet another operating system. If I was to abandon my OS right now - in it's current rather crappy and barely usable state - I'd still consider the project a success. I've learned a lot about OS theory and internals, I've picked up a new programming language in the process (x86 assembler) and I've had a lot of fun while doing so.

Writing a programming language is, to me, just as interesting and worthy a goal as writing an operating system. I'm certainly not in any hurry, so I'm happy to divert my time and effort in to what I see as a promising line of enquiry. Especially if, as I suspect, I'll have fun along the way and it'll pay off later when I eventually work my way back to OS development.

I agree wholeheartedly. I've been spending a lot of time recently working on an emulator that relates to what I'd like to do in the way of OS development. I'll almost certainly use none of the code from the emulator if/when I write the OS, but I've learned a -lot- from the emulator that makes it actually possible for me to write the OS.

[Schol-R-LEA]

And i'm not even mentionning the large amount of macros that extends my C program with list management (lCar, lCons, lCdr ... do you recognize them )

And so, Greenspun's Tenth Rule is once again confirmed. ;D

(Except I'm sure your LISP isn't bug-ridden.)

[soap]

Now just get a look at *how* it's done. I'm using existing tools to make my tools: language and libraries that have built-in text manipulation like perl and flex and my compiler/code generatorz generates C code for GCC, not assembly code ...

So i made my own tools, but not *from scratch* ...

Sensible, but not always applicable.

Sometimes you can put existing tools together and have them interact in wonderful and surprising ways to create a shiny new living thing, buzzing with potential, far more than just the sum of its parts. It's great when that happens. I love it.

But sometimes you can overreach, trying to bully, coerce and coach your tools in to doing things they just weren't designed or intended to do. And it just doesn't work. The various limitations of your tools original designs combine to crush your spirit. You find yourself swearing. You stay up all night trying to get everything to work. You end up a mess with a mess. Personal hygiene goes out the window. Your bloodshot eyes and unkempt appearance makes people avoid you on the street. You wake up every morning with an empty bottle of Southern Comfort still clutched in one hand. Your wife dies and your dog leaves you. Etcetera, etcetera.

In those situations, I believe it's more sensible to make your own 'right tool for the job' rather than sacrifice your sanity before the Altar of Reuse.

Imho, most "new languages" could be outputting C code rather than assembly, and would automatically gain from C compiler's optimizer and multiplatform support ...

In my case, the nature and design of my language means that's not practical (yes, I notice you did say "most").

Figuring out how to optimise and cross-compile code sound like interesting challenges. In fact, they're the best kind of challenges - hard ones that I can visit any time I like or, should I lose interest, not at all.

[soap]

The order I did things in was to start writing bits of the Forth runtime in NASM, extending it until I had a working interpreter, then extended that until I had a working compiler. After that I added a couple tools for doing self-hosted development and wrote the assembler in Forth.

I suppose a programmer's approach to something like this is a reflection of his confidence and personality.

I prefer to wade in from the shallow end. I'm building the assembler first because I want to make sure I understand the fundamentals of generating code before I move on to more complex things.

If you know what you're doing and work on it regularly, you can have it all working in a week or so. If not, it can take months.

I'm expecting to spend many months on this, at least. But hey, the next time I do something similar it'll go faster

It's a non-ANS ITC Forth implementation. Once you have it booted type 'edit2', and use PgUp and PgDn to look through the source.

Thanks. I've had a look and you've got an impressive amount and variety of Stuff (tm) implemented there. Sadly, Forth isn't the most accessible language and I found the experience of trying to understand someone else's system a bit overwhelming.

[mystran]

I'm currently looking at the possibility of a pure Lisp dialect, for the purpose of building my OS with one, although that probably means that I won't have a working system in the next 5 years.

And yes, I'm actually trying to design my Lisp dialect to have no constructs with side-effects, except "outside" the program logic. I'm currently thinking of building asynchronous IPC system as a set of lazy event queues. Incoming events queue would be passed as a parameter to the system, and program would return a lazy list of outbound events.

I'm also investigating the possibility of using lazy evaluation for all of the code.

[Jamethiel]

I'm currently looking at the possibility of a pure Lisp dialect, for the purpose of building my OS with one, although that probably means that I won't have a working system in the next 5 years.

That makes, what, at least three people on this board looking at Lisp systems?

I'm working on some design ideas for a bytecoded Common Lisp system based on some ideas from the old Lisp Machines, the T scheme implementation, and my own ideas about what is appropriate for a Lisp OS.

The main problem I have with a Lisp system is figuring out how to compile it in the first place.

[df]

mm im not looking at lisp, but I have designed my own virtual machine (register based cpu), and am working on the compiler/assembler right now...

[Pype.Clicker]

nothing like LISP so far (but the list management macros that reuse the LISP semantic, just because its more well-known and widely accepted than a semantic based on listInsert, or "add" ... cons/car/cdr are simple and non-intrusive accessors).

However, at the right moment, the GUi server may use some bytecodes to receive widget behaviour and implement them on serverside ...

[mystran]

I'm working on some design ideas for a bytecoded Common Lisp system based on some ideas from the old Lisp Machines, the T scheme implementation, and my own ideas about what is appropriate for a Lisp OS.

You know SECD machine? That might prove to be a nice blueprint, whether for a virtual machine or compiler.

The main problem I have with a Lisp system is figuring out how to compile it in the first place.

That's not so hard. It's just another language. Actually, compiling Lisp is somewhat easier than some other languages, since you only have a few basic constructs, and you don't need to parse it, since the parenthesis essentially already represent the parse tree.

What you do need, is a garbage-collecting memory allocator in your runtime.

My main problem is how to do the "side-effectless" lazy IPC queues in a way that is sane to program with, but does not rely on evaluation order. I'm happy to rely on lazyness rules though.

One thing I think is a must is proper tail recursion. It allows one to use "goto" style constructs with simplicity and safety of functions, without the pain of extra book-keeping:

Code: Select all

(defun ab ()
  (display-menu "ba" "quit")
  (case (read-menu-command)
     ('ba (ba))
     ('quit 0)))

(defun ba ()
  (display-menu "ab" "quit")
  (case (read-menu-command)
     ('ab (ab))
     ('quit 0)))

[Jamethiel]

You know SECD machine? That might prove to be a nice blueprint, whether for a virtual machine or compiler.

I went looking for SECD after you mentioned it, but only found references to hardcopy texts and a collection of sparsely-commented implementations.

And what I'm seeing in those implementations doesn't appear to be a good match for what I want to do. I will try tracking down the hardcopy texts, though...

That's not so hard. It's just another language. Actually, compiling Lisp is somewhat easier than some other languages, since you only have a few basic constructs, and you don't need to parse it, since the parenthesis essentially already represent the parse tree.

Perhaps, but the last compiler I wrote was for Forth, which is arguably simpler, and it still tied my brain in knots trying to get it to self-host. Lisp has a much more complex machine model, especially Common Lisp, and when I wrote the Forth compiler I had a working sample for non-cross compilation and wasn't dealing with compiling to an alien VM.

Is it any wonder that I'm having trouble thinking about how to write a Lisp compiler in Lisp for cross-compiling to a different VM? Especially when I have to deal with bootstrapping the new VM image and the general ickiness which is cross-compiling source code with macros? And for some reason all of the well-described examples I have been able to find are for Scheme, not Common Lisp.

I suppose that I could write the compiler in C, which would mean that the host and target are visually distinct (which may save part of my sanity), but that means writing a substantial chunk of the runtime in C just to support the compiler and then rewriting it in Lisp to be part of the system. I may end up doing this anyway.

Or I could be getting all worked up over nothing. It wouldn't be the first time. ^_-

What you do need, is a garbage-collecting memory allocator in your runtime.

Now this I'm not too worried about. It's a well-specified problem, there are copious examples, and the only complexity is where it interacts with the rest of the VM. That, and I won't need it until the system starts running out of memory.

One thing I think is a must is proper tail recursion. It allows one to use "goto" style constructs with simplicity and safety of functions, without the pain of extra book-keeping:

Mmm... Tail chain optimizations are a godsend when you're writing state machines or bytecode interpreters or other similar funstuff. I used to wish that I could get GCC to perform them, back when I was trying to squeeze the last bit of performance out of my 6502 core...