OSDev.org

Alternative title: "klange finally writes that Python clone they've been talking about doing for years"



I am pleased to announce that my winter project, Kuroko, has reached a stage where I'm comfortable enough to make an official announcement.

Kuroko is a light-weight, embeddable, dynamically-typed, bytecode-compiled, garbage-collected Python-like scripting language. It features native list and dictionary (associative array) collection types, first-class functions, classes, exception handling, optional function arguments, keyword arguments, native iteration ("for-each") loops, function/method decorators, loadable modules (including native code modules), and more. Kuroko is easily embeddable, with minimal C standard library requirements and no external dependencies. The interpreter can be easily extended with C modules to provide bindings for native libraries or integration into a host application. The full-featured interpreter core can be built down to a 166KB static binary (with musl), or a 140KB shared object (with glibc). Kuroko's syntax is highly compatible with Python, so it is easy to get started and port existing code. Kuroko also features a syntax-highlighted repl with tab completion, so you can get started right away!

Source repository: https://github.com/klange/kuroko
Code examples: https://github.com/klange/kuroko#code-examples

And of course it works in ToaruOS:

Looks good! I’ve never been able to get on well with dynamically typed languages though

Nice! I had a lot of fun with Python itself 20 years ago, more than any other language at the time. I even tried to make an all-Python userspace for Linux, but in those days I would get discouraged when I couldn't implement exactly what I thought I wanted. I didn't have the skills to implement mount and other system utilities in Python, and felt I as spoiling the purity by bringing them in as binaries. (Concern for imagined standards of purity has always been my biggest problem with programming.) Anyway, I liked Python very much when it was simpler. If I ever get to thinking I do want complex syntaxes (not Forth, Lisp, APL...), I'll be sure to give Kuroko a try.

Before I saw the reference to the anime, I websearched and found kuroko are stagehands in traditional Japanese theatre. I think that's an excellent metaphor for a scripting language.

Looking at the examples..

That's an interesting way of creating a static local variable.

I think you've got a typo under Lambda functions. The example defines myLambda but references myLambdaa.

Woot! Implicit `self`! I got so tred of typing that, back in the day. XD

Typo in the `set` example: the last print is "pritn".

The C-style loop has `i = i + 1`. Nothing wrong with that, but `i++` is the more instantly recognizable form.

Oh oh oh! `if` can be part of an expression and return a value! I always liked that much better than `?:`

What's the purpose of `with`? Just temporary variables?

Evaluating default arguments at call time is good. I remember being very confused by the way Python did it.

Hey eekee, thanks for the feedback on the README - cleaned up the typos

I websearched and found kuroko are stagehands in traditional Japanese theatre. I think that's an excellent metaphor for a scripting language.

Kuroko were dressed in full-body black clothing and would be tasked with manipulating set pieces on stage as a kabuki play progressed - their black outfits were supposed to hide them. Historians believe our modern image of ninjas actually comes from appearances in plays where they were dressed like the stagehands to surprise the audience!

The C-style loop has `i = i + 1`. Nothing wrong with that, but `i++` is the more instantly recognizable form.

I think I wrote that section a few days before I added the increment/decrement shortcuts

Oh oh oh! `if` can be part of an expression and return a value! I always liked that much better than `?:`

Getting the semantics for Python's ternary expressions working required a lot of thought, since the condition comes after the first result expression and my compiler has no AST stage; glad I got there in the end!

What's the purpose of `with`? Just temporary variables?

The intent of `with` blocks is to ensure that cleanup tasks happen regardless of early returns (and also exceptions, but still working on that one). In some languages, this is done by allowing objects to automatically call functions when they leave scope or are de-allocated, but with a dynamic type system the overhead of checking for these methods every time a variable goes out of scope is considerable, so instead the "with" block explicitly marks such objects and the scope for which their entry and exit should be tracked. The most common usage of this in Python is opening files using a "with" statement - they'll close themselves automatically on exit. Python gets more use out of `with`, though, as its approach to scoping means you can't just throw a random block construct at it to reduce the lifetime of a variable.

Looks pretty good!

If I understand correctly, it is interpreted and not JIT compiled. Do you have some performance measurements? (FYI for security reasons I prefer interpreters even though they are slower)
Is the bytecode yours? Or did you use an existing one? (Unlike Java, WASM etc. Python bytecode is not standardized, that's why I'm asking)

Could you please compare your solution with this implementation? This is the one I was playing with so far (also not bytecode-compatible):
https://github.com/micropython/micropython/
This is a great embeddable Python interpreter, but has some shortcomings and very difficult to maintain, so yours looks very promising! (uPython also available on microcontrollers, but I'm not interested in those, only in x86, the version in "ports/unix" to be precise) It doesn't matter if your language is not 100% Python, just Python-like, that's perfectly okay.

Well done,
bzt

If I understand correctly, it is interpreted and not JIT compiled. Do you have some performance measurements? (FYI for security reasons I prefer interpreters even though they are slower)

Yeah, execution is entirely running the bytecode through the VM and calling appropriate C functions to churn out results. I have a number of code snippets I've ported from Python that I run regularly and the results are promising - CPython's highly-tuned object implementations beat me out a bit, but the difference isn't all that big. I'll see if I can get some proper numbers and do a complete comparison...

Is the bytecode yours? Or did you use an existing one? (Unlike Java, WASM etc. Python bytecode is not standardized, that's why I'm asking)

The bytecode started from the Lox bytecode in Crafting Interpreters, though I rather quickly introduced features for larger code sizes; since then, it's organically grown towards something very similar to CPython's - often the disassembly of the output from my compiler looks nearly identical to the same code run through CPython, though it does have a better optimizer.

klange wrote:

If I understand correctly, it is interpreted and not JIT compiled. Do you have some performance measurements? (FYI for security reasons I prefer interpreters even though they are slower)

Yeah, execution is entirely running the bytecode through the VM and calling appropriate C functions to churn out results. I have a number of code snippets I've ported from Python that I run regularly and the results are promising - CPython's highly-tuned object implementations beat me out a bit, but the difference isn't all that big. I'll see if I can get some proper numbers and do a complete comparison...

Is the bytecode yours? Or did you use an existing one? (Unlike Java, WASM etc. Python bytecode is not standardized, that's why I'm asking)

The bytecode started from the Lox bytecode in Crafting Interpreters, though I rather quickly introduced features for larger code sizes; since then, it's organically grown towards something very similar to CPython's - often the disassembly of the output from my compiler looks nearly identical to the same code run through CPython, though it does have a better optimizer.

Thanks! Performance is not everything, sometimes having a dependency-free and easy to maintain, easy to port source code worth more.

Cheers,
bzt

Looks nice. Where do you see the biggest differences compared to Python? Where does Kuroko improve upon Python?

Personally, I like Python a lot and use it regularly (our "meta-meta" build system xbstrap is written in Python, as well as our IDL compiler and some other tools) but IMHO some parts of it could really be done better. For example, attribute access is a mess and goes through 3+ layers of double underscore functions.

bzt wrote:

klange wrote:

If I understand correctly, it is interpreted and not JIT compiled. Do you have some performance measurements? (FYI for security reasons I prefer interpreters even though they are slower)

Yeah, execution is entirely running the bytecode through the VM and calling appropriate C functions to churn out results. I have a number of code snippets I've ported from Python that I run regularly and the results are promising - CPython's highly-tuned object implementations beat me out a bit, but the difference isn't all that big. I'll see if I can get some proper numbers and do a complete comparison...

Is the bytecode yours? Or did you use an existing one? (Unlike Java, WASM etc. Python bytecode is not standardized, that's why I'm asking)

The bytecode started from the Lox bytecode in Crafting Interpreters, though I rather quickly introduced features for larger code sizes; since then, it's organically grown towards something very similar to CPython's - often the disassembly of the output from my compiler looks nearly identical to the same code run through CPython, though it does have a better optimizer.

Thanks! Performance is not everything, sometimes having a dependency-free and easy to maintain, easy to port source code worth more.

Cheers,
bzt

No, not everything, just 80% of it . To be fair, I would have to say that speed and security are number one, and stability is number two, as crashing for me is less annoying then bad performance. Looks nice, though @klange!

kuroko as in shirai?

xeyes wrote: kuroko as in shirai?

Yes, or did the logo, association with ToaruOS, and link to the Fandom article for the character in the README not make it clear

Anyway...

I have some exciting news: I built an in-browser REPL with a WASM-compiled version of Kuroko, so you can try it out right now:

https://klange.dev/kuroko/

Try some of the examples from the README - and let me know if any of them are wrong, it was written as development progressed to track feature coverage and some things may have changed since their original README section was written.

I've officially marked off a release candidate for Kuroko 1.0.

If you want to try it out from your browser, the WASM repl has been updated and includes a detailed interactive tutorial.

I also cut a new release of ToaruOS that includes the Kuroko 1.0 RC and a demo of using it to make GUI applications.

Since the last post, Kuroko got its first official release... and a bug fix release shortly after that.

I've also set up a little page around the WASM REPL so it's a more inviting website and not just a big terminal in a browser tab: https://kuroko-lang.github.io/

It seems there is a bug with the online Kuroko terminal thing at https://kuroko-lang.github.io/.
This bug could not be reproduced on the full-screen terminal thing at https://klange.dev/kuroko/.

Steps to reproduce:
1. Type any statement, for example "let a = 1"
2. Press enter
3. Press any printable key except enter
4. Observe two extra newlines added prior to the character

Additionally, if you type another line of script with those two newlines prior, (for example, "let b = 2") and press enter, it will not execute that line but instead create a newline. You have to remove the two prior newlines OR press enter again before it will execute the script.

Correct me if I'm wrong, but I believe the intended result is that, if Shift+Enter is pressed, it will not execute the current line but instead add a newline; then when Enter is pressed normally, it will execute all that you have written.

(If you prefer, I can open an issue on the repo in GitHub)

foliagecanine wrote:It seems there is a bug with the online Kuroko terminal thing at https://kuroko-lang.github.io/.
This bug could not be reproduced on the full-screen terminal thing at https://klange.dev/kuroko/.

Looks like this is a bug in the code editor I'm using (Ace) when disabling its line wrapping mode; I've turned wrapping back on and it seems to have fixed this.