Scripting / Extension Language

[NickJohnson]

I'm trying to find a good scripting and extension language to eventually implement in a library for my OS, so certain tools can have modifiable policy in a common format, sort of like EMACS LISP does for EMACS, but embedded in things like system daemons and shell utilities. It would also be useful to use the same language as a "standard" scripting language for the platform, because it would have to be in the base system either way. I don't have a ton of experience with writing interpreters, so I'd like to find a language with a small usable subset that I can try to implement first. I also need it to be relatively fast, although I realize this depends mostly on implementation.

Here are my current ideas and my view of their pros and cons:
1. ECMAScript + system extensions: ECMAScript/JavaScript/JScript/ActionScript are clearly used in a similar fashion to extend a web browser, and there are good open implementations (SquirrelFish, V8, TraceMonkey, Presto, etc.) which have had a lot of research into optimization. Uncommon for system scripting AFAIK.
2. Python: Used for the whole Gentoo init/packge manager, as well as many server-side applications, so can't be bad for system scripting. Not standardized IIRC, seems to change syntax.
3. LISP dialect: Simple syntactically and semantically, has been used at one time or another for system scripting/programming (old systems, LISP machines) and extension (EMACS). Funky syntax and fewer users.
4. LLVM interpreter + other language: Fastest, most flexible assuming compilers for languages are written, which I may do for C eventually anyway. Requires compilation (extension only, no scripting); not human-readable after compilation.

What would other people recommend for a language/solution?

[Firestryke31]

PHP or maybe PERL?

I know PHP can be extended to expose different system functionality.

[Owen]

I would personally go with ECMAScript, probably implemented via SpiderMonkey (I've found scaling issues with V8). Python is going through breaking changes right now (2->3), and the LISPs, while nice, are rather unconventional.

As for the LLVM script idea? It will work out slower considering the JIT overhead for most scripts.

[NickJohnson]

Thanks - SpiderMonkey seems like a really good option for ECMAScript embedding (not to mention that it's obviously the reference implementation for JavaScript.)

The thing is, I'm sort of looking to implement the interpreter myself, partially so as not to import another project twenty times the size of my OS for one feature's sake, and partially for the experience. I guess it's probably too hard to do that for a complex language like ECMAScript, which is why I was leaning toward something simpler like a LISP dialect.

It also seems like Lua fits in a similar way to LISP here (simple syntax, homogeneous datatypes, designed for embedding, reference implementation is < 20 KSLOC). Does anyone have experience with Lua?

I also realized it would be ridiculous to try and implement a runtime suitable for any interpreted language behind a LLVM interpreter, without compiling in a C implementation of the interpreter into every extension, so that's probably out the window. The main reason I was considering LLVM is that I plan to base my native toolchain on LLVM (written from scratch but binary-compatible with it.)

[inx]

The thing is, I'm sort of looking to implement the interpreter myself, partially so as not to import another project twenty times the size of my OS for one feature's sake

Although Python has already been discussed and dismissed, I felt it might be constructive to let you know that the python interpreter is rather small, is already a library by nature, and some or all of the libraries can be left out, making it extremely small.

[JackScott]

If you're going to write the interpreter yourself, don't rule out inventing the language yourself as well. If you want a solution for writing system utilities, then what is better than a language that is designed for writing system utilities on your operating system?

[Owen]

Thanks - SpiderMonkey seems like a really good option for ECMAScript embedding (not to mention that it's obviously the reference implementation for JavaScript.)

The thing is, I'm sort of looking to implement the interpreter myself, partially so as not to import another project twenty times the size of my OS for one feature's sake, and partially for the experience. I guess it's probably too hard to do that for a complex language like ECMAScript, which is why I was leaning toward something simpler like a LISP dialect.

It also seems like Lua fits in a similar way to LISP here (simple syntax, homogeneous datatypes, designed for embedding, reference implementation is < 20 KSLOC). Does anyone have experience with Lua?

I also realized it would be ridiculous to try and implement a runtime suitable for any interpreted language behind a LLVM interpreter, without compiling in a C implementation of the interpreter into every extension, so that's probably out the window. The main reason I was considering LLVM is that I plan to base my native toolchain on LLVM (written from scratch but binary-compatible with it.)

Lua is nice (Its quite like JavaScript actually), but they make incompatible changes for every single release...

Oh, and begin and end are really irritating...

[Thomas]

Hi,
You seem to have forgotten Forth

--Thomas

[Solar]

Disclaimer: I just voice my opinion, I don't say it's the "true" sollution.

The decision depends heavily on whether you want to take profitableness into account (i.e., you want your OS to be actually successful), or you want to build something for the technology's sake (i.e., best / most interesting solution, market concerns be damned).

If you're going for the former, it will be a big bonus if this "ingrained scripting language" of your OS is something people wouldn't have to learn just for your OS (because, especially while your OS is little known, nobody will bother to do so).

I'll just write about the former case; if you're developing for the experience sake, go with whatever makes you feel good.

So, who will use your scripting / extension language? I assume it will be application programmers, both professional and the bedroom type. People who write C++, C#, Java, Bash, Perl, Python, you name it.

I guesstimate about 80% of these have never written a single line of LISP in their life. LISP, being a radically different paradigm compared to the languages mentioned above, isn't that easy to pick up as an additional language.

I also guesstimate that the intersection between app developers and web developers is rather small - and ECMAScript has very little meaning outside the web developer community.

The LLVM approach means compilation - not nice, you named the disadvantages yourself.

So, I would absolutely vote for Python, with LUA coming in as second place.

[Owen]

JavaScript is something your average app developer can pickup quickly. So is Lua. JavaScript is also becoming relatively common as an embedded scripting language inside other apps. Really, theres not much difference between JavaScript and Lua besides syntax.

Python's interpreter, on the other hand, is quite large, and really, really doesn't support threads.

[Solar]

...and really, really doesn't support threads.

Erm... what?

I've only scratched the very surface of Python so far, but I find *that* very hard to believe. Could you give some pointers?

[Owen]

...and really, really doesn't support threads.

Erm... what?

I've only scratched the very surface of Python so far, but I find *that* very hard to believe. Could you give some pointers?

From the documentation:

The Python interpreter is not fully thread safe. In order to support multi-threaded Python programs, there’s a global lock, called the global interpreter lock or GIL, that must be held by the current thread before it can safely access Python objects. Without the lock, even the simplest operations could cause problems in a multi-threaded program: for example, when two threads simultaneously increment the reference count of the same object, the reference count could end up being incremented only once instead of twice.

I could understand a lock around a given context - after all, you do need to keep an the state consistent - but this goes far beyond. It is one lock which all contexts use.

In other words, a single process can only run one thread of Python code at any time.

[Solar]

*gulp*

That is...

...well...

...not nice.

[NickJohnson]

I think based on what I've heard and researched about Lua, it is the best language for the job. The reference implementation is 12 KSLOC with the standalone interpreter and compiler (small), is written in pure ANSI C (portable), has been used in embedded devices and handheld console games (fast), and seems to have the most ingenious set of primitives, even rivaling Scheme. As far as I can tell, the syntax is actually pretty stable, because even though they break compatibility often between releases, releases are many months apart even for minor revisions and many years apart for major ones (see the dates here). Not to mention that it's thread-safe, and my system daemons are extremely multithreaded.

[Owen]

I think based on what I've heard and researched about Lua, it is the best language for the job. The reference implementation is 12 KSLOC with the standalone interpreter and compiler (small), is written in pure ANSI C (portable), has been used in embedded devices and handheld console games (fast), and seems to have the most ingenious set of primitives, even rivaling Scheme. As far as I can tell, the syntax is actually pretty stable, because even though they break compatibility often between releases, releases are many months apart even for minor revisions and many years apart for major ones (see the dates here). Not to mention that it's thread-safe, and my system daemons are extremely multithreaded.

Its thread safe only to the point that you can have separate contexts in different threads. A Lua context is not at all thread safe.

lua_newthread is to do with coroutines, not actual threads.