Command Line Parser Ideas

[SpyderTL]

So, I'm in the process of designing a new command line parser, and I wanted to get some opinions.

I've looked through several documents explaining the process of scanning, lexing, parsing, and executing hierarchal commands from a string of characters. They typically break each of these steps up into separate components, and pass data structures from one component to the next, but for my first implementation, I'm trying to come up with a combined solution, for simplicity.

So, after working through a few ideas on paper, I think I've found one that may work for what I need in the short term. Here is a summary of what I'm trying to accomplish:

Take a string, such as:

Code: Select all

Pci.GetDevices().Skip(Integer.Parse("1")).First()

And find and execute the commands, in order, to perform the requested operation(s). I want arbitrary (unlimited) depth, so that I can nest a large number of function calls inside other function calls, and I want to be able to chain as many function calls together as needed. This will be similar to C# code, or (more specifically) PowerShell commands.

So, my current idea is a single-pass, dual-stack setup.

The first stack is for "commands" and the second stack is for "arguments". As the string is scanned, character by character, special characters will trigger certain events. For example, the period "." character will take the previous string, and dynamically find classes or methods by name, and push them on the command stack. The open parentheses "(" and comma "," characters will start pushing strings and integers onto the arguments stack. The close parenthesis ")" character will pull the command off of the command stack, and pass it the argument stack, and any return value will be pushed onto the argument stack. Any commands after that will use the top of the argument stack to find the correct class, and push it to the command stack, and then push the next command onto the command stack, and so on, and so on.

I will also need a similar procedure that I could use for things like autocomplete, which will walk through the classes and methods, but not execute them.

If this works, I'll eventually move this code to its own (reusable) function that will simply return a list of commands to execute in the correct order, but for now, I'll probably hard-code this into the console application.

Let me know if anyone sees any reason why this would not work, or if my description above is too hard to follow, I'll try to elaborate.

EDIT: Slight correction/clarification...

Numbers and quoted "" strings (separated by commas) will be pushed directly to the argument stack.

[max]

Hey SpyderTL,

I'm planning to do something similar. To the parsing part itself, sure that will work.
The only issue I see (for that I also have to find a solution) is how the programs should receive each of these calls. Like if you have "Vga.getScreen(0).setResolution(1024, 768)"; for this you would first have to call the application "vga" with "getScreen" at "0", and then use what you get back to set the resolution - but what do you get back?

I was planning to do it like this:
- the program itself is instantiated once
- if you use a function (like a getter), you get back an "internal handle" to that resource that the process gives
- the second function call uses that handle to ask the process to do the next thing
nested calls might get a little tricky, because you'll have to use something like pipes to get the return value from process B to the call on process A.

Greets,
Max

[Brendan]

So, I'm in the process of designing a new command line parser, and I wanted to get some opinions.

My opinions are:

• You're designing a scripting programming language, not a CLI.
• A CLI is a crappy thing that should've become obsolete 20 years ago (and mostly did).

My suggestion is to design a scripting language for your GUI (sort of like Applescript, or AutoIT, or Tcl/Tk]).

Note: Also in my opinion; to do this right you need to split applications into "front-end" and "back-end"; where both the front-end (the application's user interface) and scripts are designed to talk to the back-end (and no front-end is needed when a script is using the back-end). The interface/protocol for the back-end needs to be relatively well standardised so that it's easy for scripts to talk to the back-end; and supported by most applications.


Cheers,

Brendan

[embryo]

• You're designing a scripting programming language, not a CLI.

What do you think a suggestion list is? Is it a language or a CLI part?

• A CLI is a crappy thing that should've become obsolete 20 years ago (and mostly did).

When we type letters in our preferred GUI editor we accidentally start some actions like suggestion list display or cursor position change or insertion of a text or something else.

It means there is a union of CLI and new features, that now is called GUI.

[Brendan]

Hi,

• You're designing a scripting programming language, not a CLI.

What do you think a suggestion list is? Is it a language or a CLI part?

It's one type of user input assistant, that could be used for CLI or various parts of a GUI (where text input is involved but nowhere else); that has nothing to do with whether the text is a complex scripting language (e.g. with functions, variables, etc), or if the text is a simple "executable name followed by executable's arguments" thing with no complex features beyond basic text substitution.

• A CLI is a crappy thing that should've become obsolete 20 years ago (and mostly did).

When we type letters in our preferred GUI editor we accidentally start some actions like suggestion list display or cursor position change or insertion of a text or something else.

It means there is a union of CLI and new features, that now is called GUI.

Fine. Implement a scripting language for a "CLI that has buttons, icons, drop down lists, checkboxes, tree-view widgets, etc; where everything is graphics and not limited to plain text, and where the user is able to use mouse or touchpad instead of typing; that isn't a GUI at all".


Cheers,

Brendan

[SpyderTL]

I'm trying to narrow down the scope a bit.

The "thing" that will be running this code will initially be a simple CLI, text based, default 80x25 screen. But it will (probably) be reused in the future for a scripting language, or full blown programming language, and I'm even considering giving my "windowed" GUI elements their own "console area", which would also need this type of functionality.

But for now, I just want to take a string of characters, and execute it as a command.

I'm specifically looking for a) opinions on the two-stack, single-pass approach, and b) if anyone can think of a situation where this would not work (or that may be difficult to implement).

Also, I have a related question... If I was only using one stack, things would be (fairly) simple / straight-forward. However, if I'm keeping up with two stacks, in Assembly, I'm tempted to "swap" between the two stacks by swapping the SP register with some other register. But I can see where that might cause unforeseeable problems later when dealing with function calls, or stack frames, or multithreading and process swapping, especially with multiple CPUs.

Is this something that I should be worried about? Is it safe to, say, swap SP and DX for a few instructions, and then swap them back? Or would it be "safer" to leave the SP register alone and use two "software" stacks, instead?

[SpyderTL]

I'm planning to do something similar. To the parsing part itself, sure that will work.
The only issue I see (for that I also have to find a solution) is how the programs should receive each of these calls. Like if you have "Vga.getScreen(0).setResolution(1024, 768)"; for this you would first have to call the application "vga" with "getScreen" at "0", and then use what you get back to set the resolution - but what do you get back?

In my case, fortunately, I'm only dealing with a single process, which has access to all of the "classes" in memory at the moment. (Similar to C# or PowerShell)

If I were to need to cross-over to a different process, then you would just need to make sure that your "interface" between your applications included enough "reflection" information (or metadata) so that the caller could understand the data that was being returned.

Or, another option would be (and this is pretty much what I'm doing) to separate the application from the data structures. So, communication with an application would require that both the caller and the callee agreed that only "externally defined" structures could be used. In windows terms, it would mean that you could only use interfaces defined in a TLB or COM DLL.

So, in your example:

Code: Select all

Vga.getScreen(0).setResolution(1024, 768)

First, the "Vga" program would be started. Then a message would be sent to the program that contained the name of the method to call "getScreen", and the parameters -- a zero, either as raw bytes, or wrapped up in some sort of well-known cross-process wrapper. Then the Vga application would return you a well-known object (defined in a separate location) as a return value.

Then the next step would be to call the setResolution method on that object. I can see a few ways that this could work:

1. The already running "Vga" application could be sent another message asking it to run the "setResolution" method, and pass it the two parameters 1024 and 768, and also pass it the object that was returned from the previous method call.
2. Use the metadata on the object returned from the getScreen call to find the setResolution method in memory, and call it directly, passing the two parameters, and possibly getting back an object based on whether the function succeeded or not.
3. Use the metadata on the object returned from the getScreen call to find the application responsible for the "setResolution" method, and then run that application and send it the "setResolution" message.

The key is that the metadata for the object is "public", either exposed by the application, or stored in a completely separate file or location. (or both...)

The good news is that once this inter-process communication system is in place, you could easily add additional features like automatically loading any needed code or data (like Windows .DLLs), or even seamlessly making calls to other machines over the network, or services on the internet.

Code: Select all

Netflix.GetRelatedMovies(System.CurrentUser.Preferences.GetFavoriteMovies().Randomize().First()).Randomize().First().Play();

This is essentially what I'm shooting for.

[embryo]

Implement a scripting language for a "CLI that has buttons, icons, drop down lists, checkboxes, tree-view widgets, etc; where everything is graphics and not limited to plain text, and where the user is able to use mouse or touchpad instead of typing; that isn't a GUI at all".

You can download it here. But it requires Eclipse IDE to be used. However - the idea is independent of any IDE.

[embryo]

in Assembly, I'm tempted to "swap" between the two stacks by swapping the SP register with some other register.

You are tempting to trade efficiency for debugging nightmare. Of course, if you will use low level native means, then you will achieve better performance, but the high level design approach is not happened just accidentally. If you will stick with high level then you will get the so needed ease of development.

But of course, it is up to you how to spend your time.

[brunexgeek]

Your CLI/OS would be fully based on object-orientation?

Code: Select all

Vga.getScreen(0).setResolution(1024, 768)

For me, this commands should be equivalent to

Code: Select all

set_resolution --screen 0 -w 1024 -h 768

instead

Code: Select all

 get_screeen --id 0 | set_resolution -w 1024 -h 768

unless your OS can "return" objects from applications.

[AndrewAPrice]

If you remember the microcomputer era, it was common for shells to be BASIC interpreters. It's possible you could make your shell an interpreter for a full language, but you need to be careful that it's not tedious to do simple things like:

Code: Select all

System.Processes.Run("gcc", new char[] { "-o","test.o","test.c"});

vs

Code: Select all

gcc -o test.o test

I'm leaning towards a language like Haskell for my shell. It has nice features like function composition (for chaining together calls), lazy evaluation (a program's output could be a lazy evaluated string), executing a program can merely return a lazily evaluated string (that keeps printing to the terminal until it reaches an EOF).

Combine this with a terminal markup language that you can print tables and styles, and you have something pretty powerful.

[SpyderTL]

You pretty much hit it, exactly. I'm really trying to re-imagine the OS as a smart, object-oriented environment, that provides the user with tons of functionality that is easy to use. (As easy as Java/.NET development, at least.)

I've posted detailed information on other threads, but what I want is to be able to go to the store, buy a brand new laptop, bring it home, install an OS, and immediately be able to do things like:

• Stream video from the internet, in any (legally usable) format, and simultaneously save it to disk and display it on screen.
• Take a snapshot with the webcam every 10 seconds, and compare it to the last snapshot, and save the image if the image has changed beyond some threshold.
• Stream audio from a file on a local network DNLA server, and use the waveform data to render the waveform to the screen, while the song is playing.

These are all things that I could do using free, built-in components in Java or .NET as a developer. But as a user, these are pretty much impossible without either purchasing additional software, or finding and downloading various utilities and learning how to use each one of them.

And, as for the command line format, this is mostly just preference, but I really don't like the MS-DOS or the Unix standard for passing parameters. I think it comes down to my preference for modern IDEs, which provide you with things like Intellisense, Tooltips and Autocomplete. It would be difficult to provide these features using short, cryptic commands and parameter switches.

Code: Select all

wget http://osdev.org

But, they make a lot more sense if you are using longer, more descriptive commands.

Code: Select all

Web.WebPage.Download("http://osdev.org")

Plus, you can use things like classes and namespaces to organize your functions into logical groups. (Instead of dumping thousands of utilities in a /bin folder, for example)

As a matter of fact, I'd be perfectly happy getting rid of "named" programs, altogether. (wget, gcc, ls, Internet Explorer, Outlook, Firefox, Chrome, etc.) That's just more "learning-curve" that the user must go through before they can do what they actually want to do.

My phone does all of that just as well as my PC, but it doesn't make me remember which programs do what. It just says "Mail", "Maps", "Music", "App Store", etc.

Much better, in my opinion...

Edit: I probably should mention that I got this working using the dual-stack method. The autocomplete version isn't done yet, but I can now call methods and pass the results to other methods as parameters. I've been busy implementing all sorts of methods that I use in C# on a daily basis. Things like math functions, stream readers/writers, etc. I may even start working on XML parsing next.