Basically, if you're familiar with Windows Powershell, certain aspects of the design are along similar lines. I want to make a shell that has a concept of "data structures" that can be passed between programs (as opposed to just binary or text streams) and which incorporates certain ideas of object-oriented programming - at the same time I want it to have some of the flavor of more traditional Unix environments.
So probably the biggest conceptual change between, for instance, Bash and my shell design is that my shell would explicitly support a set of data types and structures which could be passed over the streams between processes. Most data would be passed by value, though I definitely want to have support for passing objects by reference in the future. The usual interchange format would be a new binary stream format (a binary representation of XML might work here, but at present I'm planning to make my own format. I may change my decision at some point - I'm sure people would adopt an XML solution more readily than something I cook up myself)
When a command is run it generally returns one or more values as a result (via the process's "standard output" file descriptor) - in terms of the structure of the result there's no difference between a command that returns one value and a command that returns more than one. (This is meant to help the definition of pipeline commands be more consistent - they can basically do a "for each" over their input even if the program from which they get their input doesn't allow for the possibility that there will be more than one value output...)
Values will have datatypes associated with them - the shell design will break with Unix tradition in that numerical values will not be text. The baseline data types will include numeric types, text in various encodings, symbols, file references, and structural stuff like lists and data records.
If the output format of one process doesn't match the input format of a process to which its output is being piped, data will be converted (if possible) to the format expected by the receiver. If the conversion is lossy, incomplete, or functionally impossible (some SHIFT-JIS characters, I believe, can't be written in Unicode, for instance - and of course a bigint may not fit in a 32 bit int, or conversion of an image to JPEG would be lossy) some sort of error or warning will be made.
In addition to regular executable files, non-executable data files may be used as command names if there is an "object layer interface" on the PATH for that file's data type. So, for instance, one could run "./picture.jpeg --size" as a command and get the pixel size of the image back as a pair of integers. The file in question needn't have the executable bit for this to work, it just needs to be recognized as a data type which has an object layer. In order to help protect users from deceptive files (for instance, malicious binaries, named as if they were data files, but with the executable bit set and program code inside) data files will be made visually distinct from true exectuables when the filename is typed in.
Of course, I'm not under the delusion that people will suddenly abandon their own data formats and work in whatever format I choose. It's just not practical or realistic. So my plan is to provide the following provisions:
- There will be a certain set of "recognized file types" which can go over streams between shell process without having to be tagged in any way - so if a process writes out XML, the receiving process will recognize that and have the option of dealing with the data as XML or attempting to convert it to something else.
- Commands can communicate their input or output format statically in cases where it's possible - basically an executable file could be tagged with metadata which would say "this program writes output in the form of a binary ISO9660 file system" or something like that - this data would then be communicated to the receiving process via a different channel, if necessary.
# The comma is a "sequencing" operator - it has lower precedence than command invocation. It yields all of the values of the left-hand argument followed by all the values of the right-hand argument.
> (primes >> head 3), "something"
(2, 3, 5, "something")
> ((1, 2, 3), (4, 5 6))
(1, 2, 3, 4, 5, 6)
# The ampersand operator works as it does in bash - two commands can be run concurrently or one can be run without the shell waiting for it to finish... If two commands are run concurrently and both generate output, the output of the two commands is interleaved - values appear in the interleaved result according to how soon they were made available.
> (primes >> head 5) & (dictionary >> head 3)
(2, 3, "A", 5, "A's", "AOL", 7, 11)
# List data structure: when a sequence is encased in a list it becomes a single object...
> ([1, 2, 3], [4, 5, 6])
([1, 2, 3], [4, 5 6])
> [0, 1, 2, 3] --at 2
3
> [0, 1, 2, 3] --all
(0, 1, 2, 3)
# Environment variables can hold just about anything - if a variable containing a data structure is exported, the subprocess will get its encoded form, unless it's something like PATH - which some processes are going to expect in the traditional colon-delimited form...
> $a = ./image.png # $a contains a copy of the binary data in the PNG file, plus explicit type information identifying it as a PNG file...
# Equivalent of the traditional "find" command... it works the same but returns a sequence of filenames rather than a newline-delimited text stream...
> find -iname "*.png"
("foo.png", "bar.png", "contains spaces in the filename.png")
# In bash this command would be "find -iname "*.png" -print0 | xargs -0 rm", if you want it to handle filenames with spaces correctly...
> rm (find -iname "*.png")
# Scale a GIF file down and convert it to PNG:
> ./image.png = ./image.gif --scale --maintain-aspect (128, 128) : image/png
# (128, 128) is a sequence of values. Colon is a low-precedence type coersion operator. "image/png" is a data type constructor found on the PATH. The assignment operator can create files if the left-hand side is a filename.
# One potentially awkward decision I made in the design is to make different kinds of object references distinguishable by the parser, and have them be uniform in all contexts...
# An undecorated name is a reference to object on the PATH:
> cmd
# A name with a leading slash, dot-slash, or dot-dot-slash, tilde-slash, etc. is an absolute or relative file name... This will make some people uncomfortable since you can't refer to a file in the current directory without using the leading dot-slash...
> ./file.jpeg
# As a (hopefully) handy shortcut, names leading with a dot refer to files stored in a particular subdirectory of the user's home directory - a convenient way to refer to data stored in a persistent location...
> .data #Equivalent to ~/.shell-persistent-data/.data, or something like that...
# When I say these forms work the same "in all contexts", I mean it... Since there's really no telling where a command's arguments end unless you case it in parenthesis, however, the rule is that arguments belong to the first command to which they could possibly belong... For instance:
> rm find #removes all files in the current directory
> rm find -f #equivalent to rm (find) -f - the -f is an argument to "rm", not "find"
> rm (find /some/path) #This is what you'd have to do to provide arguments to "find"...
# If somebody wants to pass in a name that isn't some kind of file reference, there's three ways to do it.
> "some text" # Make it a string
> 'sometext # Or there's this string syntax, convenient for strings with no white space
> --symbol # Or it can be a symbol that's recognized as an argument by the program you're calling...
# I use curly braces to denote program blocks - for cases where somebody wants to do something that doesn't quite fit in my shell syntax they can write a code block in another syntax...
> {#!regexp /foo|bar/} #Equivalent to (regexp "/foo|bar/") - creates a regular expression object - except that it's the regexp utility that determines how much of the following text belongs to it... Obviously this sort of thing will only work for languages that have a syntax where you can tell, when encountering a closing curly brace, whether that curly brace is part of the program or not... So I think it'd mostly be useful for small tools that augment the shell syntax, though there's a chance it'll be adequate for some full-on programming languages as well...
Assorted other concepts:
- Utilities like "grep" would be replaced with separate "filter" and "regexp" utilities - the former would filter a sequence of any type according to some callable predicate, and the latter would create a predicate object based on a regular expression... This makes it easier to use regexps in other cases, or to specify other regexp syntax without all tools having to support the same set of regexp syntaxes...
- I hope to have a rich set of infix operators... Part of the reason I changed the redirection operators is because I wanted the pipe character as a "where" operator, and to support less than, greater than comparisons as part of the basic syntax...
- I haven't worked out exactly how "structs" would work, syntactically speaking, but I feel there should be some sort of support for data structures with named fields, however. One idea I am considering is to support -> as a name binding operator - specifically it'd yield the right-hand value, but with metadata attached identifying the left-hand value as its name. Then the list syntax could be used to encase a sequence of such bindings in an object...
- Various bits of the plan suggest that programs can supply information to the shell (possible calling arguments, input/output datatypes, etc.), statically or otherwise... Of course programs not written for the shell wouldn't normally supply this, so the plan is that such support could be layered on top of existing binaries through filesystem metadata or wrapper scripts
- I'd like for archive file formats like ZIP, etc. to act more or less like directories - of course, making the shell support that and making everything the shell might run support that are two separate problems... I guess to go all-out with it I'd need to use FUSE or something to actually mount it, rather than just making it a shell-level thing...
- Commands like "cvs" or "chmod" don't follow my usual assumptions about GNU-style command argument format. Running something like "cvs checkout" in my current design would result in PATH searches for both "cvs" and "checkout"... I'm considering different solutions to this - possibly abandoning the idea that the rules are the same "in all contexts" - or possibly providing a way that "cvs" can provide its own name space, so that "checkout" is first tried as a "cvs" command, and as something else only if that fails...
- I use the colon character for type coercion - this complicates the possible use of the shell on Windows (due to drive letters) and, perhaps more importantly, complicates the use of URLs as well, and X display variables - unless some kind of exception is made to recognize these forms...
- I have generally wanted to avoid the use of the backslash character as an escape character (because it's used so much on Windows as a directory separator, and because it's visually too similar to the slash) but if I don't use backslash, I'll need to figure out an alternative... I'm not sure if I need backslash only inside quotes, or elsewhere, too...
- I wanted the shell to be able to do math with infix operators - but I'm not sure that will work out, given the special meanings assigned to "-", "/", and "*" in the shell...
