ExC: Custom programming language

[Roman]

I suggest you to look at Rust first – I guess, it may be what you're looking for.

[Schol-R-LEA]

I would like to point the OP towards OSDev's sister site, compilerdev.org, as a suitable venue for further discussion, However, that site is very low traffic still (hence the shameless promotion), so continuing here for now might make sense.

Also, given that you seem to dislike the C style syntax, why choose that at all? The syntax has nothing to do with it's viability as a systems language (its popularity might be another matter); languages as diverse as Modula-2, Ada, Forth, PL/1, and BLISS have all served such purposes well in the past. If you don't want C style syntax, find something you like better.

[Elttob]

I'm personally not a big fan of the Rust syntax; though some syntax features are inspired by Rust.

Also; I don't like some parts of C syntax, other parts of it are fine. For example, I don't like the way arrays are initialised, but I do like things such as the way variables are assigned values and operated upon.

Edit: thanks for letting me know about the other website :p

[joegantic]

This is just Java.

[dozniak]

This is just Java.

Syntactically, maybe. But java uses a JVM, possibly with AOT or JIT, and this is a compiled language.

[Elttob]

Alright, I've finished defining the syntax of ExC, and now am working on JxCC (Java ExC Cross Compiler) which will be a Java-based program which compiles ExC code, targeted at my new operating system. I've opted to write the cross-compiler in Java because:

- Java is my strongest language
- Platform-independency
- Somewhat easier to port to ExC due to similar syntax (since an ExC compiler in ExC is needed for my OS to be self-hosting)

Currently the reference doc is fairly empty, and I'll add more than the syntax definition over time, but it's here anyway if you want to look at it: https://docs.google.com/document/d/1o7b ... sp=sharing
I'll eventually add more of an explanation as to how to interpret the optimised notation; I initially started to write it in EBNF, but it ended up getting messy so I made optimised notation to clear it up. I'll readd the EBNF at a later date.

(also, stop telling me that "This is just <language>", because I'm already aware of the similarities, and it's not adding to the discussion in any meaningful way)

[Brendan]

Hi,

Currently the reference doc is fairly empty, and I'll add more than the syntax definition over time, but it's here anyway if you want to look at it: https://docs.google.com/document/d/1o7b ... sp=sharing
I'll eventually add more of an explanation as to how to interpret the optimised notation; I initially started to write it in EBNF, but it ended up getting messy so I made optimised notation to clear it up. I'll readd the EBNF at a later date.

I'd recommend separating "syntax" from "semantics"; and separating both from "standard library", and separating all of it from "implementation" (how any specific implementation of the compiler works internally). Semantics includes things like how each operator behaves, what the operator precedence rules are, what (and if) there's a type system and how it works and if there's any automatic type conversion, what the scope rules are, etc.

Note that in the early stages of language design, syntax is an irrelevant distraction - you have to know the semantics that the syntax needs to describe before you create the syntax that describes the semantics. In a similar way the standard library is an irrelevant distraction until after you have both semantics and syntax (partly because "standard library" tends to be used as a dumping ground for things that were important but not important enough to build into the core language).

For your document; there is very little relevant information. However (in my opinion):

• Declarations (assuming similar semantics and purpose as in C and C++) are an unnecessary annoyance. They exist in C (and therefore in C++) for 2 reasons - because the ancient compilers were too crappy to look ahead when the definition is in the same source file, and because of the way source was separated into isolated pieces was so crappy that it's impossible find the definition in a different compilation unit. The root cause for both of the reasons for declarations to exist is that ancient computers only had a tiny amount of RAM (and couldn't fit the source code, compiler and the output code in memory at the same time). Java (being "more modern than C but still old") doesn't really need/use declarations.
• Inline assembly either needs "input, output and clobber" lists, or needs to be built into the compiler so that the compiler can figure out "input, output and clobber" without being told.
• For portability, inline assembly needs more - e.g. either some ugly/bad macro system (conditional compiling) or a way to provide different pieces of inline assembly for different targets.
• The "#int" directive can't work properly (no arguments needed for software interrupts) and shouldn't exist (it's a non-portable detail of the target architecture that should be abstracted and/or done via. inline assembly)
• All of your "print()" functions need to be replaced with something capable of being useful for modern software (Internationalisation). More specifically, I'd probably want some kind of "use this stringID to find a format string for the locale, then use the format string and locale and other parameter/s to construct the UTF-8 output" (where you might have a file for each locale that contains the format strings).

Cheers,

Brendan

[Solar]

Just for the nitpicking, there is a minority view (albeit a sizeable one) that considers the "header files" brought about by the C/C++ declaration style to be actually a benefit for various reasons, self-documentation of the API being one of them. But I won't get into a hot discussion here, just wanting to point it out.

[zaval]

Personally I am sure C doesn't need a replacement. Especially for the OS development. But such opinions hardly are what you want to read. So I want to emphasize another thing.
You are in process of OS development. Developing yet another bicycle tool when there is already what you need is not helping when trying to do that complex thing. I mean if you create an OS, it is too much of complexity to do so using a tool which by itself needs to be developed. How about bugs in it which such a complex thing as OS wouldn't tolerate (it just won't compile, link, run with those)? How about optimisations of code generation? Your compiler will be lame with these respects (and endless others) for long time until you polish its internals. This problem establishes even with existing languages for which a developer is trying to create his/her own compiler. You are going to bring even a new language. It's just unbelievebly hard, let alone any benefits of doing so.
But anyway, good luck! everyone is free to go crazy their way.)))
Hope the compiler/linker toolchain of your new language will support PE/COFF format so that I could use it for my future OS if it turns to be a thing.

[Brendan]

Hi,

Just for the nitpicking, there is a minority view (albeit a sizeable one) that considers the "header files" brought about by the C/C++ declaration style to be actually a benefit for various reasons, self-documentation of the API being one of them. But I won't get into a hot discussion here, just wanting to point it out.

"Raw header file as documentation" is extremely poor documentation. Poor documentation is better than extremely poor documentation; which is why "header file with extra markup for a utility like Doxygen" exists.

One of the trends for more modern languages is to build that "extra markup for documentation" into the design of the language rather than slapping it on as a crude afterthought later. An example of this is "docstrings (multiple languages). However; this is still bad.

The problem is that you really want to be able to design and document, and then implement; so the documentation of the design exists before any code does. Tools that assist with "design and documentation, before implementation" do exist, but in practice this approach tends to suffer from problems caused by the design and documentation becoming out-of-sync with the implementation.

Ideally; what you want is integration. Specifically; the ability to do design and documentation before any code is written; the ability to "auto-generate" code (stubs, etc) from that design; and the ability to ensure that the design/documentation and the implementation are "locked in sync" thereafter. The closest we've got to this (that I'm aware of) is things like [url=https://www.visual-paradigm.com/features/code-engineering-tools/]this[url] that allow you to convert in either direction but still don't do "locked in sync".

In any case; I'd consider "raw header file as documentation" as part of the problem and not part of an effective solution.


Cheers,

Brendan

[Schol-R-LEA]

@Brendan: I would like to ask, in light of what you just said, what you think of Literate Programming (whether Knuth's original WEB or derivatives such as noweb and Funnelweb) and attempts at developing a 'Hyper-Literate' offshoot (I would say 'such as my own plans', but those plans are as good as written on the wind for all the progress I have made on them).

On a related note, what do you think of such usually-honored-in-the-breach Agile principles as Test-Driven Development, Cross-Reviewing (I don't recall the more common name for it offhand), and Continuous Integration?

More to the point, why do you think these ideas have generally been ignored or discarded by even their advocates? I have my own thoughts on that, of course, but I am wondering what you think about it.

Urk, this probably needs to get Jeffed to another thread, doesn't it...

[Brendan]

Hi,

@Brendan: I would like to ask, in light of what you just said, what you think of Literate Programming (whether Knuth's original WEB or derivatives such as noweb and Funnelweb) and attempts at developing a 'Hyper-Literate' offshoot (I would say 'such as my own plans', but those plans are as good as written on the wind for all the progress I have made on them).

Excluding specific situations (e.g. educational material where it can be a very good idea); the scale goes:

• bad: not enough comments
  good: the right amount of comments (e.g. things that are relevant and not obvious from the code itself)
  bad: too many comments (things that are irrelevant or obvious)
  worse:literate programming

Note that I used to do something similar to literate programming for my OS project, and the "build utility" I wrote used to convert the source code into nicely cross-linked HTML pages with the code in boxes surrounded by paragraphs of English text, with headings, diagrams, etc (example from The Wayback Machine). It started out fine; but the amount of English text I was adding gradually decreased the further I went, and eventually I gave up. While there may have been benefits (as educational material) for people trying to learn OS development, it didn't help the project at all (and cost a "not insignificant" amount of development time).

On a related note, what do you think of such usually-honored-in-the-breach Agile principles as Test-Driven Development, Cross-Reviewing (I don't recall the more common name for it offhand), and Continuous Integration?

When I was looking into language/tool design I decided to build unit testing into the language/tools and support "multi-user IDE" (for peer programming and other reasons). However I've never actually done test driven development or peer programming.

To me; code reviews look like great way to cause arguments between team members by taking something programmers do informally in an ad hoc way and making it a formal process involving stress (and knives, and maybe a dome shaped cage surrounded by people chanting "two men enter, one man leaves"). I know that code reviews aren't supposed to be like that, but human nature and office politics are rarely constrained by wishful thinking.

More to the point, why do you think these ideas have generally been ignored or discarded by even their advocates? I have my own thoughts on that, of course, but I am wondering what you think about it.

I think people that dream up these "one size fits all silver bullet" approaches to project management are the modern equivalent of snake oil salesmen. When their methodologies meet the real world they don't work, and the victims start chopping and changing in an attempt to make it into something vaguely practical for their unique combination of project, team members and customer/s.


Cheers,

Brendan

[Elttob]

Personally I am sure C doesn't need a replacement. Especially for the OS development. But such opinions hardly are what you want to read. So I want to emphasize another thing.
You are in process of OS development. Developing yet another bicycle tool when there is already what you need is not helping when trying to do that complex thing. I mean if you create an OS, it is too much of complexity to do so using a tool which by itself needs to be developed. How about bugs in it which such a complex thing as OS wouldn't tolerate (it just won't compile, link, run with those)? How about optimisations of code generation? Your compiler will be lame with these respects (and endless others) for long time until you polish its internals. This problem establishes even with existing languages for which a developer is trying to create his/her own compiler. You are going to bring even a new language. It's just unbelievebly hard, let alone any benefits of doing so.
But anyway, good luck! everyone is free to go crazy their way.)))
Hope the compiler/linker toolchain of your new language will support PE/COFF format so that I could use it for my future OS if it turns to be a thing.

Yeah, it's a bit crazy lol. Though it should be just about as hard as rolling my own compiler for an existing language now that I've finished the language specification (except maybe the lack of resources online) and, if I program the compiler well enough, and extensively test lots of corner cases, it should be easier to detect errors. I've already got a testing system planned out:

- Level 1 tests look for errors in basic things, such as function and variable detection.
- Level 2 tests also look for errors in basic things, but also in the way they interact, such as referencing variables and multiplying them together.
- Level 3 tests look for errors in additional features, such as arrays and string data types.
- Level 4 tests look for errors in the way all of these features interact, such as passing strings into functions and setting values into arrays.
- Level 5 tests look for errors in syntactic sugar, such as += or ++ operators.
- Level 6 tests look for errors in file importing and remote referencing of functions and variables, as well as the associated syntactic features such as access scopes.
- Level 7 tests look for errors in errors; that is, making sure all corner cases are handled correctly. After Level 7 is passed, optimisations can start to be implemented.
- Level 8 tests look for errors in optimisations.

A possible Level 9 test could be looking for errors in the standard library, however this is more related to the operating system itself rather than the ExC language.

[Schol-R-LEA]

OK, fair enough. Here's what I tend to think.

For Literate Programming as Knuth and most others have approached it, they are going from too loosely coupled source documentation (as opposed to comments, which are something different) to too tightly coupled. Embedding the source code in the documentation (albeit with specialized markup and tools for extracting the code for compilation) sounds like a good idea at first, certainly; I think most code has overly-decoupled source documentation (if it has any at all, which usually isn't the case), and that could stand to change. However, embedding the code in the documentation tends lead the developer to either treat the documentation as comments, which is bad and counter to the point of source documentation; or treat the code as an afterthought, which undermines the development process itself; or split their attention between them, leaving both bad, unfinished documentation and bad, uncommented code.

I am not sure if 'Hyper-Literate' programming would really be a better alternative or not, yet. The basic idea is that the source and documentation would be written in parallel, and contain cross-linking between parts which could be automatically checked for changes made to one to encourage updating the other when necessary. Code comments would still be code comments, but everything else would be separate, though the system should be able to generate a view of the code together with the documentation for reference purposes. It wouldn't be limited to just two documents, either; multiple commentary, documentation (design, source, admin, and user, and maybe others things as well), and change and testing audit trail documents could be linked, either automatically or by the developers, and include update constraints which could be set as reminders or even auto-generated checklists.

As for snake oil salesmen, yes and no. While there is a lot of deliberate deception going around, you have to remember that most snake oil salesmen are their own best customers - they actually think it works, and aren't just trying to trick people, but are trying to convert them to their own delusions.

My own impression is that what happens is that teams who have exceptional devs, admins, and managers (not necessarily exceptionally good in a general way, just exceptional in some manner, usually one specific to the problem domain at hand) get fed up with the existing process and fiddle with it until they find a good fit, or luck into a really good fit right away, and manage to use it to succeed with some big projects right off the bat.

Somewhere along the way, they decide that it must have been the process that made them successful, rather than the exceptional qualities of the team or the specific conditions of the project. Once they have Found Religion this way, they go out to share that process with everyone the meet.

The problem is that chances are, the process was incidental to the success. Maybe it was a good process, but how can you separate that from the other aspects of the projects they used it on?

That's only part of the problem. Even if it is a good process, like any good process, using it correctly requires both skill and effort. Most of the evangelists, however, overlook the amount of effort they put into making the process work, because they were enthused over it and it didn't feel like effort. They gloss over the fact that it isn't one-size-fits-all, because they can't see that themselves.

Another possibility is that because they were exceptional, they came up with a process that requires the ones applying it to be exceptional. This is actually worst for newcomers if the way they were exceptional really was that they were excellent developers, because anyone who isn't an excellent developer who tries to use it will find themselves in over their heads with that process. I think that this is a big factor in the successes and failures of the classic Agile model - the people who came up with it were so good that they would have had some success even with a poor process - but it shows up elsewhere as well.

Next, communicating the process - and weeding out the incidental parts of the process - becomes a barrier. Everyone on the team understands the process, but because they have been swimming in that water, they have trouble explaining it to anyone who isn't in their fishbowl. So they come up with explanations which may or may not actually get the ideas across. They come up with slogans which are misleadingly appealing. They give a lot of details about what they did that may or may not be relevant. Pretty soon, those explanations become a Dogma, and most people asked to use that process are getting it second or third hand from people who were converted by the originals or later evangelists, and a lot of misinformation gets passed around by people who didn't see the original Miracle themselves. This part is particularly insidious, and is a major factor how Patterns get (mis)used by most teams, specifically - most of the ones trying to 'use patterns' don't really get what they are for, or even what they actually are (hint: they are primarily a framework for understanding how existing code works when you are refactoring it, and don't really help much when writing new code).

Finally, as the new Process Religion spreads to management, you end up with developers who don't know, care for, or even like the process having it forced down their throats by buzzword-happy managers who know nothing about the process except that it is the latest Big Thing. If the people using a process don't know the process, and don't want to learn or use the process, or haven't been given time to learn it, then there is no way even the best process can be made to work.

A similar thing happened in education in the US during the 1960s, with the infamous New Math. It was developed in response to a perceived gap in US STEM education compared to the USSR after the launch of Sputnik; the fact that neither the 'missile gap' nor the education gap were real wasn't realized until much later. There was a difference in reported test scores between the US and the Soviet Union, but the Soviets were padding the numbers, and the people screaming about the gap were discounting the fact that the US was pretty much the only industrialized nation that doesn't track students into different ability categories from an early age - which meant that they were comparing the average of all US students to the average of the 5% of students who made it into those classes in the USSR.

The response was to go talk to modern mathematicians and see what new and exciting developments had come up that might be easier for or be more interesting to students. Since set theory, Galois theory, and working in different numeric bases were hot topics at the time, the educational researchers decided that they would be the cornerstones of this New and much better Math.

Now, the New Math had a lot of good points and a lot of bad points, but the people who developed it had considerable success despite the problems. Unfortunately, no one considered whether that success was due to the method of teaching, or abilities of the researchers and teachers who were testing it, nor did they give enough thought to how to get other teachers up to speed on it.

When mandates to use it started going around school systems, however, it was an abject failure. The main reason it failed was simple: teachers didn't know it, weren't told how to use it - just use it, that's an order! - and weren't comfortable using even the parts they did understand. Worse, it was a method that was very demanding of both the knowledge and the talent of the teachers, and required a ton of hands-on time with each student - both for the teachers and the parents, who mostly had no idea what this weird new thing was and weren't expecting to spend so much time helping their kids on something they saw as the teachers' responsibility - at a time when school enrollment was at an all-time high.

In other words, it failed not because it didn't work for the students, but because it didn't work for the teachers. The reformers pushed it for about ten years before the teachers themselves revolted and refused to use it, insisting that the process was flawed for students when in fact it didn't even get to the students most of the time in the first place.

Sort of like how most places who say they use Methodology X actually use the same ad-hoc cowboy coding everyone had used since the 1960s (and don't even mention Waterfall - it never really existed as anything other than a strawman, even in the paper that first presented it, and any manager who says they use it is lying through their teeth even if they don't realize it) while occasionally going through the motions of doing some incidental part of the project the X way.

[MichaelFarthing]

Well argued!