ExoForth Design concepts for critique.

[jemo07]

Hello all, I have not been lurking long in the forum as I just found it a couple on moths ago looking for some novel ways to consider IPC handing.

I´m new to OS kernel development, but have built and architected some simplex systems in the past. Most recently I have created some minimalist VMs and was looking at creating a bare metal VM, which started to look a lot like a Kernel. Since I hace created simple Forth systems in the past, I started to consider how a combination of Exo-Kernel with Forth might look like. I did not find and Exo Kernel examples to better grasp this, maybe did not search hard enough, so any directions might be helpful. On a last note, I’m not a C guru and since most of my code has only been consumed by me, my style is very poor although I focus on efficient output of the compilers… I need to get into the gist of C and thought creating a little toy OS with a Forth interfase might be a project to start working on.

Ok, here is my open thoughts on what I would like to accomplish. Would love you guys to look at this in a meaningful way to critique some of the design concepts. I know Forth is not for everyone, but there might be some ideas worth considering. BTW, I have no focus on user interfaces, so please consider most of my efforts as process / functional / lambda programs that will be running with in the CPU.

The Forth (Exo)Kernel system design, based on an architecture that merges the Exokernel philosophy with the flexibility and customization of the Forth programming environment. This design is tailored for high-efficiency and scalability, particularly in advanced computing environments with high-core-count ARM CPUs. Here’s a detailed summary of its key aspects:

Core Philosophy
Exokernel Influences- The system adopts the minimalist approach of Exokernels, focusing on exposing hardware resources directly to applications rather than abstracting them away.
Forth Flexibility- Integrating Forth’s language capabilities allows for deep system customization and efficient, low-level hardware interaction.

System Components
Self-Contained Forth VMs- Each application runs within its own isolated Forth Virtual Machine, which includes a dedicated dictionary and stack. This isolation enhances security and prevents interference between processes.
C to Forth Compiler- Integrated within each Forth VM, this compiler allows applications written in C to be compiled into Forth, facilitating ease of development and portability.

Memory and Resource Management
Dynamic Memory Addressing- Applications view their memory allocation as starting from address 0x0, regardless of the actual physical memory location. The system dynamically maps this perceived address space to the real physical memory addresses.
CPU Redource Allocation- The Forth (Exo)Kernel assigns CPU resources to each application, optimizing for the large number of cores available in modern CPUs. Conceptually using large pages and share noting as a means os performance gains and high level of parallelization.

IPC Mechanism**
Dedicated Stacks for IPC- Each application has a unique stack for inter-process communication, based on its process ID.
Tag-Data Structure- Messages on the stack include a tag indicating their priority, with certain values capable of triggering immediate interrupts, and a data portion containing the message.
Value Stack and RPN Operations- An additional stack calculates the combined priority of messages using Reverse Polish Notation (RPN), facilitating effective message batching.
Flag System- Each stack entry has a flag indicating its state (Free, Being Pushed, Being Popped), ensuring orderly access to the stack.
Process Query System- Each process is able to pool the CPU Resource Allocation to maintain a list of Process IDs.

Execution Model

Application Loading- The Forth (Exo)Kernel loads applications, either pre-compiled or compiled on-the-fly, and sets up dictionary entries pointing to the start of the binary blobs.
Resource Loop Initiation-A resource loop is initiated for each application, managing its execution and resource usage.

Motivation
Resource Efficiency- By allowing direct control over resources, the system is highly efficient, especially in compute-intensive environments.
Security and Isolation- The use of separate VMs for each application ensures a high level of security and isolation.
Customization and Flexibility- The integration of Forth allows for deep customization and adaptability to various hardware architectures.

Concerns and Challenges
Memory Mapping Overhead- Managing the virtual-to-physical memory mapping could introduce overhead, especially in systems with numerous processes.

** Not sure is this is new or feasible.

I have plenty of questions, but I can bring those up in the discussion.

Thank you and look forward to all the comments!

[eekee]

I'm a Forth fan, but I only just now learned what an exokernel is, so I'm not sure I understand all the implications. I'm sure Forth will "allow for deep system customization and efficient, low-level hardware interaction," though customization via code can make updating the system difficult. It can be done, it's just more work.

I don't see why you want to compile C to Forth. Do you intend to redefine @ and ! to remap memory and limit access? That could work, but will have a performance penalty. You may be able to make the penalty small, I don't know. I don't think you could use it to create a large, linear memory mapping out of the PC's hole-y architecture without quite a large perfomance penalty. I've considered doing this for smaller programs. The idea seemed attractive at first, but I had to admit that it would probably be simpler to use the MMU and certainly higher-performance.

If you do use the MMU and IOMMU, you don't need to compile C to Forth. The two languages can interoperate well, with only the opposing string standards creating minor obstacles, normally solved by adding null-terminated strings to the Forth. This makes the whole project almost infinitely easier as you don't need to create a high-performance C compiler in addition to an OS.

[jemo07]

Hello eekee,

Yes, I can see you are a Forth fan, good to greet you and thank you for your feedback. Ok, a little background around the motivation of C to forth, simple’s way to put it is, there are more C programmers than Forth programmers.

However, looking at this form another prism, if you follow my concept to it’s most extreme potential, you would be looking at an OS that could be the basis for a massive parallel application platform, where each application includes a subset of the forth kernel… and you potentially can have thousands of applications running all at a very small footprint. You would need a C interface or otherwise to build these apps, and they can be sort of like Lambda functions in their simplest forms… if there would be any chance at getting any adoption, I would need to target a broader developer base than Forth. Makes sense?

Think of it as dockers yet that infrastructure to run most of it would be like 10/20 megabytes at the os level and about a meg each little kernel. It’s paravirtualization at its most simplistic form. Currently I have only considered share nothing. But threads can be a better reality.

It’s kind of like that movie, if you build it they will come… I can build the greatest App arena, but If I can’t find developer, then I will not see a single tenant in that arena…

Now, I’m no where near building a functional OS yet, I have the foundations to create loops dedicated to a single core and working on a binary format for the application, using simple UTF-8 rapers to build a compiled dictionary…

I would love feedback on the IPC for these little VMs.

[eekee]

I still don't see the sense in compiling C to Forth unless you're using very simple CPUs without an MMU. Using an MMU will make memory access faster than overloading Forth's fetch and store, and it will permit compiling C direct to machine code. What would you gain by ignoring the MMU? I've considered every way to do without an MMU, and they're all more complex or at least slower. It's all right for a toy OS, but you're talking about developing a compiler to attract programmers.

I don't feel qualified to comment on the IPC. It's new to me, so I'd have to have a really long think about it, and I'm just not feeling up to it.