Implementing an Operating System (The LowEST Kernel)

[~]

NOTE: Absolutely all of my programs and my source code are Public Domain, so they can be used freely and have no author restrictions at all.

I'm currently making a full text recording of my previous kernel code ( LowEST_Kern_2012-07-03.zip, found here).

The link to the ZIP file above is regular x86 assembly source code. Now, the following link through clicking the image is the recording of the source code character by character so as to make it dramatically more self-explaining. Of course I'm still learning to organize the lowest and most complex details of OS development, so this is an incipient work that seeks to improve all the documentation around in this same way to make a sort of a hierarchical tree of source code.

With this we don't just have the sequential code but we can always see what part of the code came to exist first, and then which others to make obvious the sequence of dependencies at the code line level.

By now it just contains the raw skeleton of the kernel to simply halt inside an infinite loop, and a boot sector code for FAT12, but it should be enough to demonstrate how to actually start creating an operating system (at leaast I haven't seen any text recording of a kernel or of any other code snippet that actually contains all the text, the typing sequence including typos and corrections, and the directory tree).

I guess it will be well received by most beginners and could be very useful to show the logical dependencies of all the content of the wiki and other references with a much finer "granularity".

LowEST_Kernel Level 1 (Just Press Play)

And then an archive with the individual code files:
LowEST Kernel (containing all raw files inside a TAR, including the text recording.)


LowEST Kernel stands for "Low Executive System Tests Kernel". I call it "executive" instead of operating both for the matching mnemonic and because it is too basic to be considered a proper operating system. It's more a platform to play around with I/O ports, memory and for loading custom programs to do that with a small, raw kernel call export interface.

By the way, the next thing to add would probably be the code to clear the screen and to write text in the standar 80x25x16 color text mode. Of course I will need a lot of help but it makes things much more resumable, scalable, achievable and clearer to actually being able to do so and to fulfill the goal to make a truly understandable kernel or application source code (I just hope there was a better text recorder integrated into all development, audio and graphics renderization suites instead of having to write it in JavaScript as a barely useable program).

[onlyonemac]

I'm not sure if you use Windows or Linux, but if you use Linux you might want to look into the "script" utility (particularly with the "-t" option to record timing information) which will record a terminal session although it doesn't work with curses-based applications (such as vim, nano, less, etc.). Nevertheless it's good for recording text and if nothing else it might be good to adopt the same file format as it is somewhat standardised and can be easily replayed on a Linux system (and I believe, although it's been a while since I last used it, that the timing file is in a fairly easy-to-parse format so writing e.g. a javascript player for it shouldn't be too difficult).

[~]

I'm not sure if you use Windows or Linux, but if you use Linux you might want to look into the "script" utility (particularly with the "-t" option to record timing information) which will record a terminal session although it doesn't work with curses-based applications (such as vim, nano, less, etc.). Nevertheless it's good for recording text and if nothing else it might be good to adopt the same file format as it is somewhat standardised and can be easily replayed on a Linux system (and I believe, although it's been a while since I last used it, that the timing file is in a fairly easy-to-parse format so writing e.g. a javascript player for it shouldn't be too difficult).

I use Windows/MinGW/Cygwin more than Linux unless I'm in a stage of heavy learning from other sources that are more advanced and that require the Linux toolchain.

Currently the program just keeps track of 16 fields. Some of the most important are:

• The milliseconds between typed keys or actions such as adding directory entries.
• The start and end cursor position and the horizontal and vertical scrolling (to show also the text selections that are made).
• The ID of the action (be it "keypress", "delete" -for text characters or selection-, "fileinsert", "directoryinsert", "direntselect").
• The character, string or metadata for the action.
• The mouse position and click (unused by now but useful to implement mouse actions and a more or less simple drawing viewer/editor)
• The order in which files were created, and what file is selected/opened from a given file and pointer (when manipulating or selecting directory entries).

Using the format of the "script" program could be a good thing mostly to abbreviate the size of the data format used from JavaScript (apart from using Typed Arrays), but it could probably be better to have a suite that can add and create directories and files without using the names of binaries that are too specific.

The best thing would be to have a master format that can later be used from a web application, a Linux or Windows command line shell (would most likely be limited to text only), or a proper development suite.

The advantage of it is that it can be viewed in any browser, and it can generate TAR files on the fly with a function written in JavaScript to write USTAR-formatted archives, so the application provides a text recording and the option to extract all individual files to compile them regularly, which are things that should be possible to convert to other text recording formats later.

It looks like a "text-mode video" format and codec should be specified and implemented formally by some standards group to work as a proper recording format to be used by any text and mouse/touchscreen/drawing tablet/multimedia application in a way that can be reproduced back in any other program interchangeably as a driver/plugin, even by JavaScript or Linux shells.

[~]

I got the idea that for big project like the Linux kernel, libraries and toolchain (probably doing an extreme compile of Linux From Scatch) we could start with the very first version, no matter if it is a "test" or "incomplete" version.

Then we can find the immediate next version or sub-version, make a diff on both directories searching for existing and new files, their size difference and the difference between both files versions (with diff).

We can do it with the following projects' source code to enrich our learning:

- MenuetOS
- VisOpSys
- PDCLib
- FreeDOS
- Firefox
- Snippets at the wiki, Geezer's website, OSDever.net, and any other OS-related resource on websites or books
- Books like "C By Discovery" and any book containing source code
- Anything for which you can get the first version of, or at a very small size and complexity, and increase over versions, scanning with diff and other file/directory compare tools to later type it again manually.


It seems to be a huge task, but it seems a much clearer path than trying to decipher the monoliths that are any final versions or programming problems/projects of the past decades directly.

[onlyonemac]

I'm not sure it's necessary to re-type everything manually; can't we just "simulate" typing? Isn't this more or less what version controls systems like Git do already, to keep track of what lines of code came first?

[~]

I'm not sure it's necessary to re-type everything manually; can't we just "simulate" typing? Isn't this more or less what version controls systems like Git do already, to keep track of what lines of code came first?

Unless there's a way to reflect exactly the real sequence in which the program was created as to show the logical dependencies of the lines of code and the files through their order of creation, if we don't type everything again, but not sequentially from the beginning to the end of file but in a logical and meditated way, it is always so much more illustrative, understandable and educative to have both the regular code and the sequenced order of creation of everything.

Even the developers themselves would be able to refresh what they were doing years after with this feature.

Just think about how fast newcomers to a project could learn if they could see how everything was created from the first character of the first file of the first test version, with breakpoints to test each feature and subsystem (menus, scripting engines, rendering engines, etc...).

It's better than a video because we have access to actual source code at all times of the text recording.

It would be like having the developers in front of every computer apply their programming and editing style, be it a single developer or several developers reflecting changes (which we could always see, and we could select replaying the edition of 1 or several simultaneous developers) so it is incomparably more rich than just reading a changelog, although with the manual typing/automatic recording method I propose or with a similar one, we can get near to that same result for projects that have a long time to exist.

It's smarter and more realistic than trying to start studying the current version which may contain hundreds of megabytes of code against some Kilobytes or Megabytes for the initial version, without untangling the logical precedence/dependencies inside a source code file itself.

In fact, the change logs that currently exist could be used for this, but we always need to use diff on files and directories, etc...

[onlyonemac]

I'm not sure it's necessary to re-type everything manually; can't we just "simulate" typing? Isn't this more or less what version controls systems like Git do already, to keep track of what lines of code came first?

Unless there's a way to reflect exactly the real sequence in which the program was created as to show the logical dependencies of the lines of code and the files through their order of creation, if we don't type everything again, but not sequentially from the beginning to the end of file but in a logical and meditated way, it is always so much more illustrative, understandable and educative to have both the regular code and the sequenced order of creation of everything.

I understand that your intention is to show the exact order of characters, but I doubt that the developers of any of your proposed projects will be able to remember exactly what order they typed things in.

[~]

I'm not sure it's necessary to re-type everything manually; can't we just "simulate" typing? Isn't this more or less what version controls systems like Git do already, to keep track of what lines of code came first?

Unless there's a way to reflect exactly the real sequence in which the program was created as to show the logical dependencies of the lines of code and the files through their order of creation, if we don't type everything again, but not sequentially from the beginning to the end of file but in a logical and meditated way, it is always so much more illustrative, understandable and educative to have both the regular code and the sequenced order of creation of everything.

I understand that your intention is to show the exact order of characters, but I doubt that the developers of any of your proposed projects will be able to remember exactly what order they typed things in.

The matter is that being able to replay/reenact the very development sequence without leaving any important detail it is vitally important for an OS project. It allows us to save an incredible amount of effort and energy and understand what we were doing even after months of years of "shelving" the code.

The idea is to have the standard and existing IDEs or a suite with a text recorder implemented, like having Eclipse or Qt record not just the typing sequence but the drawing of icons, the edition of different resources and scripts, and of course even the order the files are created in.

In this way we could distribute a TAR file containing both the individual files and the encoded recording in such a way that we could even reproduce the whole project by playing the recording and see how the developer used the suite.

It could be some sort of special PDF/Carousel/PostScript file, or it could use a new format with lossless compression algorithms to make the recording smaller.

It's true that they mostly don't remember which order they did things in, but such IDEs could save in that format as part of their main project files. In this way those project files would be much more interesting.

But think in practical terms and see how if they were to type them again, they would have to follow the logical precedence in order instead of writing them sequentially. It is much clearer and allows finding bugs and make all the above possible.

It could be possible for any of us, or even me, to type the code from the very first test pre-pre-alpha version following the logical sequence, starting by the simplest code, files and subsystems and finding the points where I can get to test the code without coding everything again, and then mark breakpoints in the recording along with comments explaining what we can achieve up to that point.

Then we can use diff, change logs and maybe some other cumulative subversion metadata resources I'm not aware of right now, and create a recording per version.

The Text Recorder program I have made can do most of that, but I'm currently trying to mnake the recording as tiny as possible (I need to implement at the very least the LZW algorithm used for GIF files to encode the JavScript text stream containing the whole sequence as if it was an 8-bit -256-color- pixel array).

Then anyone, including me, could get to sequence interesting source code of projects to enrich a little bit the understanding of that technology.

But it would be much better to make the existing IDEs use a standarized text recording format along with standarized commands (copy/cut/paste, draw sequences, create/rename/delete/copy/move files or directories, etc..........) so that every single created project contains single-developer or simultaneous-developer recording sequences, so that we can see as well the development of a single file or of the whole project.

Again, the Text Recorder I have done is already capable of mostly all of that, but it's a JavaScript application and it needs lots of size, speed and capability optimizations, but it is so vital to make even abandoned rewrites from scratch of a kernel a treasure that shows how we were thinking.

At least it is also useful publishing the tools that this accompany this OS project, but what is good is that I don't need to wait for that OS to run properly for its tools to be useful, and they are generic enough as to be used in something else as well as any great idea that could be better used by someone else than myself.

[apamment]

The matter is that being able to replay/reenact the very development sequence without leaving any important detail it is vitally important for an OS project. It allows us to save an incredible amount of effort and energy and understand what we were doing even after months of years of "shelving" the code.

Isn't that what comments are for?

Maybe I am missing the point, but wouldn't it be easier to write a simple comment for the future me saying what each part is meant to do that have future me watch hours of the old me typing out code?

[~]

The matter is that being able to replay/reenact the very development sequence without leaving any important detail it is vitally important for an OS project. It allows us to save an incredible amount of effort and energy and understand what we were doing even after months of years of "shelving" the code.

Isn't that what comments are for?

Maybe I am missing the point, but wouldn't it be easier to write a simple comment for the future me saying what each part is meant to do that have future me watch hours of the old me typing out code?

Any file could be selected and we would replay from there. Then the directory tree would also be reconstructed only showing the files that were created before the selected file, which gives more information on which files the selected one depends on and which ones come later.

I would be glad to see hours and hours of a text creation of the Linux kernel, Firefox or absolutely any other open source project because I know that it would fill my time with the study of an information that is much more fine-grained and of immediate deep learning than standard comments.

I could resume the recording later anyway, so in any way or another, the fact of having to sequence the code in its logical dependency/creation order of files and code would need to be done to understand any big project.

[Combuster]

The main point is to give insights in how developers think rather than what they want you to see in the end.

I think it's an interesting concept. Would've been nice to try it on the Java compiler I was writing for the wiki, but that's a bit too late now

[~]

The main point is to give insights in how developers think rather than what they want you to see in the end.

I think it's an interesting concept. Would've been nice to try it on the Java compiler I was writing for the wiki, but that's a bit too late now

I have uploaded the latest version of that program. It has 410 Kilobytes.

The previous versions produced brief recordings that could reach around 750 Kilobytes. Now the same recording (for example, the root directory and a directory containing a small text file) is around 420 Kilobytes.

The next thing to do is adding simple things like showing the size of the selected file:

http://archive.org/download/LowEST_Kern ... order.html