Kernel Design - Every thing a database.

[BASICFreak]

I'm looking for constructive criticism...

I'm a database programmer (almost from the start) so most of my code is structs and I have adopted a lot of knowledge from my PHP & MySQL days.

The way I see it is:
The file system is a database of where files are located on the disk.
The GDT is a database of operating modes and segments
The IDT is a database of Int Handlers
The HAL is a database of drivers
The VFS is a database of File Systems (and partitions in my case)
The Video Driver is a database of pixels (though this is not a struct in my case)
Even the disk is a database of sectors (though again not a struct)
The PMM is a database of available RAM
The VMM is a database of pages
The HMM is a database of page segments

The unmade parts:
The Thread Manager is a database of threads
The Message System is a database of messages (really adopted from my php days - the same way I did private messaging in all my online apps, Src, Dest, Msg)
The Keyboard itself is a database of keys, but the driver is a database of key-presses / key-releases (in FIFO style though it will be offset-able)
The Mouse driver is a database of mouse actions (movements, clicks, scrolls, etc.)

(I think you get the point...)



So, I'm curious if anyone can give me a situation where the computer is not just a bunch of databases; if so I may have to redesign my backend.

Yes, I do like databases - I understand them so well...

[SpyderTL]

I think we all tend to think of the OS in terms that we are comfortable with. For instance, all of the components in my OS are modeled after C#/Java/JavaScript objects (Classes, Methods, Fields, References, etc.).

In reality, none of this really reflects the physical system. They are just logical representations to make it easier for us to understand and document.

But, specifically, I think the term "database" implies that you can "index" items and "query" them in various ways.

But, whatever helps you keep track of everything in your head, should be fine. I will mention that a lot of the structures that you need to build in memory have a very specific format, and will only work if everything is exactly as the hardware expects. This means that you won't be able to store all of your "databases" in the same format, so this may affect your decision.

Let us know what you decide. I'm always interested in new OS design ideas.

[BASICFreak]

...But, specifically, I think the term "database" implies that you can "index" items and "query" them in various ways.
... I will mention that a lot of the structures that you need to build in memory have a very specific format, and will only work if everything is exactly as the hardware expects. This means that you won't be able to store all of your "databases" in the same format...

That to me just seems like an abstraction layer, such as - Let us take a FDC for example:

The FDC driver by itself is not a database, the FDC requires the use of port I/O (just like almost every device in a PC).

But the interface that is seen would be something along the lines of (I'm going to use SQL here to show a point):
SELECT * FROM `FDC` WHERE `SECTOR` >= `3` AND `SECTOR` <= `5`;
and assuming the HAL uses SQL (which I do not) it should return a buffer of sector 3, 4, and 5.

And yes to be a true database, I would have to have search features in place, but that couldn't be too hard (should be easier than writing an OS lol)

Now lets take the GDT as an example:
Everyone knows the struct for GDT

Code: Select all

typedef struct gdt_entry
	{
		uint16_t limit_low;
		uint16_t base_low;
		uint8_t base_middle;
		uint8_t access;
		uint8_t granularity;
		uint8_t base_high;
	} __attribute__((packed)) x86_GDT_ENT_u, *x86_GDT_ENT_p;

Now if I wanted to add a GDT entry I could use something similar to:
INSERT INTO `GDT` SET `Base` = 0, `Limit` = 0xFFFFFFFF, `ring` = 3.....
and the layer of code that is seen will change the values to represent the hardware dependent structure.

H311 most of us use the following (or similar):
InstallGDTRing(uint8_t num, uint32_t base, uint32_t limit, uint8_t access, uint8_t gran);
which is an abstraction layer already as the values do not go directly into the structure.



EDIT: decided to lookup the actual definition of database and here it is:

da·ta·base
/ˈdatəˌbās,ˈdā-/
noun
a structured set of data held in a computer, especially one that is accessible in various ways.

[SpyderTL]

INSERT INTO `GDT` SET `Base` = 0, `Limit` = 0xFFFFFFFF, `ring` = 3

So... would you be typing this into a text editor and then compiling it with some sort of SQL-to-ASM compiler? Or would you be typing this into a console application while the OS was running? (Or both?)

Another thing worth mentioning is that the keywords "INSERT INTO", "SELECT", "SET", "WHERE", etc. are pretty well defined across the database industry. However, what you would need is the "Schema" for all of the available "Tables", "Columns", "Views", and "Stored Procedures", and they would need to be "standardized" and well documented before you could use them.

But, on the other hand, if you could "query" the tables themselves, i.e. "SELECT * FROM INFORMATION_SCHEMA.TABLES", then you could read the metadata for the whole schema at runtime, instead of having to look everything up.

The other cool thing would be that you could "emulate" the system by building a, let's say, SQL Server database with the exact same schema, and you could test all of your "queries" and "updates" against that before actually trying it in the real system. (i.e. use SQL Server as a sort of Virtual Machine.)

It's an interesting concept. I'd like to see a list of common problems that OS developers deal with and the SQL statements that would solve those problems. Tasks, Threads, Consoles, Windows, Events, Drivers, etc.

EDIT: It also just occurred to me that remoting into this OS would be trivially easy. You might even be able to use the same network protocol as SQL Server or MySQL...

[BASICFreak]

INSERT INTO `GDT` SET `Base` = 0, `Limit` = 0xFFFFFFFF, `ring` = 3

So... would you be typing this into a text editor and then compiling it with some sort of SQL-to-ASM compiler? Or would you be typing this into a console application while the OS was running? (Or both?)

I used SQL as an example since it is well known, I was thinking of having a structure of pointers in HAL labeled something like INSERT, UPDATE, SELECT, DELETE, etc...

Another thing worth mentioning is that the keywords "INSERT INTO", "SELECT", "SET", "WHERE", etc. are pretty well defined across the database industry. However, what you would need is the "Schema" for all of the available "Tables", "Columns", "Views", and "Stored Procedures", and they would need to be "standardized" and well documented before you could use them.

And again this seems like an abstraction layer such as GDT, it has base, limit, ring, io permissions, etc., so you would use a simple database design to split up the hardware dependent struct into segments of data you require.

The other cool thing would be that you could "emulate" the system by building a, let's say, SQL Server database with the exact same schema, and you could test all of your "queries" and "updates" against that before actually trying it in the real system. (i.e. use SQL Server as a sort of Virtual Machine.)

That's a very cool idea, I wouldn't of thought of it, but I can gamble (knowing me) I would go strait to building the kernel.

It's an interesting concept. I'd like to see a list of common problems that OS developers deal with and the SQL statements that would solve those problems. Tasks, Threads, Consoles, Windows, Events, Drivers, etc.

I would too... And maybe we will...

EDIT: It also just occurred to me that remoting into this OS would be trivially easy. You might even be able to use the same network protocol as SQL Server or MySQL...

Also a very interesting idea...



I have parts of this concept in my kernel now, and am so early on this build anything is possible.

The more I'm talking about it the more convinced I am to make a proof-of-concept kernel...

[AndrewAPrice]

What is a database?

Any container type could be thought of a database. An array of structures can be thought of as a table. I guess what you're looking for is something like a registry/debugger that allows you to view and query the data structures of the operating system?

[embryo2]

The more I'm talking about it the more convinced I am to make a proof-of-concept kernel...

Unfortunately, your excitement will diminish in a few days, while the time required to write an OS in SQL is much much bigger.

But you can claim you did it by implementing at least one select for any one of your data structures. And it will show you the tremendous mountain ahead of you. Usually, the next step is to forget about the idea.

However, here are a few points about SQL OS. It requires you to implement the database engine with a compiler or interpreter for your subset of SQL. And it requires such engine to be essential for the OS and not being just another database engine for existing OSes. For the engine to be essential it is required to do some low level stuff with the help from it. And the efficiency of such approach is highly questionable, if you wouldn't create an architecture, where a highly efficient compiler can optimize the raw SQL sentences. In effect it leads to the replacement of the language the OS is written in. It will be just another OS in high level language. But there is still a way of introducing the SQL as a helper language, just like assembly is for many C based OSes. The problem with such approach is to ensure that the high level language is essential for the OS, because if we have an engine, that can be safely remove from an OS, then what is the point to claim the OS is being built somehow differently?

[iansjack]

I can't really think that the OP means to write an OS in some dialect of SQL. And if not, saying "everything is a database" is pretty meaningless, really. Yes, you can regard the various OS structures as databases if you like, but it's just words; they are certainly not normalized relational databases (which would be what people generally understand by the term "database").

Let's keep it simple. An OS is just a collection of bytes, and any collection of objects can be thought of as a database. True, but ultimately useless, information.

[BASICFreak]

Well guys, I did never mean for this to turn into a SQL Kernel, just something that allows all devices to be accessed in a standardized way - I only use SQL as example since an 8 yo can read it.

I would make my own standard, and probably way less text phasing - which would be hard to do before MM and IDT.



I think I'm taking the day off, already booted into Windows and about to open Morrowind. Been working too hard on my Kernel all week, and what better day to take off than the shabbat...

[Rusky]

This reminds me of Linux's /sys and /proc interfaces for getting system information. I think a more uniform interface on top might be useful in some situations, although I don't know that it would make sense to include any of the traditional database tools in the kernel. Maybe just design the kernel interface to be amenable to database-like queries?

[AndrewAPrice]

It probably wouldn't be that hard. What you're essentially saying is that you want some kind of internal reflection into the operating system'so data structures during runtime.

Perhaps give every object a to_json method that turns it into JSON format. Then you can use or build a tool that allows you to query and view JSON data. You could even have a high level to_json that prints pretty much every single variable and allocated structure in the entire kernel.

[eryjus]

I'm looking for constructive criticism...

I'm a database programmer (almost from the start) so most of my code is structs and I have adopted a lot of knowledge from my PHP & MySQL days.

The way I see it is:
The file system is a database of where files are located on the disk.
The GDT is a database of operating modes and segments
The IDT is a database of Int Handlers
The HAL is a database of drivers
The VFS is a database of File Systems (and partitions in my case)
The Video Driver is a database of pixels (though this is not a struct in my case)
Even the disk is a database of sectors (though again not a struct)
The PMM is a database of available RAM
The VMM is a database of pages
The HMM is a database of page segments

The unmade parts:
The Thread Manager is a database of threads
The Message System is a database of messages (really adopted from my php days - the same way I did private messaging in all my online apps, Src, Dest, Msg)
The Keyboard itself is a database of keys, but the driver is a database of key-presses / key-releases (in FIFO style though it will be offset-able)
The Mouse driver is a database of mouse actions (movements, clicks, scrolls, etc.)

(I think you get the point...)



So, I'm curious if anyone can give me a situation where the computer is not just a bunch of databases; if so I may have to redesign my backend.

Yes, I do like databases - I understand them so well...

My first real job was programming on an AS/400. Most of the OS data on an AS/400 (now known as "IBM i") are stored in tables. In fact the object file system actually operates like a database itself. The list of running jobs are stored in tables which are accessible from APIs. Job queues are also tables, and are accessible from APIs. There are far too many APIs provided to discuss, but the point would be that these APIs either access data mostly stored in databases or data areas (think of it as a single record table). Even message queues are stored as special purpose tables with APIs to access them. In short, everything is an object with its data stored in system tables.

So, to directly respond to your question, yes it can be done.

[intx13]

From the other replies so far it sounds like there's two ways to look at this:
1. The operating system is a normal operating system (whatever that is!) and exposes its well-defined interfaces in the form of table-oriented databases.
2. The operating system is entirely built around a database model of computation.

In (1), the kernel contains all the usual x86 object code you'd imagine and does things like hardware abstraction, multitasking, memory management, interrupt handling, etc. When userland programs (and potentially the kernel itself) want to access these things, they do so via table-oriented database-style interfaces, using commonly understood database-style queries.

In (2), a database is not just a representation of the kernel's interfaces, but its model of computation. For example, maybe every hardware device is a database and a program/kernel driver is nothing more than a series of queries. Since your PC doesn't natively understand these constructs you are essentially designing a database-oriented virtual machine. At the bottom there is a kernel with the minimum drivers needed to represent raw hardware as databases, plus the interpreter/execution environment needed to execute queries-as-programs. Then you can write a full-blown OS (multitasking, inter-thread communication, driver architectures, etc.) in a query language on top of that virtual machine.

There's nothing really stopping you from implementing (1). You could start by taking an existing kernel and rewriting some mostly-standalone component of it (for example, maybe the PCI driver) so that it presents a database-esque interface rather than whatever set of callable functions it currently uses as an interface. I think the problem you're likely to run into is that many components of a kernel just don't lend themselves to being interfaced with as a database.

The issue I see with (2) is that a database is not a very powerful model of computation. I suppose it could be, but I think most databases provide very little computational power, instead expecting that they'll be interfaced with via some full-blown programming language.

Well guys, I did never mean for this to turn into a SQL Kernel, just something that allows all devices to be accessed in a standardized way

Ultimately, though, all devices are not similar enough to be accessed in a standardized way. Sure, maybe for diagnostic output to the user or for configuration, but I don't think there's a single "simple" representation that's suitable for things as distinct as task scheduling and interfacing with an AHCI driver, for example.