Data/Resource Manager

[zeitue]

I have an idea to have a to sever that keeps track of all the data and configurations.

* Locations
* Passwords
* Settings
* Environment Variables
* Available software
* Localization
* Search Indexes
* Data Types/MIME
* Services/Associated programs
* Store all information in formats: XML, YAML, JSON, S-expression, and binary.

I'm not sure on the format?
Is this a possible idea?
would this have too much overhead?

[Brendan]

Hi,

Is this a possible idea?

Yes.

I'm not sure on the format?

In the most generic sense, there's 2 kinds of data - unordered data and ordered data.

For unordered data the order doesn't matter. Typically you'd arrange the data as a table of values where the "tag" is the column name (and the row is irrelevant). In this case you search for the data you want (e.g. "find all rows where the value in the "foo" column is less than 123"). Examples of this are almost every database.

For ordered data the order matters. Typically you'd have a tree of "tag and value" pairs, and you'd navigate from one "tag and value" pair to another using things that make sense for trees (e.g. "select this node's parent", "select this node's next sibling", "select this node's first child", etc). Examples of this are XML, YAML, JSON, S-expression, etc.

For an example of unordered data:

Code: Select all

Name    Address             Goats   Chickens
"Fred"  "88 Eighth Street"  23       0
"Jane"  "99 Ninth Street"   (null)   44

You can store unordered data as ordered data. For example:

Code: Select all

<people>
   <entry>
      <name>Fred</name>
      <address>88 Eighth Street</address>
      <goats>23</goats>
      <chickens>0</chickens>
   </entry>
   <entry>
      <name>Jane</name>
      <address>99 Ninth Street</address>
      <chickens>44</chickens>
   </entry>
</people>

You can also store ordered data as unordered data. For example:

Code: Select all

ID     Parent    FirstChild   NextSibling  Tag        Value
002    (null)    004          (null)       "People"   (null)
004    002       005          010          "Entry"    (null)
005    004       (null)       006          "Name"     "Fred"
006    004       (null)       007          "Address"  "88 Eighth Street"
007    004       (null)       008          "Goats"    "23"
008    004       (null)       (null)       "Chickens" "0"
010    002       012          (null)       "Entry"    (null)
012    010       (null)       014          "Name"     "Jane"
014    010       (null)       013          "Address"  "99 Ninth Street"
013    010       (null)       (null)       "Chickens" "44"

Storing unordered data as ordered data means that all the "tag" values get repeated everywhere and it becomes hard to store any indexing. This is just plain bad (for space and performance).

Storing ordered data as unordered data might not be bad for space (the extra ID, parent, first child and next sibling fields are likely to cost less space than the original "raw character" data) and might also remove the need for a lot of parsing work. It would make navigating the data slower though. For example, rather than parsing the data and keeping it in RAM as structures (with pointers) you'd be using searches (e.g. to navigate from one item to another, rather than doing something like "currentItem = currentItem->nextSibling" you'd search for an entry that has "ID = currentItemNumber" and get that entry's next sibling field). Of course once you add decent indexing the performance difference could be quite minor. Also note that this only applies if the data is small enough to fit in RAM (ordered data that is too large to fit in RAM is a performance disaster, and converting it to unordered data is likely to improve performance).

If you add all this up; you either end up with 2 different data formats (one for unordered data and another for ordered data) and an overcomplicated mess, or you just convert the ordered data into "unordered data format" and implement a database management system.

would this have too much overhead?

Overhead is relative - you'd want to ask if it would have more or less overhead than a specific alternative when used for a specific purpose. If you're planning to use it to store a function's local variables (rather than stack space) then the overhead is going to be hideous. If you're planning to use it to store applications' configuration (rather than "plain text file, with parser hacked together by application developer") then it might be a lot less overhead.


Cheers,

Brendan

[zeitue]

Hi, Brendan Thanks, I have a greater understanding of how database and configuration storage works

I have one question of this, you speak of unordered data and ordered data.
So are they unordered and ordered in memory(RAM) or when being stored on the Hard Drive or both?

And out of XML, YAML, JSON, S-expression, which do you really think would work for my idea as the main format?
One of the goals is to make it so the configurations are human readable and yet keep speed up; allowing the Resource Manager to give information to any that need it (Programs, Servers, localized strings).

Thank you

[bluemoon]

So are they unordered and ordered in memory(RAM) or when being stored on the Hard Drive or both?

You probably want to have ordered and structured data after loaded into memory for easy & quicker access.

And out of XML, YAML, JSON, S-expression, which do you really think would work for my idea as the main format?

I suggest not to tie your implementation with the design.
You may, however, pick any format that is usable within your design requirements/limitations.

One of the goals is to make it so the configurations are human readable and yet keep speed up; allowing the Resource Manager to give information to any that need it (Programs, Servers, localized strings).

Do you think it's important for it to be human readable with a text editor? On the other hand, how about storing data that is not text - do you need to convert it to base64 or escape sequence?
There are much more room for design if you provide a reader application and store it in binary format.

[zeitue]

What I was thinking by human readable, was that yes they could be edited by a text editor.

EXT=xml/yml/json/ss

But I figured it like this
#Kernel settings
/System/Resources/Configurations/Kernel.EXT
#Default programs settings
/Programs/$APPNAME.app/Resources/Config.EXT
#Program settings
/Homes/$USERNAME/.user/Resources/Configurations/$APPNAME/Config.EXT

To have all the files on the system in a format like so XML, YML, JSON, S-Expression, then the Resource Manager will index and convert these files into one binary file and one for each user. It will read the files instead of the human readable ones.

#System
/System/Resources/Configurations/Config.bin
#User
/Homes/$USERNAME/.user/Resources/Configurations/Config.bin

If the Config.bin files get corrupted we just regenerate them; so it won't have Windows like errors

The resource manager would be notified if a setting changed in the Configurations folders and it would update its Config.bin

as far as the formats for the human readable side, I intend to use the format as like a backend plugin, with the Resource Manager acting as the front end or face.
I'm not sure what format would allow for the settings to be stored well though.

[Brendan]

Hi,

I have one question of this, you speak of unordered data and ordered data.
So are they unordered and ordered in memory(RAM) or when being stored on the Hard Drive or both?

Typically on the hard drive it's unordered (e.g. plain text, like XML, etc); and while you load it into RAM you parse it and generate a "tree of structures". For example, the structures might look like:

Code: Select all

struct myDataStruct {
	struct myDataStruct *parent;
	struct myDataStruct *firstChild;
	struct myDataStruct *nextSibling;
	char *tag;
	char *value;
}

If you have a look at this you'll notice that it's similar to the "ordered data" example I gave earlier. The only differences are that it uses pointers (instead of unique IDs) and that it doesn't need to be stored in a table/array.

And out of XML, YAML, JSON, S-expression, which do you really think would work for my idea as the main format?

That mostly depends on what you're storing.

One of the goals is to make it so the configurations are human readable and yet keep speed up; allowing the Resource Manager to give information to any that need it (Programs, Servers, localized strings).

The idea of "human readable" is a joke - all data is just bits of 0s and 1s. Some sort of application is required to make those 0s and 1s make sense. That application could be something that's designed to be good for the purpose; or if you're lazy and don't mind large amounts of bloat (and slower than necessary parsing) it could be a generic text editor.

If you're using this for (e.g.) application configuration, then I'd use a custom designed binary format and provide a "good for the purpose" viewer/editor, as this reduces file sizes and bloat, and avoids a lot of overhead for parsing; and means that the viewer/editor can be far better (e.g. could be part of a system designed to show the user dialog boxes, context sensitive help, etc. rather than merely "lines of syntax errors"). I'd refuse to tolerate something as bad as plain text editor.


Cheers,

Brendan

[zeitue]

Hi, Brendan, thanks for the reply.
I get your point on the data types and how all data is just 1s and 0s.
Maybe I am biased against binary configurations, because of Windows Registry?
The resource manager could be the only interface to the settings. It could be set up so you would call the Resource Manager something like this.

Code: Select all

#include <ResourceManager.h>

RS=ResourceManager.get("$APPNAME");
RS.get("tag");
RS.set("tag",value);
RS.save();

but I still would worry about having everything in one system, and one for each user, binary files.

though if I just parsed XML and stored all the settings in the XML then just forgot the binary; it might be much slower.

[Nessphoro]

Damn, that looks awfully a lot like an API I'm working on (it is actually working - just need a proper interface) - its a web API for programs to store their data in the cloud with an interface as your just showed.

[zeitue]

Damn, that looks awfully a lot like an API I'm working on (it is actually working - just need a proper interface) - its a web API for programs to store their data in the cloud with an interface as your just showed.

That's cool this would be similar to your storage API I guess; it does store settings and data like localizations

[Brendan]

Hi,

I get your point on the data types and how all data is just 1s and 0s.
Maybe I am biased against binary configurations, because of Windows Registry?

If Windows Registry was stored as XML rather than binary, would that fix any of the problems with Windows Registry?

but I still would worry about having everything in one system, and one for each user, binary files.

I'd worry about that too. For example, it makes it hard to copy one application's settings from one computer to a different computer (without a whole pile of additional/unwanted stuff).

I think it'd be better to define the goals of the system (what it should achieve), and then worry about how it achieves those goals afterwards.

For example; you might want 2 sets of settings - the default settings (for all users) plus the current user's settings (where the user's settings override the defaults, and if the user happens to be using the defaults then there's no reason to store the same data twice). You could add a few generic rules to this (e.g. make it so that only the system administrator can modify default settings); and the OS's own settings might be handle slightly differently (e.g. no user settings at all).

I'd also have separate files for different applications. This means that you might have an application called "foo", with a "/sys/default/foo.cfg" file containing the default settings, plus a "/home/Brendan/.config/foo.cfg" file for my settings and a "/home/zeitue/.config/foo.cfg" file for your settings. If you want to copy your settings for application "foo" from a computer at work to a computer at home then you'd copy your "/home/zeitue/.config/foo.cfg" onto a USB stick or something and then copy it onto your computer at home. If you really wanted to get fancy, you could use something like NFS and mount the same file system at "/home/zeitue/" on several different computers (so if you change your settings on one computer then use a different computer, the other computer uses the new settings).

Next, (regardless of how settings are stored - binary or plain text) I'd want something to describe what different settings mean. For this reason I'd want applications to provide another file that describes settings. For example, an application called "foo" might have a file "/apps/foo/config.desc" that says things like:

Code: Select all

  value1 = "This is the timeout (in milliseconds) to wait for network connections before retrying."

This means that you can have some sort of utility (even if it is a fancy text editor) that can display the description of different settings when the user hovers their mouse pointer over it or something. Of course if you want internationalisation, then you might have a different file for different languages - one for English descriptions, one for French descriptions, one for German descriptions, etc.

Next, I'd want something to determine acceptable values for settings. For example, an application called "foo" might have a file that says "the data for 'value1' is an integer that must be positive, and must be between 1 and 100". That way you can have some sort of utility (even if it is a fancy text editor) that checks if values entered by a user are actually sane (e.g. and tells the user there's a problem when they want do "value1 = -8" or "value1 = chocolate"). Let's call this a "setting limits" file.

After this I'd consider is how the "configuration editor" will look to end users. I'd really want something graphical (e.g. dialog boxes), with things like text boxes, drop down lists, checkboxes, etc. The "configuration editor" can use information from the "setting limits" file to determine how to generate the dialog boxes (e.g. if something should be a text box or drop down list or whatever), and can use information from the "description file" (for whichever language) to provide a help system and maybe even the names of variables (e.g. "value1" might be called "Network Timeout" in English). It'd also need to be able to determine if a user's setting is the same as the default setting. For example, the "configuration editor" could load values from the default file, then load values from the user's file to override the defaults; and later (when saving after changes were made) could find the difference between the new data and the defaults and only store the differences.

After deciding how the "configuration editor" will look; I'd decide on file formats. There's 3 different types of files to worry about - configuration data itself, "description files" and "setting limits files". The description files could be plain text (and possibly should be plain text, as this makes it easier for internationalisation). For the "limits files" (created by application developers), I can't see any reason to use text files - application developers are smart enough to (e.g.) write a set of macros to generate their limit files (if you don't provide the macros), and eventually I'd hope someone builds it into a fancy IDE of some sort to make it really nice for developers. For the settings, the end users would be using your configuration editor so they won't need plain text at all, and software will be using your data/resource manager (e.g. application "foo" might ask your data/resource manager to find the current user's data for "value1") so software won't need plain text for anything either.


Cheers,

Brendan

[skeen]

I really like your idea, even tho you'll seem to end up with a single point of failure.
That being said, I must admit I like having text configuration files, and I do think this is the way to go, as it allows people to use whatever editing tools they like, Even custom IDEs or GUIs. The format of these files should mainly ease editing, and parsing.

If performance is an issue, then you can always compile your configuration files into a binary format. Obviously this binary will need to be up to date then. The format of this/these files should mainly focus on performance, and mostly lookups.

Note: This solution is mainly useable if you do a lot of reads compared to writes of configurations.

What you could do alternatively is to store the information in a rationel database, these are providing very good performance, even with shifting numbers of reads and writes, and these will also be able to enforce constraints for you.

[dozniak]

What you could do alternatively is to store the information in a rationel database, these are providing very good performance, even with shifting numbers of reads and writes, and these will also be able to enforce constraints for you.

Can you tell more about it? How do the rational databases provide good performance and what's the technology behind it?

[Combuster]

How do the rational databases provide good performance and what's the technology behind it?

Relational databases have billions of dollars invested by corporate and academic people alike. They're huge complex beasts you won't be able to understand to the core even in a year of study.

Ever worked with commercial oracle, or the public postgresql?

[dozniak]

Ever worked with commercial oracle, or the public postgresql?

Yes, I use postgres and mysql daily. I was wondering maybe rational databases is something newly researched. This looks like just a typo now.

[zeitue]

I have decided to store the user configurations in single files in the users Home Folder /Homes/$USERNAME/.user/Resources/Configurations/$PROGRAMNAME/config.$EXT
but I'm going to have the core system settings stored in both formats text type /System/Resources/Configurations/$FILENAME.$EXT and binary /System/Resources/Configurations/System.bin

I really like your idea, even tho you'll seem to end up with a single point of failure.
That being said, I must admit I like having text configuration files, and I do think this is the way to go, as it allows people to use whatever editing tools they like, Even custom IDEs or GUIs. The format of these files should mainly ease editing, and parsing.

If performance is an issue, then you can always compile your configuration files into a binary format. Obviously this binary will need to be up to date then. The format of this/these files should mainly focus on performance, and mostly lookups.

Note: This solution is mainly useable if you do a lot of reads compared to writes of configurations.

What you could do alternatively is to store the information in a rationel database, these are providing very good performance, even with shifting numbers of reads and writes, and these will also be able to enforce constraints for you.

thanks I figured I would update the binary when a setting changes in one of the files.

I'm for sure not using YAML I know now, but I'm not sure about the rest

If Windows Registry was stored as XML rather than binary, would that fix any of the problems with Windows Registry?

No but I think if the Windows Registry was not binary it would be easier to work with.
I think if the registry was set to single entry then it would be hard to mess up.

This means that you can have some sort of utility (even if it is a fancy text editor) that can display the description of different settings when the user hovers their mouse pointer over it or something. Of course if you want internationalisation, then you might have a different file for different languages - one for English descriptions, one for French descriptions, one for German descriptions, etc.

Next, I'd want something to determine acceptable values for settings. For example, an application called "foo" might have a file that says "the data for 'value1' is an integer that must be positive, and must be between 1 and 100". That way you can have some sort of utility (even if it is a fancy text editor) that checks if values entered by a user are actually sane (e.g. and tells the user there's a problem when they want do "value1 = -8" or "value1 = chocolate"). Let's call this a "setting limits" file.

After this I'd consider is how the "configuration editor" will look to end users. I'd really want something graphical (e.g. dialog boxes), with things like text boxes, drop down lists, checkboxes, etc. The "configuration editor" can use information from the "setting limits" file to determine how to generate the dialog boxes (e.g. if something should be a text box or drop down list or whatever), and can use information from the "description file" (for whichever language) to provide a help system and maybe even the names of variables (e.g. "value1" might be called "Network Timeout" in English). It'd also need to be able to determine if a user's setting is the same as the default setting. For example, the "configuration editor" could load values from the default file, then load values from the user's file to override the defaults; and later (when saving after changes were made) could find the difference between the new data and the defaults and only store the differences.

After deciding how the "configuration editor" will look; I'd decide on file formats. There's 3 different types of files to worry about - configuration data itself, "description files" and "setting limits files". The description files could be plain text (and possibly should be plain text, as this makes it easier for internationalisation). For the "limits files" (created by application developers), I can't see any reason to use text files - application developers are smart enough to (e.g.) write a set of macros to generate their limit files (if you don't provide the macros), and eventually I'd hope someone builds it into a fancy IDE of some sort to make it really nice for developers. For the settings, the end users would be using your configuration editor so they won't need plain text at all, and software will be using your data/resource manager (e.g. application "foo" might ask your data/resource manager to find the current user's data for "value1") so software won't need plain text for anything either.

OK I understand what you mean; this whole idea actually came from a tool on Ubuntu called Gconf-Editor.
The whole database is stored in XML and you can edit them both by hand and by the Gconf-Editor.
I like the idea of having comments and set data types for certain things; it makes it better and more user friendly.
And I for sure love the idea of localized strings in the Resource Manager; especially cause I plan for this to be a multilingual OS.

Thank you