OSDev.org

Posted: **Tue Dec 01, 2015 5:42 am**

Brendan wrote:Previous versions of my OS didn't use bytecode. For these, the executable file format included information about the CPU it was designed for (CPU type and a 256-bit bitfield of required/used CPU features) plus some other information (how much processing, memory and communication the executable does); where all this information (plus load statistics, etc) is used by the OS to determine where the executable should be run.

If there's CPU type and features information then OS should deny to run the application on a system with lesser feature set. It breaks backward compatibility. It's bad for users to find an application useless on a notebook a few years older than the workstation where application was built.

Brendan wrote:My plan is to create enough of the OS so that a native IDE and toolchain (and AOT compiler) can be implemented on top of it; and then port one piece at a time into my own language (with my own tools); so that eventually the current version will become something that does use bytecode and AOT.

Starting with native IDE and stuff is harder. Existing IDEs are feature rich and your native IDE can take years to implement these features. So, the time spent while working with the featureless IDE can be saved in case of using an existing feature rich IDE.

Brendan wrote:For the final system; the programmer will still provide some information (how much processing, memory and communication the executable does) that's used by the OS to determine where an executable should run;

Have you decided on the form of such information? Does you new language support annotations? Or is it something else? I suggest something like a database with flexible ways of usage of it's information. Like prolog's facts database. Such kind of [meta-]database can be pervasive for the entire OS.

Brendan wrote:but the "CPU type and features" information will be generated by the byte-code to native compiler, and the OS will cache multiple versions of the final/native executable (one for each different "CPU type and features").

Because the generated code most probably won't be used on another computer there's no value in storing the executable file with it's "type and features" information. I suggest it is better to use a centralized database with all the meta-information about all the software installed (precompiled or not), so, there's no need for a tricky structures paired with executable. Just pieces of code referred from the database.

Brendan wrote:When a process is being started the OS will use the processing/memory/communication information (from the byte-code) to determine the best place for the executable to be run; but then it will examine the cache of the final/native executable versions. Mostly:
If there's already final/native executable that suits where the OS wants to run the executable; use that. This will be the most common case.

Otherwise; determine if there's something "close enough" to use for now (either a final/native executable that's close enough for the best place for the executable to be run; or a final/native executable for a different "less best" place for the executable to be run); and:
If there is something "close enough" to use; then use it (to avoid delays caused by byte-code to native compiler) but also compile the byte-code to native to suit the "CPU type and features" in the background so that it exists next time the executable is executed.

If there isn't anything "close enough" (either the delays caused by byte-code to native compiler can't be avoided or the delays are preferable to slower run-time and/or worse load balancing); compile the byte-code to native to suit the "CPU type and features" and then use that.

Well, I see here the need for JIT. Isn't it there? The environment should manage the code for every possible situation and caching every piece of a code for every situation seems to be too inefficient when there's just a simple way of creation of the code on demand.

Also it is worth to consider situations like preemption. It's when more important task takes place of the current task on a preferred CPU and the current task should be run on a CPU with lesser feature set despite of the programmer's/user's annotations. So, the JIT recompiles it and scheduler starts it on a weaker processor. It's actual for ARM processors with heterogeneous cores and may be actual for Intel processors in the future.

And what is "close enough"? If there's an AVX instruction then there must be the AVX support. No fuzziness.

Brendan wrote:For my system an application is multiple cooperating processes. All of the stuff above (to determine where an executable/process should be run) only cares about processes, and not applications, and not threads. For a multi-threaded process, each of the process' threads use CPUs that match the process' requirements.

So, the communication facilities are abstracted to the very basic level of the hardware? If there are different cores and they have a need for communication then the caching issues can impact performance, so the communication layer can be implemented with care for caches or without such care. But if the care is present then the number of possible combinations looks a bit huge to be implemented.

Brendan wrote:You're under-estimating what would be required to compete favourably against existing OSs. If people start switching from existing OSs to your OS because your OS looks better; then within 6 months the existing OSs will improve their appearance and you'll be screwed.

Yes, when you have no millions for advertising then it's hard to attract people. But may be it's a good idea to show your OS to some venture capitalists and to ask for millions for advertising? That's how the FaceGoogles were grown.

Brendan wrote:Immediately after switching to your OS users can switch back to whatever existing OS they were using before (there's almost no "switching costs" because they're still already familiar with that previous OS).

The switching cost here is the applications missed on the old platform. So, still there are some ways forward.

Brendan wrote:I'd estimate that it'd take a few years before the "switching costs" starts to work in your favour, and people using your OS don't want to switch to another OS

I have the browser switching experience and I should tell you the final decision was made because of just one feature which is important for me. And the monitoring for the availability of such feature in competing browsers is tedious, so I still use Firefox while (may be) there are better solutions available now.

Brendan wrote:Basically; your OS needs to be significantly better (not just better), then has to remain better (not necessarily significantly better) for multiple years (while other OSs are improving and trying to catch up). Ideally; you'd want multiple reasons why your OS is better and not just one, where at least some of those reasons are extremely hard (time consuming) for existing OSs to copy/adopt (or are impossible for existing OSs to copy/adopt - e.g. patents).

There's a feature set a user looks for. The feature set can be rich but useless for a particular user. And the feature set can be narrow, but very useful for a particular user. And feature sets form a universe of user preferences. Popular OSs target the most dense part of the universe. Your OS can target some sparse parts. To be the best for the dense part is too hard. To be the only one for the sparse part is absolutely possible.

Basically - there are ways and your choice is hard. But somebody on earth just must to try such way. That's how the trial and error method works. And it really works for things like evolution.

Posted: **Tue Dec 01, 2015 6:22 pm**

Hi,

embryo2 wrote:
Brendan wrote:Previous versions of my OS didn't use bytecode. For these, the executable file format included information about the CPU it was designed for (CPU type and a 256-bit bitfield of required/used CPU features) plus some other information (how much processing, memory and communication the executable does); where all this information (plus load statistics, etc) is used by the OS to determine where the executable should be run.
If there's CPU type and features information then OS should deny to run the application on a system with lesser feature set. It breaks backward compatibility. It's bad for users to find an application useless on a notebook a few years older than the workstation where application was built.

You can't say "all applications must be 32-bit" and forbid the existence of 64-bit software just because some users might have 32-bit CPUs. You'd have to give application developers the ability to provide both 32-bit and 64-bit native executables, and then you have the "user installed wrong version" problem (where the user installed a 64-bit application on a 32-bit system).

Basically it's the application developer's problem, not the OS's problem. For some features the application developer can choose to provide different versions of the executable, for other features the application developer can choose to have run-time checks (e.g. "if AVX exists use AVX else fall back to alternative code that doesn't use AVX") where the features aren't required (but may be used). Alternatively the application developer can say "the minimum requirements for this application are ....".

Of course the main point of shifting to byte-code is to avoid most of that; but even then you can't avoid all of it and you still end up with application developers saying "the minimum requirements for this application are....". For example, if a process must use 8 GiB of virtual space then it won't run on a 32-bit system regardless of whether it's byte-code or not, and if a process requires a "fast enough" CPU then it won't run on "too slow" CPUs regardless of whether it's byte-code or not.

embryo2 wrote:
Brendan wrote:My plan is to create enough of the OS so that a native IDE and toolchain (and AOT compiler) can be implemented on top of it; and then port one piece at a time into my own language (with my own tools); so that eventually the current version will become something that does use bytecode and AOT.
Starting with native IDE and stuff is harder. Existing IDEs are feature rich and your native IDE can take years to implement these features. So, the time spent while working with the featureless IDE can be saved in case of using an existing feature rich IDE.

If you want a new OS written in a new programming language there's an unavoidable "chicken or egg" problem. You must either:

Write "temporary tools" for a completely different OS, then write the OS using those tools, then write native tools for your new OS; or
Write a "temporary OS" using a completely different language, then write native tools, then port the OS and tools to your language

I've tried both ways. Both ways involve doing a lot of work twice; but the second way is easier, especially if the OS is modular (lots of small pieces that can be replaced one at a time). Of course the IDE and tools don't need to be full featured (as soon as the IDE and tools have minimal functionality you can start porting); and the "temporary OS" doesn't need to be full featured either.

embryo2 wrote:
Brendan wrote:For the final system; the programmer will still provide some information (how much processing, memory and communication the executable does) that's used by the OS to determine where an executable should run;
Have you decided on the form of such information? Does you new language support annotations? Or is it something else? I suggest something like a database with flexible ways of usage of it's information. Like prolog's facts database. Such kind of [meta-]database can be pervasive for the entire OS.

The IDE will have a nice dialogue box that programmers use to set various metadata (e.g. sliders for how much processing, memory and communication the executable does). I haven't designed the "source code file format" yet (it's going to take a few prototype IDEs and more research before I can design the "source code file format").

embryo2 wrote:
Brendan wrote:but the "CPU type and features" information will be generated by the byte-code to native compiler, and the OS will cache multiple versions of the final/native executable (one for each different "CPU type and features").
Because the generated code most probably won't be used on another computer there's no value in storing the executable file with it's "type and features" information. I suggest it is better to use a centralized database with all the meta-information about all the software installed (precompiled or not), so, there's no need for a tricky structures paired with executable. Just pieces of code referred from the database.

It's a distributed system. If you've got a cluster consisting of 5 different types of computers then you'd have (up to) 5 different versions of the final/native executable in the distributed file system.

embryo2 wrote:
Brendan wrote:When a process is being started the OS will use the processing/memory/communication information (from the byte-code) to determine the best place for the executable to be run; but then it will examine the cache of the final/native executable versions. Mostly:
If there's already final/native executable that suits where the OS wants to run the executable; use that. This will be the most common case.

Otherwise; determine if there's something "close enough" to use for now (either a final/native executable that's close enough for the best place for the executable to be run; or a final/native executable for a different "less best" place for the executable to be run); and:
If there is something "close enough" to use; then use it (to avoid delays caused by byte-code to native compiler) but also compile the byte-code to native to suit the "CPU type and features" in the background so that it exists next time the executable is executed.

If there isn't anything "close enough" (either the delays caused by byte-code to native compiler can't be avoided or the delays are preferable to slower run-time and/or worse load balancing); compile the byte-code to native to suit the "CPU type and features" and then use that.
Well, I see here the need for JIT. Isn't it there? The environment should manage the code for every possible situation and caching every piece of a code for every situation seems to be too inefficient when there's just a simple way of creation of the code on demand.

Aggressive/thorough optimisation takes a significant amount of time (especially whole program optimisation). For JIT, you're optimising at run-time and can't afford the time it takes to do optimisation properly.

embryo2 wrote:Also it is worth to consider situations like preemption. It's when more important task takes place of the current task on a preferred CPU and the current task should be run on a CPU with lesser feature set despite of the programmer's/user's annotations. So, the JIT recompiles it and scheduler starts it on a weaker processor. It's actual for ARM processors with heterogeneous cores and may be actual for Intel processors in the future.

Migrating a process from one type of CPU to another while it's running is insanely complex and extremely expensive (in terms of performance); and not because of differences in code but because of differences in data/state. I have no plans to ever support "live migration" in the OS or its IDE/tools. Note that this doesn't mean I won't support (e.g.) "check-pointing and restarting" (where a process periodically saves its state as a check-point, and could be terminated and restarted on a different CPU where it restores its state from its own previous check-point).

embryo2 wrote:And what is "close enough"? If there's an AVX instruction then there must be the AVX support. No fuzziness.

If the "best" CPU supports SSE and AVX, but the cache only has a version of the final/native executable that uses SSE (and doesn't have a version that takes advantage of AVX), then is it better to start the "less fast" version immediately or spend 5 minutes generating a "faster" version?

That's what I mean by "close enough" - it'd be guesswork based on heuristics (how much performance difference, how important does the executable say performance is, how long would it take to compile from byte-code to native again, how long will the process be running for, ...).

embryo2 wrote:
Brendan wrote:For my system an application is multiple cooperating processes. All of the stuff above (to determine where an executable/process should be run) only cares about processes, and not applications, and not threads. For a multi-threaded process, each of the process' threads use CPUs that match the process' requirements.
So, the communication facilities are abstracted to the very basic level of the hardware? If there are different cores and they have a need for communication then the caching issues can impact performance, so the communication layer can be implemented with care for caches or without such care. But if the care is present then the number of possible combinations looks a bit huge to be implemented.

Communication is done via. messages (regardless of whether it's communication between processes on the same computer or communication between processes on different computers). Processes communicate without needing to care if they're on the same computer or not.

embryo2 wrote:
Brendan wrote:You're under-estimating what would be required to compete favourably against existing OSs. If people start switching from existing OSs to your OS because your OS looks better; then within 6 months the existing OSs will improve their appearance and you'll be screwed.
Yes, when you have no millions for advertising then it's hard to attract people. But may be it's a good idea to show your OS to some venture capitalists and to ask for millions for advertising? That's how the FaceGoogles were grown.

Maybe; but I doubt any sane venture capitalist is going to give me lots of $$ for something as risky as an OS until/unless I can show/prove that it's very likely that it will compete favourably against existing OSs. I'd still need a variety of reasons why the OS is "better" (to convince the venture capitalist to give me $$).

embryo2 wrote:
Brendan wrote:Immediately after switching to your OS users can switch back to whatever existing OS they were using before (there's almost no "switching costs" because they're still already familiar with that previous OS).
The switching cost here is the applications missed on the old platform. So, still there are some ways forward.
Brendan wrote:I'd estimate that it'd take a few years before the "switching costs" starts to work in your favour, and people using your OS don't want to switch to another OS
I have the browser switching experience and I should tell you the final decision was made because of just one feature which is important for me. And the monitoring for the availability of such feature in competing browsers is tedious, so I still use Firefox while (may be) there are better solutions available now.

I think it's the same for everyone - there's always something they've grown used to, where the hassle of finding/switching to something better is too high.

embryo2 wrote:
Brendan wrote:Basically; your OS needs to be significantly better (not just better), then has to remain better (not necessarily significantly better) for multiple years (while other OSs are improving and trying to catch up). Ideally; you'd want multiple reasons why your OS is better and not just one, where at least some of those reasons are extremely hard (time consuming) for existing OSs to copy/adopt (or are impossible for existing OSs to copy/adopt - e.g. patents).
There's a feature set a user looks for. The feature set can be rich but useless for a particular user. And the feature set can be narrow, but very useful for a particular user. And feature sets form a universe of user preferences. Popular OSs target the most dense part of the universe. Your OS can target some sparse parts. To be the best for the dense part is too hard. To be the only one for the sparse part is absolutely possible.

Basically - there are ways and your choice is hard. But somebody on earth just must to try such way. That's how the trial and error method works. And it really works for things like evolution.

You're right; but it's still similar - e.g. if you target a narrow niche, your OS needs to be significantly better than alternatives (for that niche) and remain better for multiple years. Even if you can think of a new market for an OS where there's currently no competition at all, other OSs will try to move into that market.

Cheers,

Brendan

Posted: **Wed Dec 02, 2015 4:50 am**

Brendan wrote:Of course the main point of shifting to byte-code is to avoid most of that; but even then you can't avoid all of it and you still end up with application developers saying "the minimum requirements for this application are....". For example, if a process must use 8 GiB of virtual space then it won't run on a 32-bit system regardless of whether it's byte-code or not, and if a process requires a "fast enough" CPU then it won't run on "too slow" CPUs regardless of whether it's byte-code or not.

The difference here is less development constraints in case of bytecode. I have a code and know if a processor supports AVX then the code will run faster, but if there's no AVX the code also will run at a lower speed. So, I can forget about multiple choices related to the hardware features. The number of choices is great if we consider just Intel's chips, but it is much worse in case of processors from other vendors, including processors with different architectures. The developer's efforts here are really saved.

Brendan wrote:If you want a new OS written in a new programming language there's an unavoidable "chicken or egg" problem. You must either:
Write "temporary tools" for a completely different OS, then write the OS using those tools, then write native tools for your new OS; or

Write a "temporary OS" using a completely different language, then write native tools, then port the OS and tools to your language
I've tried both ways. Both ways involve doing a lot of work twice; but the second way is easier, especially if the OS is modular (lots of small pieces that can be replaced one at a time). Of course the IDE and tools don't need to be full featured (as soon as the IDE and tools have minimal functionality you can start porting); and the "temporary OS" doesn't need to be full featured either.

Your choice includes new language. It means there are no IDEs to help you significantly. But still, to have something like notepad++ your OS should implement 2D graphics in a very extended way. While you have no good graphics you still need a good IDE. After you have graphics you still can use foreign IDE, but now the new language requires you to write your own IDE. In case of more generally accepted language you can use the foreign IDE for very long time, much longer than in case of a new language + new OS pair. But it's your choice and for it the native IDE approach seems to work well.

Brendan wrote:
embryo2 wrote:Because the generated code most probably won't be used on another computer there's no value in storing the executable file with it's "type and features" information. I suggest it is better to use a centralized database with all the meta-information about all the software installed (precompiled or not), so, there's no need for a tricky structures paired with executable. Just pieces of code referred from the database.
It's a distributed system. If you've got a cluster consisting of 5 different types of computers then you'd have (up to) 5 different versions of the final/native executable in the distributed file system.

When it's about cluster there should be some software which is able to distribute a code across cluster nodes. So, it seems very suitable for such software to be the actor that manages not only the transmission of the code, but also the transmission of the metadata. And here is the important question - what metadata is required on the side of another node? Generally, it depends on the cluster architecture. So, we can abstract the metadata from the code and send only some required part of it when the actual cluster architecture will be completed. And if you to design an executable format with the metadata included, then the decision on the metadata transmission will be made too early, much before the clustering is ready to deal with it. It will keep you in the "change and test" loop for very long time.

Brendan wrote:Aggressive/thorough optimisation takes a significant amount of time (especially whole program optimisation). For JIT, you're optimising at run-time and can't afford the time it takes to do optimisation properly.

Here is the trade - start time for fastest speed in the beginning. I prefer to get the small start time and next to do complex optimizations in the background as long as it can take.

Brendan wrote:
embryo2 wrote:So, the communication facilities are abstracted to the very basic level of the hardware? If there are different cores and they have a need for communication then the caching issues can impact performance, so the communication layer can be implemented with care for caches or without such care. But if the care is present then the number of possible combinations looks a bit huge to be implemented.
Communication is done via. messages (regardless of whether it's communication between processes on the same computer or communication between processes on different computers). Processes communicate without needing to care if they're on the same computer or not.

Is the communication optimized for the heterogeneous systems? I mean the messaging affects performance because of the need for the state switch among the sender, the mediator and the receiver. If the switch is performed across a set of different processors, then every time there's new caching or available instructions/features situation. What do you think about the overhead of such switch?

Brendan wrote:I think it's the same for everyone - there's always something they've grown used to, where the hassle of finding/switching to something better is too high.

Here's where the heavy advertising helps. May be there's some value in it? But costs are too high.

Brendan wrote:if you target a narrow niche, your OS needs to be significantly better than alternatives (for that niche) and remain better for multiple years. Even if you can think of a new market for an OS where there's currently no competition at all, other OSs will try to move into that market.

I think the time required for somebody to move into niche market with new OS is underestimated by you. The efforts required often can't pay off. But in your case the efforts are cheaper because it's your hobby, so your efforts can pay off if the hobby satisfaction is involved. It means you have an advantage and it is the barrier for your competitors.

Posted: **Wed Dec 02, 2015 8:25 am**

Hi,

embryo2 wrote:
Brendan wrote:Of course the main point of shifting to byte-code is to avoid most of that; but even then you can't avoid all of it and you still end up with application developers saying "the minimum requirements for this application are....". For example, if a process must use 8 GiB of virtual space then it won't run on a 32-bit system regardless of whether it's byte-code or not, and if a process requires a "fast enough" CPU then it won't run on "too slow" CPUs regardless of whether it's byte-code or not.
The difference here is less development constraints in case of bytecode. I have a code and know if a processor supports AVX then the code will run faster, but if there's no AVX the code also will run at a lower speed. So, I can forget about multiple choices related to the hardware features. The number of choices is great if we consider just Intel's chips, but it is much worse in case of processors from other vendors, including processors with different architectures. The developer's efforts here are really saved.

Not really.

In general; if you write your source code then compile it, it doesn't make much difference if the compiler generates byte-code or native code or something else (an assembly listing, a list of dependencies, etc). The only difference is what you tell the compiler to generate.

embryo2 wrote:
Brendan wrote:If you want a new OS written in a new programming language there's an unavoidable "chicken or egg" problem. You must either:
Write "temporary tools" for a completely different OS, then write the OS using those tools, then write native tools for your new OS; or

Write a "temporary OS" using a completely different language, then write native tools, then port the OS and tools to your language
I've tried both ways. Both ways involve doing a lot of work twice; but the second way is easier, especially if the OS is modular (lots of small pieces that can be replaced one at a time). Of course the IDE and tools don't need to be full featured (as soon as the IDE and tools have minimal functionality you can start porting); and the "temporary OS" doesn't need to be full featured either.
Your choice includes new language. It means there are no IDEs to help you significantly. But still, to have something like notepad++ your OS should implement 2D graphics in a very extended way. While you have no good graphics you still need a good IDE. After you have graphics you still can use foreign IDE, but now the new language requires you to write your own IDE. In case of more generally accepted language you can use the foreign IDE for very long time, much longer than in case of a new language + new OS pair. But it's your choice and for it the native IDE approach seems to work well.

It's worse than that. The source code file format will store source code as tokens or AST; but it's also going to store indexes (for user defined types and the symbol table) to speed up the IDE and compiler, plus cached copies of pre-compiled byte-code (to speed up compile times more) and maybe other data (unit test results, profiler results, etc). The IDE will manage all of this, and provide a graphical editor (possibly a little like Flowgorithm). A plain text editor (e.g. notepad++) will be completely useless.

embryo2 wrote:
Brendan wrote:
embryo2 wrote:Because the generated code most probably won't be used on another computer there's no value in storing the executable file with it's "type and features" information. I suggest it is better to use a centralized database with all the meta-information about all the software installed (precompiled or not), so, there's no need for a tricky structures paired with executable. Just pieces of code referred from the database.
It's a distributed system. If you've got a cluster consisting of 5 different types of computers then you'd have (up to) 5 different versions of the final/native executable in the distributed file system.
When it's about cluster there should be some software which is able to distribute a code across cluster nodes. So, it seems very suitable for such software to be the actor that manages not only the transmission of the code, but also the transmission of the metadata. And here is the important question - what metadata is required on the side of another node? Generally, it depends on the cluster architecture. So, we can abstract the metadata from the code and send only some required part of it when the actual cluster architecture will be completed. And if you to design an executable format with the metadata included, then the decision on the metadata transmission will be made too early, much before the clustering is ready to deal with it. It will keep you in the "change and test" loop for very long time.

There's 2 different types of distributed systems - some that use a "client/server" model (which is simpler to implement), and some that use a "peer to peer" model (which is more flexible, more fault tolerant and take less end-user configuration). Mine is the latter.

When a process is started; the "local" kernel asks the other computer's kernels for information (e.g. current load, etc) and uses this and the executable's metadata (described previously) and some other things (which final/native versions exist already) to decide where the executable will be run (and which final/native version of the executable to use). If the local kernel decided it should run the executable itself it does, and if the local kernel decides a remote computer should run the executable it tells that remote kernel "Congratulations, you've won a process!". There is no central agent that's responsible for deciding which processes get executed where (and no single point of failure).

embryo2 wrote:
Brendan wrote:Aggressive/thorough optimisation takes a significant amount of time (especially whole program optimisation). For JIT, you're optimising at run-time and can't afford the time it takes to do optimisation properly.
Here is the trade - start time for fastest speed in the beginning. I prefer to get the small start time and next to do complex optimizations in the background as long as it can take.

I prefer to optimise the code as much as possible once (for each type of CPU in the cluster) and then (after the final/native version exists) re-use the "already optimised far more than JIT can dream of" final/native version with no additional overhead at all.

embryo2 wrote:
Brendan wrote:
embryo2 wrote:So, the communication facilities are abstracted to the very basic level of the hardware? If there are different cores and they have a need for communication then the caching issues can impact performance, so the communication layer can be implemented with care for caches or without such care. But if the care is present then the number of possible combinations looks a bit huge to be implemented.
Communication is done via. messages (regardless of whether it's communication between processes on the same computer or communication between processes on different computers). Processes communicate without needing to care if they're on the same computer or not.
Is the communication optimized for the heterogeneous systems? I mean the messaging affects performance because of the need for the state switch among the sender, the mediator and the receiver. If the switch is performed across a set of different processors, then every time there's new caching or available instructions/features situation. What do you think about the overhead of such switch?

Worst case latency is where the message goes from one computer to another to another to another before it reaches its destination; using extremely slow/congested network connections for each "hop". The messaging itself is asynchronous; which means sending a message doesn't cause an implied task switch, and receiving a message doesn't cause an implied task switch. For example, (assuming 2 processes both running on the same single-CPU computer) one process can send 1234 messages (at up to 2 MiB of data per message), then the scheduler might decide to do a task switch, then the receiver might receive all 1234 messages; and in that way you can transfer 2468 MiB of data with only one task switch. For 2 processes running on different CPUs in the same system, one process can be sending while the other is receiving and there might be no task switches at all.

Sending and receiving does imply a switch from CPL=3 to CPL0 and back; but here I support "batched kernel functions" to help avoid that. How this works is you create a list of kernel API functions you want to call (function number and its parameters), then call the kernel to process the list, then examine the results after. On top of that each thread has 2 message buffers. This means that a thread can prepare 2 messages; then create a list of kernel API functions that includes sending both messages, receiving 2 messages (one per buffer) and any other kernel API functions it wants; then ask the kernel to process the list (and send both messages and receive up to 2 messages and do anything else). In this way you can send 2 messages and receive 2 messages with a single "CPL=3 -> CPL=0 -> CPL=3" switch.

The overhead for all the different permutations ranges from "a quarter of very little" (for SYSCALL/SYSENTER where you're processing a list of 2 sent and 2 received, and where sender and receiver are running on different CPUs in the same computer at the same time) up to "infinity" (e.g. network failure between sender and receiver after message sent, causing messages that were successfully sent but never received).

embryo2 wrote:
Brendan wrote:I think it's the same for everyone - there's always something they've grown used to, where the hassle of finding/switching to something better is too high.
Here's where the heavy advertising helps. May be there's some value in it? But costs are too high.

If I ever actually achieve half of what I'm planning; I'd be able to submit a story on a relevant news site (e.g. OSnews) with a link to a you-tube demonstration video, and within 1 week there's be a story about the OS on virtually every IT news site on the Internet. The idea is to create a huge "wave of interest" and then ride that wave. Most people make the mistake of trying to release/publicise the OS far too early and/or not showcasing (and/or not having) unique features; so instead of causing a large "wave of interest" they only create minor ripples.

embryo2 wrote:
Brendan wrote:if you target a narrow niche, your OS needs to be significantly better than alternatives (for that niche) and remain better for multiple years. Even if you can think of a new market for an OS where there's currently no competition at all, other OSs will try to move into that market.
I think the time required for somebody to move into niche market with new OS is underestimated by you. The efforts required often can't pay off. But in your case the efforts are cheaper because it's your hobby, so your efforts can pay off if the hobby satisfaction is involved. It means you have an advantage and it is the barrier for your competitors.

A single person writing a new OS for a new niche market will take ages (partly due to research, etc). A group of many people modifying an existing OS (e.g. Linux) to create a crappy clone of those ideas for the same niche market (without doing much/any research because it was already done) will be much much faster.

Cheers,

Brendan

Posted: **Wed Dec 02, 2015 10:07 am**

Brendan wrote:Most people make the mistake of trying to release/publicise the OS far too early and/or not showcasing (and/or not having) unique features; so instead of causing a large "wave of interest" they only create minor ripples.

I very much agree with all of this. That wave could be very critical time-wise, so you should carefully plan proper timing also. The week will be very busy and the initial attention given by the media does not last too long. The worst thing there could be announcements like "available next year" or something like that. It really must be available at the time of publication and be somewhat usable for some purposes at least.

Posted: **Thu Dec 03, 2015 6:35 am**

Brendan wrote:In general; if you write your source code then compile it, it doesn't make much difference if the compiler generates byte-code or native code or something else (an assembly listing, a list of dependencies, etc). The only difference is what you tell the compiler to generate.

The distribution issue is forgotten. You need to compile a lot of permutations and distribute them without mistakes.

Brendan wrote:It's worse than that. The source code file format will store source code as tokens or AST; but it's also going to store indexes (for user defined types and the symbol table) to speed up the IDE and compiler, plus cached copies of pre-compiled byte-code (to speed up compile times more) and maybe other data (unit test results, profiler results, etc). The IDE will manage all of this, and provide a graphical editor (possibly a little like Flowgorithm). A plain text editor (e.g. notepad++) will be completely useless.

Yes, you are going to redefine the world. I'm afraid your life is too short for it.

Brendan wrote:There's 2 different types of distributed systems - some that use a "client/server" model (which is simpler to implement), and some that use a "peer to peer" model (which is more flexible, more fault tolerant and take less end-user configuration). Mine is the latter.

When a process is started; the "local" kernel asks the other computer's kernels for information (e.g. current load, etc) and uses this and the executable's metadata (described previously) and some other things (which final/native versions exist already) to decide where the executable will be run (and which final/native version of the executable to use). If the local kernel decided it should run the executable itself it does, and if the local kernel decides a remote computer should run the executable it tells that remote kernel "Congratulations, you've won a process!". There is no central agent that's responsible for deciding which processes get executed where (and no single point of failure).

When the "congratulation" arrives the "foreign" kernel can decide it has much more important load. Because there's no responsibility the conversation can be very long.

Brendan wrote:Worst case latency is where the message goes from one computer to another to another to another before it reaches its destination; using extremely slow/congested network connections for each "hop".

Best case is more interesting.

Brendan wrote:The messaging itself is asynchronous; which means sending a message doesn't cause an implied task switch, and receiving a message doesn't cause an implied task switch.

Tasks should cooperate when are trying to send message. It means they should get some lock and wait. The wait time is inescapable. And if there is a malicious task, then it can block other tasks forever just by hacking the message exchange protocol.

Brendan wrote:Sending and receiving does imply a switch from CPL=3 to CPL0 and back; but here I support "batched kernel functions" to help avoid that.

Despite of the batch the switch is actually already here. Far jump and usage of the different data (kernel's data) require the processor to recache almost everything. Also, the need "to batch" requires programmer's attention. How hard it will be?

Brendan wrote:If I ever actually achieve half of what I'm planning; I'd be able to submit a story on a relevant news site (e.g. OSnews) with a link to a you-tube demonstration video, and within 1 week there's be a story about the OS on virtually every IT news site on the Internet.

May be. But it is very easy to overestimate the impact. Even if there's a lot of usable features. It's like breaking a wall. You know that it's possible, but also you know that the speed is required. After you have some speed you start thinking "may be it's right time?". With your approach "right time" may come when the wall will be completely removed and installed in another point.

You have chances, but it's only some probability, not something guaranteed. Despite of any feature you can have implemented.

Brendan wrote:A single person writing a new OS for a new niche market will take ages (partly due to research, etc). A group of many people modifying an existing OS (e.g. Linux) to create a crappy clone of those ideas for the same niche market (without doing much/any research because it was already done) will be much much faster.

It means there's no room for a single person in the house. But there's plenty of room for teams. Single person often do not want to cooperate. And less skilled teamworkers win in this game. Just because the single person refuse to accept reality and cooperate with the society.

It's social. Even if it looks like technical.

Posted: **Thu Dec 03, 2015 5:57 pm**

Hi,

embryo2 wrote:
Brendan wrote:It's worse than that. The source code file format will store source code as tokens or AST; but it's also going to store indexes (for user defined types and the symbol table) to speed up the IDE and compiler, plus cached copies of pre-compiled byte-code (to speed up compile times more) and maybe other data (unit test results, profiler results, etc). The IDE will manage all of this, and provide a graphical editor (possibly a little like Flowgorithm). A plain text editor (e.g. notepad++) will be completely useless.
Yes, you are going to redefine the world. I'm afraid your life is too short for it.

I'm estimating around 20 years before an initial release, and do hope to live that long.

embryo2 wrote:
Brendan wrote:There's 2 different types of distributed systems - some that use a "client/server" model (which is simpler to implement), and some that use a "peer to peer" model (which is more flexible, more fault tolerant and take less end-user configuration). Mine is the latter.

When a process is started; the "local" kernel asks the other computer's kernels for information (e.g. current load, etc) and uses this and the executable's metadata (described previously) and some other things (which final/native versions exist already) to decide where the executable will be run (and which final/native version of the executable to use). If the local kernel decided it should run the executable itself it does, and if the local kernel decides a remote computer should run the executable it tells that remote kernel "Congratulations, you've won a process!". There is no central agent that's responsible for deciding which processes get executed where (and no single point of failure).
When the "congratulation" arrives the "foreign" kernel can decide it has much more important load. Because there's no responsibility the conversation can be very long.

No; the remote kernel can't decide it has more important load and must run the process it's told to start. There is a race condition (remote computer's load may change after they've reported their current load but before local kernel tells one to start the new process) and the end result of this race condition is a small chance of a computer getting higher load than it otherwise would have; but load balancing isn't an exact science (and is only a performance optimisation) and if a computer does get "higher than average" load it's likely to be only a temporary problem that will correct itself.

embryo2 wrote:
Brendan wrote:Worst case latency is where the message goes from one computer to another to another to another before it reaches its destination; using extremely slow/congested network connections for each "hop".
Best case is more interesting.
Brendan wrote:The messaging itself is asynchronous; which means sending a message doesn't cause an implied task switch, and receiving a message doesn't cause an implied task switch.
Tasks should cooperate when are trying to send message. It means they should get some lock and wait. The wait time is inescapable. And if there is a malicious task, then it can block other tasks forever just by hacking the message exchange protocol.

No. The kernel's API has "sendMessage()", "getMessage()" and "getMessageWithTimeout()", and these are the only way to send and receive messages. The kernel would have to be maliciously hacked to make it acquire a receiver's message queue lock and fail to release it.

The worst that a malicious task can do is send a huge amount of "spam messages"; but it's easy enough to put mechanism/s in place to mitigate this; including a system to reduce the sending task's priority when it's sending too much and/or boost receiver's priority when their queues are holding on to too many messages (which is needed to manage memory anyway - e.g. if a computer has 8 GiB of RAM you don't want 8 GiB of message data in queues waiting to be received).

Note that it will also have:

an error reporting system; where if a process crashes or does something stupid the OS can auto-inform the developers that their code is buggy and what went wrong, and/or auto-inform administrators of processes that had problems
a "complaints" system; where a process can complain about other processes (e.g. complain that another process is sending spam, is sending wrong replies, etc); where if there's too many complains about a process it triggers the error reporting system (and auto-informs developers/administrators).

embryo2 wrote:
Brendan wrote:Sending and receiving does imply a switch from CPL=3 to CPL0 and back; but here I support "batched kernel functions" to help avoid that.
Despite of the batch the switch is actually already here. Far jump and usage of the different data (kernel's data) require the processor to recache almost everything.

The micro-kernel's send/get message functions only need to touch a small number of cache lines (that are probably in cache anyway) and doesn't require the processor to "re-cache almost everything".

embryo2 wrote:Also, the need "to batch" requires programmer's attention. How hard it will be?

The difficulty of using "batched kernel functions" (rather than just using the kernel API one function at a time) ranges from "trivial" to "impossible" depending on specific circumstances.

For an over-simplified pseudo-code example; for a "service" style process (which mostly just receives requests, handles them and sends replies) you might do something like this:

Code: Select all

init:
    do_batch(set thread name, set thread priority, allocate some virtual pages, ....);  // Trivial

    getMessage();                                                                       // Impossible
handleMessage:
    switch(messageType) {
    case FOO:
        handle_foo();
        do_batch(send reply message, get next message);                                 // Trivial
        goto handleMessage;
    case BAR:
        handle_bar();
        do_batch(send reply message, free temporarily used pages, get next message);    // Trivial
        goto handleMessage;
    case BAZ:
        handle_baz();
        do_batch(send nofitication message 1, send nofitication message 2);             // Trivial
        do_batch(send nofitication message 3, send nofitication message 4);             // Trivial
        do_batch(send nofitication message 5, get message);                             // Trivial
        goto handleMessage;
    case TERMINATE:
        terminateProcess();                                                             // Impossible
        goto handleMessage;
    default:
        getMessage();                                                                   // Impossible
        goto handleMessage;
    }

embryo2 wrote:
Brendan wrote:If I ever actually achieve half of what I'm planning; I'd be able to submit a story on a relevant news site (e.g. OSnews) with a link to a you-tube demonstration video, and within 1 week there's be a story about the OS on virtually every IT news site on the Internet.
May be. But it is very easy to overestimate the impact. Even if there's a lot of usable features. It's like breaking a wall. You know that it's possible, but also you know that the speed is required. After you have some speed you start thinking "may be it's right time?". With your approach "right time" may come when the wall will be completely removed and installed in another point.

You have chances, but it's only some probability, not something guaranteed. Despite of any feature you can have implemented.

The only guarantee is that you're guaranteed to fail to attract users if you don't try to maximise your chance of successfully attracting users.

embryo2 wrote:
Brendan wrote:A single person writing a new OS for a new niche market will take ages (partly due to research, etc). A group of many people modifying an existing OS (e.g. Linux) to create a crappy clone of those ideas for the same niche market (without doing much/any research because it was already done) will be much much faster.
It means there's no room for a single person in the house. But there's plenty of room for teams. Single person often do not want to cooperate. And less skilled teamworkers win in this game. Just because the single person refuse to accept reality and cooperate with the society.

It's social. Even if it looks like technical.

It's both. For the first half of the time spent developing an OS (while you're implementing things like kernel/s, GUI, file system, device driver interfaces, etc; and while you're writing specifications/documentation) the only team members you'll attract are "bikeshedders" that slow you down - it's 100% technical plus 0% social. For the second half of the OS (after you've reached the stage where people can start writing device drivers, applications, etc) it changes and becomes "100% technical plus 100% social" (likely more than one person can handle).

Cheers,

Brendan

Posted: **Fri Dec 04, 2015 10:39 am**

Brendan wrote:the remote kernel can't decide it has more important load and must run the process it's told to start.

Then I suppose you have an algorithm for the local kernel which makes it "the server" (in client-server terms) for a moment. After the local kernel get rid of it's load (did it itself or has it sent) then here again is something you can call "peer to peer". But actually the "peer to peer" involves no action at all. So, it's client-server with some algorithm for selecting the server. But why not to get rid of extra algorithm and to have just one dedicated server?

Brendan wrote:The kernel's API has "sendMessage()", "getMessage()" and "getMessageWithTimeout()", and these are the only way to send and receive messages.

There's no events? A task should loop until it gets the message?

Brendan wrote:The difficulty of using "batched kernel functions" (rather than just using the kernel API one function at a time) ranges from "trivial" to "impossible" depending on specific circumstances.

Your example looks too complex. Have you thought about something like this:

Code: Select all

batchStart();
sendMessage(m1);
sendMessage(m2);
sendMessage(m3);
batchEnd(); // here the actual transmission happens

Brendan wrote:
embryo2 wrote:It's social. Even if it looks like technical.
It's both. For the first half of the time spent developing an OS (while you're implementing things like kernel/s, GUI, file system, device driver interfaces, etc; and while you're writing specifications/documentation) the only team members you'll attract are "bikeshedders" that slow you down - it's 100% technical plus 0% social. For the second half of the OS (after you've reached the stage where people can start writing device drivers, applications, etc) it changes and becomes "100% technical plus 100% social" (likely more than one person can handle).

Teams usually manage to cope with the bikeshedders during all development stages. Including the kernel/s, GUI, file system, device driver interfaces, etc.

But if it's the point to have "only your" solution, then yes, all team members are bikeshedders (except you).

Posted: **Sat Dec 05, 2015 12:46 am**

Hi,

embryo2 wrote:
Brendan wrote:the remote kernel can't decide it has more important load and must run the process it's told to start.
Then I suppose you have an algorithm for the local kernel which makes it "the server" (in client-server terms) for a moment. After the local kernel get rid of it's load (did it itself or has it sent) then here again is something you can call "peer to peer". But actually the "peer to peer" involves no action at all. So, it's client-server with some algorithm for selecting the server. But why not to get rid of extra algorithm and to have just one dedicated server?

For my way; if you've got 100 computers in a cluster (all doing this independently and telling each other when to start processes) and one of the computer explodes, then regardless of which computer exploded you end up 99 computers in a cluster (all doing this independently). For your way; if you've got 100 computers and the dedicated server explodes, then you end up with 100 unusable computers.

For my way; if the cluster consists of 50 computers in a high speed LAN on one side of the world and another 50 computers in another high speed LAN on the other side of the world, with a much slower Internet link between these LANs; then each of the computers can decide the communication overhead is too high for 50 of the computers without asking them and avoid a bunch of traffic across that slower Internet link. For your way, all the traffic from 50 computers has to go over the slow Internet link to the server on the other side of the world.

For your way; both of these problems could be worked around (e.g. a complicated mess to nominate a new computer to use for the server; or having a "one server per LAN" model instead of a "one server per cluster" model); but these work-arounds are just trying to get closer to matching the characteristics/behaviour that my way provides.

Note that there's also a "split brain" problem to worry about. For that "50 + 50 computers (with slower Internet link between)", what happens if the Internet link goes down for an hour? You don't want 50 computers to be unusable while the link is down, so you nominate a new computer to use for the server. Then when the link starts working again you end up with 2 servers that both think they're in charge. Of course this can happen anywhere (it's not just the slower Internet link that could go down). For example, maybe you've got a high speed ethernet switch connecting 5 sub-nets (with 10 computers per sub-net) and that high speed switch dies; leaving you with 5 isolated groups (with 10 computers per group) for a while.

Now (for a worst case scenario) imagine you've got 20 groups of people (with laptops) riding buggies around the desert; where at various times any group can be within wifi range of any of the other group/s. There is no reliable network, just "random" short term ad-hoc network connections between groups.

embryo2 wrote:
Brendan wrote:The kernel's API has "sendMessage()", "getMessage()" and "getMessageWithTimeout()", and these are the only way to send and receive messages.
There's no events? A task should loop until it gets the message?

Messages are events. When a task calls "getMessage()" and there are no messages the kernel blocks the thread (it gets no CPU time) until a message arrives; and when a task calls "getMessageWithTimeout()" the kernel blocks the thread until either a message arrives or the time-out expires. This means that a task doesn't loop until it gets a message (a task has a "get one message at a time" loop and the kernel sorts out blocking/unblocking as needed).

Note: In previous versions of the kernel I used to have a "checkForMessages()" function that would never block (if there were no messages to get it'd return an error instead). I don't provide that any more because "getMessageWithTimeout()" does the same thing if the time-out is set to zero. With the old "checkForMessages()" function or with "getMessageWithTimeout(0)", a thread can do a polling loop. However, this is mostly only sane if the thread is busy doing heavy processing, sort of like this maybe:

Code: Select all

    while(doPartOfTheWork() != WORK_FINISHED) {
        if( getMessageWithTimeout(0) != TIMEOUT ) {
            handleMessage();
        }
    }

In this way a thread that's doing heavy processing for 1234 seconds can still check for and respond to messages reasonably quickly (e.g. without waiting until the 1234 seconds of processing if completed before responding); and this includes being able to respond to messages that tell it to cancel that 1234 seconds of processing.[/i]

embryo2 wrote:
Brendan wrote:The difficulty of using "batched kernel functions" (rather than just using the kernel API one function at a time) ranges from "trivial" to "impossible" depending on specific circumstances.
Your example looks too complex. Have you thought about something like this:
Code: Select all
batchStart();
sendMessage(m1);
sendMessage(m2);
sendMessage(m3);
batchEnd(); // here the actual transmission happens

It'd be trivial to write a library that implements that on top of my approach - basically the behaviour is the same and the only difference is "syntactical sugar".

Note that there's actually 2 types of "batched kernel functions". For the first one; the kernel does all of the functions in the list (even if one of them returns an error) and does them in "random" order (mostly so that the kernel can optimise the order the functions are done in - e.g. do more important things like sending messages and freeing pages first, then do things that might cause a thread to block like "getMessage()" and "sleep()", then do things like allocating pages last). For the second one; the kernel does the functions in strict "as listed" order and stops processing the list as soon as any function returns an error.

If you want to set/change the thread's name, and set/change the thread's priority, and do "getMessageWithTimeout(0)" to check for a message; you probably don't care what order these are done in and want the kernel to do the others if one fails; and so you'd want "do all functions in any order".

For receive message faster (with "double pumping"); you'd want to do "getMessage()" then "getMessageWithTimeout(0)", and you'd want "do in order, stop on error". That way if there's no messages you block until there's a message; if there's 1 message you get it and don't block, and if there's 2 or more messages you get 2 messages at once and don't block. If you accidentally used "do all functions in any order" instead, then if there's one message the kernel might do "getMessageWithTimeout(0)" first and then do "getMessage()" last (causing you to block because there's no messages left in the queue).

embryo2 wrote:
Brendan wrote:
embryo2 wrote:It's social. Even if it looks like technical.
It's both. For the first half of the time spent developing an OS (while you're implementing things like kernel/s, GUI, file system, device driver interfaces, etc; and while you're writing specifications/documentation) the only team members you'll attract are "bikeshedders" that slow you down - it's 100% technical plus 0% social. For the second half of the OS (after you've reached the stage where people can start writing device drivers, applications, etc) it changes and becomes "100% technical plus 100% social" (likely more than one person can handle).
Teams usually manage to cope with the bikeshedders during all development stages. Including the kernel/s, GUI, file system, device driver interfaces, etc.

But if it's the point to have "only your" solution, then yes, all team members are bikeshedders (except you).

Once you've reached the stage where all the interfaces are designed, the specifications/documentation are written, and at least some code is using the interfaces (e.g. kernel, VFS, GUI, etc are implemented); there's very little left for "bikeshedding" - each programmer can go and implement their driver or application in isolation (just using the specifications/documentation) without ever talking to anyone else.

Cheers,

Brendan

Posted: **Sat Dec 05, 2015 1:11 pm**

Brendan wrote:Now (for a worst case scenario) imagine you've got 20 groups of people (with laptops) riding buggies around the desert; where at various times any group can be within wifi range of any of the other group/s. There is no reliable network, just "random" short term ad-hoc network connections between groups.

Even in worst case scenario there's the need for server. One laptop should be a host and another should be a slave. In your case it's just the protocol for determining who is the host can differ from the general client-server approach. But in your case it's automatic. It means if there are ways to define the cluster more efficiently, then an admin should know all the intricate parts of your protocol and it makes the administrator's job more complex. And usually it is preferable for the human to define a cluster. Only in some "really worst case" scenarios with buggies in desert your protocol can benefit it's creator. Else there will be no explosion, but just ordinary client-server job.

Brendan wrote:Messages are events.

Events are things that occur asynchronously. And synchronous job means there's some blocked call and no way to go until the call returns. So, your "messages are events" mixes both worlds. Of course, you can define your personal meaning of asynchronous processing, but there's some generally accepted way and it is better not to invent new definitions.

Brendan wrote:Note that there's actually 2 types of "batched kernel functions". For the first one; the kernel does all of the functions in the list (even if one of them returns an error) and does them in "random" order (mostly so that the kernel can optimise the order the functions are done in - e.g. do more important things like sending messages and freeing pages first, then do things that might cause a thread to block like "getMessage()" and "sleep()", then do things like allocating pages last). For the second one; the kernel does the functions in strict "as listed" order and stops processing the list as soon as any function returns an error.

Ok, there are "optimized" and "as defined" ways of doing job. And there is the way to send and receive many messages at a time. But it's interesting - when such complex processing is required? If I want to send some data I can use many available data structures as function arguments. If I want to receive the data then again I have the same data structures as the function's output. So, where the abstraction of messaging can help? Why not to call a function, providing it with a handler if there's a need for asynchronous processing, and then just do the generally accepted stuff as is the case for every function call?

Underneath the call some messaging can do whatever is required, but the call, as I see it, should be as simple as the standard function call. No new entities, no new pain.

Brendan wrote:Once you've reached the stage where all the interfaces are designed, the specifications/documentation are written, and at least some code is using the interfaces (e.g. kernel, VFS, GUI, etc are implemented); there's very little left for "bikeshedding" - each programmer can go and implement their driver or application in isolation (just using the specifications/documentation) without ever talking to anyone else.

Yes, the "without ever talking to anyone else" is the goal. But what if your youtube video won't gather enough attention? Then you just have to start talking to somebody to convince more people to use your OS. It's social again and there's no way to circumvent the "talking". Even youtube's video and documentation is not enough. All those googles usually setup a lot of forums for a lot of talks when they promote a new product.

And from the other side. If there's some talking from the beginning then yes, the kernel, GUI, etc won't be completely your's. But as a result not only a lot of time will be saved, there will be many people for all the talking required when the promotion to begin.

Posted: **Sat Dec 05, 2015 9:12 pm**

Hi,

embryo2 wrote:
Brendan wrote:Now (for a worst case scenario) imagine you've got 20 groups of people (with laptops) riding buggies around the desert; where at various times any group can be within wifi range of any of the other group/s. There is no reliable network, just "random" short term ad-hoc network connections between groups.
Even in worst case scenario there's the need for server. One laptop should be a host and another should be a slave. In your case it's just the protocol for determining who is the host can differ from the general client-server approach. But in your case it's automatic. It means if there are ways to define the cluster more efficiently, then an admin should know all the intricate parts of your protocol and it makes the administrator's job more complex. And usually it is preferable for the human to define a cluster. Only in some "really worst case" scenarios with buggies in desert your protocol can benefit it's creator. Else there will be no explosion, but just ordinary client-server job.

I want everything to be automatic (if/where possible), with the least possible end user configuration. I don't want admin to be chained to a desk trying to micro-manage software in real time (e.g. constantly changing where the "server" is each time conditions change) with normal users waiting for an hour just to start "calculator" because the admin guy is on his lunch break.

The "dune buggies in the desert" is a worst case scenario. A typical case is some stationary computers and some mobile computers (laptops, smartphones, tablets); where the mobile computers are switching between direct wired ethernet (when plugged in at a desk), wifi (when away from desk but still in range), Internet (e.g. 3G/4G mobile phone network) and "no networking available". Also note that (especially for those stationary computers) power management includes (automatically) telling computers to turn themselves off when the OS decides they aren't needed, and then turning them back on again (using wake on LAN) when the OS decides they are needed again.

For a normal office; maybe there's 4 headless computers in a back room (for heavy processing), 6 computers near desks with 4 keyboards/monitors per computer (for up to 24 users), plus 10 laptops (used by sales people and/or managers) that are coming and going all the time. At night it might wind back to 1 computer in the back room and turn the rest off, and in the morning (as people arrive at work) it might go from "1 computer, no users" all the way up to "20 computers, 34 users" within a 20 minute period.

embryo2 wrote:
Brendan wrote:Note that there's actually 2 types of "batched kernel functions". For the first one; the kernel does all of the functions in the list (even if one of them returns an error) and does them in "random" order (mostly so that the kernel can optimise the order the functions are done in - e.g. do more important things like sending messages and freeing pages first, then do things that might cause a thread to block like "getMessage()" and "sleep()", then do things like allocating pages last). For the second one; the kernel does the functions in strict "as listed" order and stops processing the list as soon as any function returns an error.
Ok, there are "optimized" and "as defined" ways of doing job. And there is the way to send and receive many messages at a time. But it's interesting - when such complex processing is required? If I want to send some data I can use many available data structures as function arguments. If I want to receive the data then again I have the same data structures as the function's output. So, where the abstraction of messaging can help? Why not to call a function, providing it with a handler if there's a need for asynchronous processing, and then just do the generally accepted stuff as is the case for every function call?

The "batch kernel functions" isn't just for 3 functions (for sending and receiving messages); it's for the entire kernel API; which includes virtual memory management, spawning and terminating threads and processes, controlling thread priorities, sleep/delays, finding "named services", handling "device resources permissions" (which process have access to which IO ports and memory mapped IO areas, and which threads are notified when an IRQ occurs), etc.

For messaging; it's a little bit like normal networking, where you have an "application protocol" (e.g. FTP, HTTP, NTP, ...) and "transport" (e.g. routing, TCP, UDP, IP). Messaging is just "transport". To make sense of the messages software needs "messaging protocols". It's these "messaging protocols" that define the use and format of messages.

Some messaging protocols are standardised (device driver interfaces, VFS/file IO, keyboard/mouse/video/sound, "services", etc) and will use open standards. Other messaging protocols aren't (e.g. a company that creates an application might create their own messaging protocol for internal communication between different parts/processes of their own application).

The "messaging" abstraction helps because it means any thread can communicate with any other thread without caring where the other thread is; and without caring if communication actually happens via. RAM buffers, or "raw InfiniBand packets", or "raw ethernet packets" or "TCP/IP across the Internet" or something else; and without caring about whether or not the actual data is compressed and/or encrypted, or split into multiple packets and reconstructed at the other end, or has some errors and needs to be retried/resent a few times, or....

embryo2 wrote:
Brendan wrote:Once you've reached the stage where all the interfaces are designed, the specifications/documentation are written, and at least some code is using the interfaces (e.g. kernel, VFS, GUI, etc are implemented); there's very little left for "bikeshedding" - each programmer can go and implement their driver or application in isolation (just using the specifications/documentation) without ever talking to anyone else.
Yes, the "without ever talking to anyone else" is the goal. But what if your youtube video won't gather enough attention? Then you just have to start talking to somebody to convince more people to use your OS. It's social again and there's no way to circumvent the "talking". Even youtube's video and documentation is not enough. All those googles usually setup a lot of forums for a lot of talks when they promote a new product.

You're mixing development (where bikeshedding is a problem in early stages of the project, and there's no need for much communication between developers in the later stages, and is 100% technical initially but becomes 50% technical plus 50% social later), and promotion/marketing (which doesn't/shouldn't happen in early stages, but could be considered pure "social engineering" when it begins).

embryo2 wrote:And from the other side. If there's some talking from the beginning then yes, the kernel, GUI, etc won't be completely your's. But as a result not only a lot of time will be saved, there will be many people for all the talking required when the promotion to begin.

Realistically; it would most likely be done in stages - some very limited initial release/s intended to attract testers/programmers/artists/promoters (and not users), followed by a "grand unveiling" intended to create the largest "tidal wave of interest" possible (and attract as many normal users as possible), followed by regular updates from that point on.

Note that I expect it'll take at least 10 years before I can even consider the first very limited initial release.

Cheers,

Brendan

Posted: **Sun Dec 06, 2015 7:59 am**

Brendan wrote:I want everything to be automatic (if/where possible), with the least possible end user configuration. I don't want admin to be chained to a desk trying to micro-manage software in real time (e.g. constantly changing where the "server" is each time conditions change) with normal users waiting for an hour just to start "calculator" because the admin guy is on his lunch break.

Everything automatic means AI or a lot of unpredictable troubles in case there's no algorithm for a specific case. People get used to the servers for a long time and there are systems already in place that minimize the impact of the "admin's lunch break". Even your automatic algorithm in the end should understand that there's a server machine, which isn't turned off every day and has extended processing capacity. So, why not to allow the standard way of office computing to take place? Just because you want a smart algorithm? But you can add the algorithm on top of the existing system with human administrators. And when it's really profitable to use the algorithm the administrators, of course, will use it. But the first actor here is the human. I don't think you can create an algorithm that can replace the human for every case.

Brendan wrote:A typical case is some stationary computers and some mobile computers (laptops, smartphones, tablets); where the mobile computers are switching between direct wired ethernet (when plugged in at a desk), wifi (when away from desk but still in range), Internet (e.g. 3G/4G mobile phone network) and "no networking available". Also note that (especially for those stationary computers) power management includes (automatically) telling computers to turn themselves off when the OS decides they aren't needed, and then turning them back on again (using wake on LAN) when the OS decides they are needed again.

For a normal office; maybe there's 4 headless computers in a back room (for heavy processing), 6 computers near desks with 4 keyboards/monitors per computer (for up to 24 users), plus 10 laptops (used by sales people and/or managers) that are coming and going all the time. At night it might wind back to 1 computer in the back room and turn the rest off, and in the morning (as people arrive at work) it might go from "1 computer, no users" all the way up to "20 computers, 34 users" within a 20 minute period.

And the standard dedicated server is a nice match for the picture above. It can run for very long, it can spread load across available PCs, it can handle bigger part of the load. And it can be replaced quickly in case some guard PC has detected the server is missing. And if there's a war and all PCs are burnt, then I doubt the need for your system, just as for any other software in the burned office, while there still can be a few laptops alive, but no person to use them.

Brendan wrote:The "batch kernel functions" isn't just for 3 functions (for sending and receiving messages); it's for the entire kernel API; which includes virtual memory management, spawning and terminating threads and processes, controlling thread priorities, sleep/delays, finding "named services", handling "device resources permissions" (which process have access to which IO ports and memory mapped IO areas, and which threads are notified when an IRQ occurs), etc.

You want to tell the messaging can be actively used somewhere in the depth of the software stack. Ok, let it be, but why not to expose it in the very habitual manner, called "function call"? A developer can do something like this:

Code: Select all

TargetInterface c=System.getAsynchronousCommunicationChannel(destinationAddress,responseHandler,TargetInterface.class);
c.doWhateverRequired(someDataStructures);
c.release();

TargetInterface2 c2=System.getSynchronousCommunicationChannel(destinationAddress2,TargetInterface2.class);
Response r=c2.doWhateverRequired2(someDataStructures2);
c2.release();
// the same as in case of asynchronous call, but with another form of the handler invocation
ResponseHandler.doWhateverRequiredWithTheResponse(r);

instead of:

Code: Select all

prepareForMessaging(); // here we need to do a lot of things manually
createMessages(); // here we need to translate our data structures in form of messages
sendMessages();

And yes, the protocols and transports are out of developer's scope for the calls. But it should be possible to provide some hints or more rigid instructions in case a developer want a particular protocol or transport.

Brendan wrote:The "messaging" abstraction helps because it means any thread can communicate with any other thread without caring where the other thread is;

If I do not know (shouldn't care) about underlying messaging it can help more.

Brendan wrote:and without caring if communication actually happens via. RAM buffers, or "raw InfiniBand packets", or "raw ethernet packets" or "TCP/IP across the Internet" or something else; and without caring about whether or not the actual data is compressed and/or encrypted, or split into multiple packets and reconstructed at the other end, or has some errors and needs to be retried/resent a few times, or....

Ok, let my call be optimized and dragged through the RAM, TCP/IP, HDD, DHCP, martian crater or whatever, but let also it be as simple as a trivial function call. That simple.

Brendan wrote:You're mixing development (where bikeshedding is a problem in early stages of the project, and there's no need for much communication between developers in the later stages, and is 100% technical initially but becomes 50% technical plus 50% social later), and promotion/marketing (which doesn't/shouldn't happen in early stages, but could be considered pure "social engineering" when it begins).

There's a single process of software creation and distribution (it's life-cycle). The software is always targeted for a human auditory even if it controls a blast furnace. Only people buy it and actually use it.

But yes, it's possible to do some technical stuff with a single person only. The outcome here is the time and some prospects of the future distribution drawbacks. The key decisions in this case should be made far before the auditory is allowed to test the product, so, the chances are the tests will show the key decisions were wrong and the cost of change is a lot of time again.

Brendan wrote:Realistically; it would most likely be done in stages - some very limited initial release/s intended to attract testers/programmers/artists/promoters (and not users), followed by a "grand unveiling" intended to create the largest "tidal wave of interest" possible (and attract as many normal users as possible), followed by regular updates from that point on.

Ok, but it's much closer now to what corporations do - they just shorten the stages a lot. So, there's a lot of very short stages. It's contiguous flow of releases and patches. One shot and immediate success without such flow is very unlikely to be the case. But you can try and may be the luck will be with you.

Posted: **Sun Dec 06, 2015 9:55 am**

Hi,

embryo2 wrote:
Brendan wrote:I want everything to be automatic (if/where possible), with the least possible end user configuration. I don't want admin to be chained to a desk trying to micro-manage software in real time (e.g. constantly changing where the "server" is each time conditions change) with normal users waiting for an hour just to start "calculator" because the admin guy is on his lunch break.
Everything automatic means AI or a lot of unpredictable troubles in case there's no algorithm for a specific case. People get used to the servers for a long time and there are systems already in place that minimize the impact of the "admin's lunch break". Even your automatic algorithm in the end should understand that there's a server machine, which isn't turned off every day and has extended processing capacity. So, why not to allow the standard way of office computing to take place? Just because you want a smart algorithm? But you can add the algorithm on top of the existing system with human administrators. And when it's really profitable to use the algorithm the administrators, of course, will use it. But the first actor here is the human. I don't think you can create an algorithm that can replace the human for every case.

You have "cause and effect" around the wrong way. It's not a case of "I want to use a smart algorithm, therefore I need to avoid end user hassle" but it is a case of "I want to avoid end user hassle, therefore I need a smart algorithm".

End user hassles like this are just what retarded OSs (e.g. Linux) force on users because the OS developers are too stupid to do anything right. They push the hassle of making decisions onto people that have less knowledge of the system than its own designers and are therefore less able to make the decision (the end user/admin). This increases "total cost of ownership" (in the form of unnecessary system administrator wages/salary), and makes the OS far too slow to adapt to changes in circumstance (e.g. 2 hours of downtime while the guy that was "on call" tries to wake up and travel to work just to press a button).

I can (and mostly already have) create a "peer to peer" way to automate this. Your alternative is:

Use a "single server" model that causes a large number of problems for no reason
Force humans to waste their time trying to work around half the problems that the "single server" model caused
Don't solve the other half of the problems that the "single server" model caused
Don't solve the additional problems that "human error" cause either

embryo2 wrote:
Brendan wrote:A typical case is some stationary computers and some mobile computers (laptops, smartphones, tablets); where the mobile computers are switching between direct wired ethernet (when plugged in at a desk), wifi (when away from desk but still in range), Internet (e.g. 3G/4G mobile phone network) and "no networking available". Also note that (especially for those stationary computers) power management includes (automatically) telling computers to turn themselves off when the OS decides they aren't needed, and then turning them back on again (using wake on LAN) when the OS decides they are needed again.

For a normal office; maybe there's 4 headless computers in a back room (for heavy processing), 6 computers near desks with 4 keyboards/monitors per computer (for up to 24 users), plus 10 laptops (used by sales people and/or managers) that are coming and going all the time. At night it might wind back to 1 computer in the back room and turn the rest off, and in the morning (as people arrive at work) it might go from "1 computer, no users" all the way up to "20 computers, 34 users" within a 20 minute period.
And the standard dedicated server is a nice match for the picture above. It can run for very long, it can spread load across available PCs, it can handle bigger part of the load. And it can be replaced quickly in case some guard PC has detected the server is missing. And if there's a war and all PCs are burnt, then I doubt the need for your system, just as for any other software in the burned office, while there still can be a few laptops alive, but no person to use them.

I've already pointed out multiple problems with the "single server" idea (including fault tolerance when that single server goes down, including "unfortunate network topology" where you're dragging a huge amount of unnecessary traffic across a slow bottleneck, and including "split brain" problems). "Single server" is not a solution, it's an excuse to cause a large number of unnecessary problems.

If the office burns down (and takes out all the stationary computers with it); then the laptops will still be running as a functional cluster (with no human intervention/reconfiguration). You'd be able to setup a tent in the car park and continue working as if nothing changed.

embryo2 wrote:
Brendan wrote:The "batch kernel functions" isn't just for 3 functions (for sending and receiving messages); it's for the entire kernel API; which includes virtual memory management, spawning and terminating threads and processes, controlling thread priorities, sleep/delays, finding "named services", handling "device resources permissions" (which process have access to which IO ports and memory mapped IO areas, and which threads are notified when an IRQ occurs), etc.
You want to tell the messaging can be actively used somewhere in the depth of the software stack. Ok, let it be, but why not to expose it in the very habitual manner, called "function call"? A developer can do something like this:

While it'd be possible to implement something like this on top of messaging; it changes none of the details of messaging; and I really do hate "syntactical sugar" that makes it harder for programmers to know what their code actually does.

embryo2 wrote:
Brendan wrote:The "messaging" abstraction helps because it means any thread can communicate with any other thread without caring where the other thread is;
If I do not know (shouldn't care) about underlying messaging it can help more.

A typical "call/ret" takes about 5 nanoseconds. Sending a packet (message) across a network typically takes 1000000 nanoseconds. If it's not obvious to programmers where their code will/won't potentially have 200000 times higher latency, you end up with "too high level" programmers slapping together worthless puke without even realising it.

embryo2 wrote:
Brendan wrote:and without caring if communication actually happens via. RAM buffers, or "raw InfiniBand packets", or "raw ethernet packets" or "TCP/IP across the Internet" or something else; and without caring about whether or not the actual data is compressed and/or encrypted, or split into multiple packets and reconstructed at the other end, or has some errors and needs to be retried/resent a few times, or....
Ok, let my call be optimized and dragged through the RAM, TCP/IP, HDD, DHCP, martian crater or whatever, but let also it be as simple as a trivial function call. That simple.

No. At a minimum you have to deal with the fact that (as soon as any networking is involved) there's no guarantee that the message will be delivered.

embryo2 wrote:
Brendan wrote:You're mixing development (where bikeshedding is a problem in early stages of the project, and there's no need for much communication between developers in the later stages, and is 100% technical initially but becomes 50% technical plus 50% social later), and promotion/marketing (which doesn't/shouldn't happen in early stages, but could be considered pure "social engineering" when it begins).
There's a single process of software creation and distribution (it's life-cycle). The software is always targeted for a human auditory even if it controls a blast furnace. Only people buy it and actually use it.

But yes, it's possible to do some technical stuff with a single person only. The outcome here is the time and some prospects of the future distribution drawbacks. The key decisions in this case should be made far before the auditory is allowed to test the product, so, the chances are the tests will show the key decisions were wrong and the cost of change is a lot of time again.

"Design by committee" is almost always worse than "design by sole inventor".

embryo2 wrote:
Brendan wrote:Realistically; it would most likely be done in stages - some very limited initial release/s intended to attract testers/programmers/artists/promoters (and not users), followed by a "grand unveiling" intended to create the largest "tidal wave of interest" possible (and attract as many normal users as possible), followed by regular updates from that point on.
Ok, but it's much closer now to what corporations do - they just shorten the stages a lot. So, there's a lot of very short stages. It's contiguous flow of releases and patches. One shot and immediate success without such flow is very unlikely to be the case. But you can try and may be the luck will be with you.

Most corporations wait until a product is usable, mostly complete and mostly stable; then (after that) have an initial public release and start doing a continuous flow of releases and patches. It's exactly the same as what I'm proposing.

Cheers,

Brendan

Posted: **Mon Dec 07, 2015 11:31 am**

Brendan wrote:it is a case of "I want to avoid end user hassle, therefore I need a smart algorithm".

And my objection was about "it's too smart to be implemented in 10 years". But it worth to define the requirements. If all the "hassle" is just the inability to distribute some load across a network, then may be your algorithm is acceptable. And if there other cases (like the standard server functionality), then the hassle of running services like DNS on all computers will be the reason to prefer existing solutions instead of your's.

Brendan wrote:Your alternative is:
Use a "single server" model that causes a large number of problems for no reason

Force humans to waste their time trying to work around half the problems that the "single server" model caused

Don't solve the other half of the problems that the "single server" model caused

Don't solve the additional problems that "human error" cause either

What are the "problems for no reason"? Are they completely described in the next 3 list items? Single server means less pain because you always have a need to provide it with some settings. It's addresses, users, policies and so on. Where these settings would come from in case of your distributed solution? And if something doesn't solve the "human error" problem then it does it for the reason - human errors are hard to predict. And I doubt your solution can do it.

Brendan wrote:I've already pointed out multiple problems with the "single server" idea (including fault tolerance when that single server goes down, including "unfortunate network topology" where you're dragging a huge amount of unnecessary traffic across a slow bottleneck, and including "split brain" problems). "Single server" is not a solution, it's an excuse to cause a large number of unnecessary problems.

Contemporary servers are fault tolerant. It is achieved with the help of redundancy. You are trying to achieve it with the help of the same redundancy, but using user's laptops instead of a specialized hardware. But specialized things usually much more efficient. The topology problem is another "miss" in your plan. If there's no database then there's no problems with topology. And if there's only a slow GPRS connection then again it's not the topology. And if there is free network bandwidth or there isn't such thing then it's about the network planning, but not about topology. The planning includes hardware like routers and cables. And your solution can only use existing (may be badly planned) resources. It's better to make the bottleneck visible for the person in charge to make better plans instead of hiding the bad outcome of the bad plan. Also it is unclear if you target the enterprise market or something for homework. In the latter case there's just no need for the tricky networking solutions. Here again I see the need for the requirements definition.

Brendan wrote:If the office burns down (and takes out all the stationary computers with it); then the laptops will still be running as a functional cluster (with no human intervention/reconfiguration). You'd be able to setup a tent in the car park and continue working as if nothing changed.

Ok, the greater fault tolerance can be achieved. But if there's no database then there's no "working as if nothing changed". Or do you plan to create a fault tolerant and distributed database? And all other enterprise level services? Imagine an enterprise which is working even if there's only one smartphone of it's CEO still alive, because all enterprise's systems are there

Brendan wrote:and I really do hate "syntactical sugar" that makes it harder for programmers to know what their code actually does.

When you work with your browser - do you still hate the way it constrains you by hiding it's source code? You just need a job to be done. And if your job is to develop a new browser, then you need some information about browser's internals, but it can be explained just as a plain text in browser's documentation.

Brendan wrote:
embryo2 wrote:If I do not know (shouldn't care) about underlying messaging it can help more.
A typical "call/ret" takes about 5 nanoseconds. Sending a packet (message) across a network typically takes 1000000 nanoseconds. If it's not obvious to programmers where their code will/won't potentially have 200000 times higher latency, you end up with "too high level" programmers slapping together worthless puke without even realising it.

Well, it means your system can make very bad mistakes (5 orders of magnitude) if it isn't monitored closely by a human. Then why do you describe your messaging as a better solution than existing technologies?

Brendan wrote:At a minimum you have to deal with the fact that (as soon as any networking is involved) there's no guarantee that the message will be delivered.

It's very simple. And you know many ways how to add the failure information to your library.

Brendan wrote:"Design by committee" is almost always worse than "design by sole inventor".

Often it is the case. But the resources available just make it irrelevant.

Posted: **Tue Dec 08, 2015 12:46 am**

Hi,

embryo2 wrote:
Brendan wrote:it is a case of "I want to avoid end user hassle, therefore I need a smart algorithm".
And my objection was about "it's too smart to be implemented in 10 years". But it worth to define the requirements. If all the "hassle" is just the inability to distribute some load across a network, then may be your algorithm is acceptable. And if there other cases (like the standard server functionality), then the hassle of running services like DNS on all computers will be the reason to prefer existing solutions instead of your's.
Brendan wrote:Your alternative is:
Use a "single server" model that causes a large number of problems for no reason

Force humans to waste their time trying to work around half the problems that the "single server" model caused

Don't solve the other half of the problems that the "single server" model caused

Don't solve the additional problems that "human error" cause either
What are the "problems for no reason"? Are they completely described in the next 3 list items? Single server means less pain because you always have a need to provide it with some settings. It's addresses, users, policies and so on. Where these settings would come from in case of your distributed solution?

These setting are completely unnecessary for my distributed solution - all computers know they make their own decisions without settings.

Where do the settings come from in your "single server with end user configuration" solution? If there are 20 computers, 2 ethernet switches and 21 network cables; then there are 33 points of failure where any number of them can be online or offline at any time. This works out to (roughly) 8 billion different cases. Do you invent a special programming language that system administrators can use to describe what should happen in each of these 8 billion cases (and expect people to rewrite it/maintain it whenever new computers and/or new networking equipment is added)? Is this "decision script" stored on each computer; and how do you deal with disk-less computers?

Or do you do the traditional "loss of service" method, where if a server or network connection goes down the you're screwed until an (authorised) human intervenes?

embryo2 wrote:
Brendan wrote:I've already pointed out multiple problems with the "single server" idea (including fault tolerance when that single server goes down, including "unfortunate network topology" where you're dragging a huge amount of unnecessary traffic across a slow bottleneck, and including "split brain" problems). "Single server" is not a solution, it's an excuse to cause a large number of unnecessary problems.
Contemporary servers are fault tolerant. It is achieved with the help of redundancy.

Yes; and to increase fault tolerance you increase redundancy. With N computers, the maximum fault tolerance possible is achieved by having the maximum redundancy possible; which is "all N computers make their own decisions".

embryo2 wrote:You are trying to achieve it with the help of the same redundancy, but using user's laptops instead of a specialized hardware. But specialized things usually much more efficient. The topology problem is another "miss" in your plan. If there's no database then there's no problems with topology. And if there's only a slow GPRS connection then again it's not the topology. And if there is free network bandwidth or there isn't such thing then it's about the network planning, but not about topology. The planning includes hardware like routers and cables. And your solution can only use existing (may be badly planned) resources. It's better to make the bottleneck visible for the person in charge to make better plans instead of hiding the bad outcome of the bad plan. Also it is unclear if you target the enterprise market or something for homework. In the latter case there's just no need for the tricky networking solutions. Here again I see the need for the requirements definition.

I'm trying to provide the best end user experience (including performance, fault tolerance, and end user hassle/configuration) with whatever hardware the end user happens to let the OS run on.

Part of the goal is to be able to go to typical small/medium businesses and convince them to switch from the current relatively standard approach (e.g. one cheap/desktop computer per user, an additional expensive/fault tolerant server, a bunch of networking equipment, and a non-distributed OS like Windows/Linux) to my approach (e.g. one cheap/desktop computer per pair of users, no expensive server, half the networking equipment, and my distributed OS) because this will:

significantly reduce purchase costs
significantly reduce running costs (e.g. less power consumption, less air-conditioning)
significantly reduce maintenance/administration costs
increase fault tolerance
increase performance (by distributing load to "otherwise idle" computers).

Also note that (in my opinion) there's 2 very different things - settings, and preferences. Preferences are things like which colour scheme the end user prefers, where it doesn't matter much if its wrong and the only one that can decide is the user themselves. Preferences are mostly fine (they indicate "end user customisability") and if the user couldn't be bothered changing any of the preferences then that's fine too.

Settings are very different - they always indicate design flaws and are "evil". For example; for PS/2 keyboards you mostly need a "default keyboard layout" setting, because there's a design flaw in hardware (no way for software to auto-detect keyboard layout). For another example; you mostly need a time-zone setting because (for stationary computers) there's a design flaw in firmware (OS can't ask firmware for the time zone when it's installed, where it can be set once for all OSs that are ever installed, possibly by the retailer before its purchased) and (for mobile devices and possibly stationary computers too) most of them have a design flaw in hardware (lack of GPS that would allow automatic time zone detection, even when user is moving between time zones). For another example; you might need an "is RTC local time or UTC?" setting because of historical design flaws (mostly in DOS) and compatibility. Note that all of these are examples of design flaws that an OS developer can't do much/anything about; but there are also settings that indicate design flaws in the OS itself (e.g. asking admin/user where different file systems should be mounted in the VFS).

embryo2 wrote:
Brendan wrote:If the office burns down (and takes out all the stationary computers with it); then the laptops will still be running as a functional cluster (with no human intervention/reconfiguration). You'd be able to setup a tent in the car park and continue working as if nothing changed.
Ok, the greater fault tolerance can be achieved. But if there's no database then there's no "working as if nothing changed". Or do you plan to create a fault tolerant and distributed database? And all other enterprise level services? Imagine an enterprise which is working even if there's only one smartphone of it's CEO still alive, because all enterprise's systems are there

What "database" are you talking about?

For the OS itself and for most normal applications there is no database (e.g. just software that deals with files). If there is some sort of database (e.g. the accountancy department uses a huge database of all financial transactions), and if that database's file/s are lost, then the laptops will still be running as a functional cluster (even if the accountancy software can't work until the database's file/s are restored from backup).

embryo2 wrote:
Brendan wrote:and I really do hate "syntactical sugar" that makes it harder for programmers to know what their code actually does.
When you work with your browser - do you still hate the way it constrains you by hiding it's source code? You just need a job to be done. And if your job is to develop a new browser, then you need some information about browser's internals, but it can be explained just as a plain text in browser's documentation.

When I work with the browser I'm not programming, I'm just using; and for people that are just using software the source code is irrelevant. If my job is to develop a new browser, then the only thing I need is a new job. If a person is writing a new web browser they need to write their new web browser's source code (and need to know what their source code does) and they might also need to write a plain text document that explains the internals of the code they've written (because different programmers implement the same thing in radically different ways, so plain text documentation for an existing/competing web browser is completely worthless).

embryo2 wrote:
Brendan wrote:
embryo2 wrote:If I do not know (shouldn't care) about underlying messaging it can help more.
A typical "call/ret" takes about 5 nanoseconds. Sending a packet (message) across a network typically takes 1000000 nanoseconds. If it's not obvious to programmers where their code will/won't potentially have 200000 times higher latency, you end up with "too high level" programmers slapping together worthless puke without even realising it.
Well, it means your system can make very bad mistakes (5 orders of magnitude) if it isn't monitored closely by a human. Then why do you describe your messaging as a better solution than existing technologies?

You'll have to be more specific - which claims about "better" are you referring to?

For programming (in general, and without having anything to do with messaging or communication at all in any way), programmers should be able to easily tell the difference between things that are extremely fast and things that are extremely expensive. For a random example; if you use C++ operator overloading to make it harder for programmers to notice the difference between extremely expensive big rational number addition and extremely fast 32-bit integer addition, then you're contributing to the "high level programmers slapping together worthless puke without even realising it" problem.

embryo2 wrote:
Brendan wrote:At a minimum you have to deal with the fact that (as soon as any networking is involved) there's no guarantee that the message will be delivered.
It's very simple. And you know many ways how to add the failure information to your library.

It's not simple, I don't know any sane way to add the failure information to a library, and don't have libraries in the first place. I have (potentially remote) services instead of libraries; where there's no guarantee a message that was successfully sent to a service/library will be successfully received by the service/library.

embryo2 wrote:
Brendan wrote:"Design by committee" is almost always worse than "design by sole inventor".
Often it is the case. But the resources available just make it irrelevant.

The only resource that matters is developer time. In the early stages wasting the only resource that matters (my time) on "bikeshedding with others" in the hope of achieving "worse due to designed by committee" doesn't seem like a sane idea at all.

If you don't agree; then how about this: Instead of developing your OS by yourself, how about if I volunteer to help your project (and do this by continually insisting that all traces of Java and "managed puss" be removed)? The benefits of "more developers" would really help your project!

Cheers,

Brendan

OSDev.org

How change non-PAE paging mode to PAE paging mode?

Re: How change non-PAE paging mode to PAE paging mode?

Re: How change non-PAE paging mode to PAE paging mode?

Re: How change non-PAE paging mode to PAE paging mode?

Re: How change non-PAE paging mode to PAE paging mode?

Re: How change non-PAE paging mode to PAE paging mode?

Re: How change non-PAE paging mode to PAE paging mode?

Re: How change non-PAE paging mode to PAE paging mode?

Re: How change non-PAE paging mode to PAE paging mode?

Re: How change non-PAE paging mode to PAE paging mode?

Re: How change non-PAE paging mode to PAE paging mode?

Re: How change non-PAE paging mode to PAE paging mode?

Re: How change non-PAE paging mode to PAE paging mode?

Re: How change non-PAE paging mode to PAE paging mode?

Re: How change non-PAE paging mode to PAE paging mode?

Re: How change non-PAE paging mode to PAE paging mode?