OSDev.org

I'd be interested to know how much you're all working together. Clearly some of you have created an OS purely for fun or learning purposes and have no real expectation of running anything much on top of it, but quite a few of you appear to have dreams about making your OS a big player on the world stage some day. That is my hope for my OS too, but what chance will it have if it can only run applications written specifically for it? It would be far better if we all tried to write our OSes in such a way that apps written to run on any one of them would be able to run on all the others too. Have any of you got together to try to come up with a common standard to enable this to happen?

I may be at a disadvantage here in that I haven't studied programming in the normal way, so I have now idea how an application communicates with Windows or Linux, but I did learn at one time how to get an app to communicate with CP/M on the Amstrad PCW, tracking down books which gave details of all the services available. I've read somewhere on this forum that it's unrealistic to think that you could run a Windows application on your OS, and I can well believe that, but I don't know if the same is true of Linux - I've always avoided looking into Linux because I didn't want to be influenced by it, so maybe they've already set the standards, or maybe it's just as unrealistic to think that you can run a Linux app on your OS as a Windows app? Whatever the case may be, it looks as if there should be a standard system for us all to work to so that all apps written by any of us automatically enhance the usefulness of all our OSes. The same applies to device drivers - should we not be aiming to use the same ones as much as possible so that we all make progress tens of times faster than by working alone and reinventing the wheel over and over again?

It may not be so simple, of course, as some differences in design will make it impossible for this to happen with certain OSes, particularly when a designer wants to do everything in a particular mode and not pollute the system with other people's ideas, but the road to success surely has to involve common standards and common components. I personally have an OS which doesn't use the paging system or segments, so all my apps (written in indexed machine code) have to be modified to run at the memory location they're loaded into whenever they load (the indexing system allowing that to happen in a fraction of a second), but there's no reason why I shouldn't also be able to provide a paged environment where apps designed to run on other OSes could run, and I could easily design my own apps to run in that environment too, giving them two different ways of linking into my OS, one of which would work on other people's OSes without modification.

Does this approach appeal to anyone? Perhaps you're already doing it? I'm new to this forum (I wish I'd known about it years ago) and although I've read through a lot of threads in the last few days trying to find out what's what, I might well have missed something or I might not have looked far enough back in time.

A lot to learn, you have, young padawan.

Thing is, everything platform-specific is non-portable. Everything platform-independent is portable. You can write in C99, using only the standard libraries, but you can never take advantage of the specific capabilities of system X.

The other thing is, there is already a ton of portable stuff around. Most of the things you might want to use already exists in the form of most of the shell utilities you see in for example linux. You can get compilers, linkers and everything to run on your OS without little more than a recompile. There is little to no work to be done in this regard, to the point that if you do want something you are partaking in application development rather than OS development.

Combuster wrote:A lot to learn, you have, young padawan.

Indeed, but I am determined to become an OS Jedi master.

Combuster wrote:Thing is, everything platform-specific is non-portable.

It was possible to run MS DOS apps on DR DOS, so there is no reason in principle why a number of different operating systems shouldn't be able to run identical apps if they're designed to be compatible. That doesn't mean the OSes can't be substantially different under the hood and be used for very specific purposes by the designer.

Everything platform-independent is portable. You can write in C99, using only the standard libraries, but you can never take advantage of the specific capabilities of system X.

Usually there are no specific capabilities that you need to take advantage of - an app is simply written to run on a specific OS because that OS is dominant. If you want to run it on a different OS, you have no way of recompiling it if the company that produced the app isn't interested in making it possible for you to run it on some little OS they've never heard of, but if there was a standard system for apps to use to communicate with multiple OSs, some of the lesser-known companies might be interested because they could capture that part of the market and begin to develop a following. It might in the future also be considered to be anti-competitive not to make your apps work with the standard system to enable them to run on any OS that meets that standard, so long as there's no particular thing which a particular app does which those OSes can't support.

The other thing is, there is already a ton of portable stuff around. Most of the things you might want to use already exists in the form of most of the shell utilities you see in for example linux. You can get compilers, linkers and everything to run on your OS without little more than a recompile. There is little to no work to be done in this regard, to the point that if you do want something you are partaking in application development rather than OS development.

So why do so many people appear to get stuck at the same point with an OS that can't do very much if there's a stack of parts just waiting for them to bolt onto it? Is the problem that your OS gets heavy restrictions put on it if you include these parts with it, or have I missed part of this site where I can find a list of things I can add to my OS with no strings attached, perhaps giving it the ability to use photo-printers and work with a wide range of USB devices? I doubt it. But other OS creators may be willing to team up to share components without restrictions, and to work together to develop them.

berkus wrote:Third, on this forum people use OS development as a sort of rewarding and fairly extreme dive into the hardcore, hardware and sometimes hard to grasp concepts and systems and use it mostly for educating themselves, think of it as "OSdev University".

Yes, I can see that a lot of people don't have any plans to take over the world with their OS and don't feel the need to move things on once they've climbed their own Everest and seen the beautiful view from the top.

DavidCooper wrote:But other OS creators may be willing to team up to share components without restrictions, and to work together to develop them.

There are several efforts like this on this forum alone (UDI, CDI). If you take some time to read you will certainly spot them.

Thanks - that's the lead I was looking for. I have no idea which things here relate to Linux and which are independent, but now you've given me a couple of things to search for.

(Edit: http://forum.osdev.org/viewtopic.php?f= ... DI#p163044 - this certainly sounds like the best route to follow.)

The question of making interoperable components goes down to specifying API and ABI. That is all. There are emulation layers (wine, posix) that allow you to run "apps on each other's operating systems", but for a hobby OS project diving into writing of a (fairly large) emulation layer before you finished your own kernel is a kind of suicide.

I wasn't thinking in terms of emulation, but apps that run directly through the processor in machine code on a variety of different OSes. What was possible with MS-DOS and DR-DOS may not be possible any more with all the extra complexity of OSes now, but it seems likely to me that some OSes could be made compatible and it could be highly advantageous for their creators to work together to that end. It's quite possible that a number of OSes could be modified in fairly trivial ways to enable the same apps to run directly on all of them, so from my point of view I'd like to know if my OS could fit within such a group, even if it involves adding in an alternative way for apps to call OS services. A lot of programs that people write are very simple and depend on very ordinary OS services which could easily be called using a common system, and I think it would make the whole business of creating your own OS a lot more rewarding if on completing it you were immediately able to run hundreds of apps on it written by other people for their OSes.

MS-DOS and DR-DOS don't share interfaces coincidentally. I wasn't around then, but the same concepts still apply. One came first, and was popular, and had many apps made for the interface it provides. The other came scond, and wanted to be compatible with the other one with an already established app base, so it emulated the other's interfaces. In order to give a concrete example, I'll assume MS-DOS came first. MS-DOS uses int 21 for the most of it's api. So when DR-DOS was being made, the coders knew MS-DOS was already popular, and had lots of apps that used int 21. So they made an isr with the same function as MS-DOS, so that MS-DOS apps could run on DR-DOS. It was all intentional. The only difference now is the complexity of the apis and the different ways to provide access to them.

Nothing is platform Independant. Everyting is platform dependant. Yes we can consider that supports multiple platform.

Take an Example,
You will find in book Java-APP are Platform Independant, But do u think so, Will Java-APP run in my System. Yes Java is Platform independant because JVM is platform Dependant.

You speak in circles. It would help if your arguments would not be contradicting each other and themselves.

TylerAnon wrote:MS-DOS and DR-DOS don't share interfaces coincidentally. I wasn't around then, but the same concepts still apply. One came first, and was popular, and had many apps made for the interface it provides. The other came scond, and wanted to be compatible with the other one with an already established app base, so it emulated the other's interfaces.

Right, I can see that I might be being confused by the word "emulated" which I've been taking as referring to code being run indirectly (interpreted), but here you appear to mean duplicating functionality with equivalent code (not duplicate code). That means there's nothing extra that needs to be in the OS and no speed cost either - it's just two OSes doing the same job from the app's point of view with potentially equal efficiency and simplicity.

In order to give a concrete example, I'll assume MS-DOS came first. MS-DOS uses int 21 for the most of it's api. So when DR-DOS was being made, the coders knew MS-DOS was already popular, and had lots of apps that used int 21. So they made an isr with the same function as MS-DOS, so that MS-DOS apps could run on DR-DOS. It was all intentional.

I've read that the original DR-DOS was incompatible with apps written to run on MS-DOS, presumably being more of a translation of CP/M to run on machines with x86 processors, but they missed the boat on that one (Gary Kildall was rather too abrupt with IBM in a phone call, hence IBM calling Microsoft instead who then rushed to buy QDOS, renaming it MS-DOS). Anyway, D.R. ended up having to copy the MS-DOS service call system to try to get some market share.

The only difference now is the complexity of the apis and the different ways to provide access to them.

Yes, but how much of that complexity would be necessary for a new standard system which doesn't attempt to run Windows apps? I'm assuming that everyone here who's written an OS must either have created their own system for their apps to call OS services or they have an OS that doesn't work because it's waiting for UDI to be added some day. If the former, it might make sense for some OS designers to work together to use identical ways of calling OS services, particularly as many apps only need to interact with mouse, keyboard, screen and floppy or hard disk.

berkus wrote:The code runs natively on the CPU most of the times (see x86 windows apps running on x86 linux in x86 wine), but you still need to emulate the API, which is different between different OSes.

I'm guessing you mean emulate in the sense that the code's doing some sort of translation step to convert calls to OS services into the non-Windows API form that the OS can handle.

UDI is trying to remove this impediment by specifying a consistent API for multiple different OSes.

Is enough of UDI ready to enable it to be used right now for an OS that might only need to use mouse, keyboard, screen, and floppy and hard drives? If so, wouldn't it make sense for OS designers to try to fit in with it now, assuming it's compatible with the way they've put their OS together? And if it isn't ready, it might still be worth working to some common blueprint which would make your OS and apps easier to adapt to using UDI at a later stage, while also enabling many apps to run on a wide range of different OSes much sooner.

With my own OS all my apps are run in the same address space as the OS where they use near calls to the OS instead of the int system (the indexed machine code enabling the links to be set up on loading regardless of where the code is put), so that leaves the int system completely free for running someone else's API - I could then tie the two systems to the same routines, with a translation step wherever it's necessary. I want to retain my own system so that I can continue to run my own apps in the same address space as the OS where they can all interact with each other easily (or more importantly be connected together by an A.I. program), but they can be written in such a way as to be able to run through the alternative system as well where they'll have their own address space and communicate with the OS using int instead. The near calls to the OS would normally be stored in the code as near jumps (233, 2, 0, 0, 0 - jumping just past a following 235 jump instruction and its single jump-distance byte) and it's only if the app is loaded into the same address space as the OS that the 233 jump instructions would be turned into 232 calls on loading, the new jump distance for the call being inserted automatically by the indexing system (and the following 235 jump instruction then bypassing the alternative code which uses int). There would be no speed cost while running the program by either method, so there's no reason not to write the app in such a way that it can run both ways from the same program file. Incidentally, the indexes are typically 3/4 the size of the machine code program, so it takes up more space than normal programs in that regard, but the indexes can simply be left out if the code is loaded into its own address space (the cost being that they aren't then in a form that a programmer or A.I. can instantly modify whenever the need arises).

Let's immediately drop UDI right here and now: it's been debated more than enough. When some implementation effort is applied then more discussion will be warranted IMHO.

I like the idea, but it requires a lot of effort. RDOS does have plug-in device-drivers to be able to run many pure DOS programs, as well as DOS-extended programs. It's native executable format is not "custom-made", rather is identical to Win32s (PE executables). This interface is simply a device-driver module that loads PE executables. RDOS can also run some character-mode Win32 applications (like TD32 from Borland) with the help of replacement KERNEL32, USER32 and some other DLLs. And it is perfectly possible to plug-in an ELF-loader and run *nix-type executables made with GCC.

I never saw any reason to hard-code kernel to only run user-applications for just one tool-set, and since there has been major changes in popular tool-sets since I started, this concept has proved useful. For the same reason I never tied the kernel to any single C-compiler, but instead coded it in assembler from the start.

rdos wrote:I like the idea, but it requires a lot of effort. RDOS does have plug-in device-drivers to be able to run many pure DOS programs, as well as DOS-extended programs. It's native executable format is not "custom-made", rather is identical to Win32s (PE executables). This interface is simply a device-driver module that loads PE executables. RDOS can also run some character-mode Win32 applications (like TD32 from Borland) with the help of replacement KERNEL32, USER32 and some other DLLs. And it is perfectly possible to plug-in an ELF-loader and run *nix-type executables made with GCC.

That's given me quite a few more things to research, but what I'm finding isn't encouraging. I've just read up on the 86open project which was abandoned when lxrun met all of its aims, and yet the list I've just found of Linux apps that work with lxrun appears to be tiny (http://www.ugcs.caltech.edu/~steven/lxr ... index.html). It doesn't look as if there's a lot of point in trying to organise a complex standard for anything if all that effort leads to so little gain, but there may be some point in hobby OSes accommodating a common app format and service call system (in addition to their own native one) so that lots of simple programs can be shared.

What I want to do with my OS is make it a platform for all kinds of artificial intelligence experiments, so I don't really need the icing on the cake of being able to do video editing or anything like that on it: disk, screen and keyboard is all I need for my linguistics work, but I would like to be able to get input from a pair of webcams and easy access to uncompressed video from them, all in a simple environment where I can write simple machine code programs to try to analyse images without the need to study other people's complex systems for many years first just in order to find out how to get to the point where I can start the real work. I'd also like to be able to get sound in and to have easy access to that data so that I can work on speech and sound recognition - I did a lot of this kind of thing long ago on a ZX Spectrum+3 using the tape interface and was able to tell all the speech sounds apart even though the input was just a string of zeros and ones where the value was determined solely by which direction the magnet had moved in the microphone (perhaps a zero for central position or further in, and a one for further out). I was able to record the input to memory and then play it back to the speaker where it was just about intelligible, but the patterns in the zeros and ones on the screen were clearer. There are lots of intelligent people who could be doing this kind of work with very little programming knowledge, but you can't get started on it with OSes like Windows without having to study tons of other stuff first.

That is why I though it could be good to team up with other OS writers to come up with common standards for a limited set of simple things that would enable us to promote our OSes as A.I. research platforms for ordinary people who just want to get at the data and get analysing. Think about trees. Look at all the different lines running in different directions and the different percentages of lines running in different directions - they have very distinctive patterns to them which not only shout "tree" at you, but they also tell you which species of tree you're looking at. There are a million children out there with fantastic minds, locked up in education institutions which waste their time: they'd love to get involved in this kind of stuff. They could collectively make a great many breakthroughs that drive us towards human-level A.I., just by thinking out of the box and trying out different ideas. My own work relates to getting a machine to pass the Turing Test (that's not my direct aim, but is will be automatic consequence of it), but even if it reaches that level it will be a blind and deaf intelligence, so there will be a lot more to be done. The important point, however, is that it isn't something you should need a lot of experience to do: most of the experts are floundering because they are at a complete loss as to where to begin (although self-driving cars and face detection are now a reality), but if you get a million gifted children involved, some of them will strike gold. Linguistics is hard (I've been involved in astronomical amounts of componential analysis for over two decades - work I began as a child) and there are no shortcuts there, but vision is something that some really simple animals can do, so the algorithms are almost certainly very simple - it's just a matter of finding them, preferably in the right order, and then building them into an advanced visual analysis system which can identify patterns that can be mapped to concepts.

The problem with an unified API is that people will never be able to agree to what is "best". There will be as many solutions as there are programmers, especially if the API is as complex as an OS API. I think Windows (and Unix) interfaces are so bad that I would never consider them as native interfaces in my OS. I can make emulation layers for providing the interface, but I would never use them as a native interface. For instance, both systems have the terrible "IOCTL-think" that even predates MS-DOS, and it is the worse thing ever designed. Another issue is with minimizing "state" information exchange with kernel. This means NO (or almost no) data-structures in the interface. Today the RDOS user API contains 407 functions which use 2 data-structures (ThreadState and Tss). Another design-goal is not to use enumerated types, and there are 2 enumerated types in the interface (File attributes and LGOP for drawing).

So, the problem is personal preferences, and when people do things for fun, we would not expect them to accept solutions that aren't "right" for them.

I'm aiming to provide a kernel API that looks as close to the common FreePascal RTL functions as possible. Simply to make porting extremely easy

In a previous iteration I managed to get the compiler to run with minimal interwork code

Device drivers are tightly linked into the kernel at compile time, so, no for them

You take what can be considered, in a way, the diametric oposite of what I'm working towards: My OS API looks very different from most existing designs. It also has the interesting factor of looking nothing like the kernel API; in fact, the kernel API/ABI is not considered a stable interface, and applications should not interact directly with the kernel.

A relatively advanced project of mine is building an emulation layer which allows Fusion applications to run on top of POSIXy operating systems; this will have the odd result of libFusion being an application in these cases (since libFusion is where the runtime dynamic linker lives)

A unified API doesn't need to do everything the best way, just so long as it does everything in reasonable, low-complexity ways, and without making a mess in your machine - you would use it as an addition to your own native API, so you'd still have your own programs communicating with your own OS in what you consider to be the best way. Even so, if it ever happens, it sounds as if it's going to be so far off in the future that A.I. will probably get in first and enable any app or device driver to be modified to run natively on any OS without anyone having to do any of the grindingly dull translation work, so putting years of effort into creating a unified API may by now be a complete waste of time. All apps will need to be rewritten to work with A.I. as well (to take full advantage of its input), and no doubt it'll have a lot to say about OS design too, so we'll probably end up with a completely new OS replacing everything that's gone before it and which none of us can hope to better in any aspect.

It's worth thinking about whether any work you're doing is actually going to live on in in any meaningful way when A.I. gets to the point where it can duplicate it in the blink of an eye and then reject it all just as quickly in favour of other ideas. So, if I'm not missing out on being in any club of OS designers who have already made their OSes compatible with each others' apps and device drivers in some simple way that already gives them all easy access to a basic range of devices like USB flash drives and webcams, I probably shouldn't spend any more time thinking about it. Instead I will concentrate on trying to find out how to program for the two or three specific devices I really need to add and then leave it at that.

Or maybe I should brush up my German and have a look at CDI.