OSDev.org

I know that we all are deeply dedicated to our own projects and we have time for ourselves hardly.

But it's a truth that we could advance faster if we organized and would start a common OS project, in which we all dedicate to different pieces of it. Since it would be a cooperative effort and more orientated to learn, it could even be the only true public-domain-source OS, but even without being like so, it would only take a fraction of the time to build given we already have information and could be integrated there, and it could even lack optimization.

Why not do that? It looks to me that it would be a good bet to finally go beyond the same development level we have seen until now in these homebrew OS projects, and the incentive would be that anybody could use it freely, and the gain wouldn't be much a propietary one, but an accelerated learning and development; and being this a relatively small group, it could be managed more ordered than the traditional GNU pals.

This is very serious, why on Earth not make an unified effort to go for this kind of OS? It would rapidly give its results and would be a very advancement in all aspects of building a better OS that now exists. So, seriously, how to do it?

I pesonally offer a big deal of my time to start organizing it, if anybody else lacks time, to try and put together pieces of code even if not optimized, but the idea here is to join efforts. I'm convinced that would be key to a not before seen development in this project area.

HOBBY = not common, we're doing this for fun
People also have VERY different tastes in OS'.

Subject has been around a dozen times... I think everything has been said on this subject already. Please use "search" function.

I've got an idea: better to take the current projects and start there.

The fact they haven't been explicitly weaved as pieces here and there to make a "common effort", doesn't mean they can't be fused, by me, you, whoever takes it... just being careful with not stealing the actual work of others...

IMO, my point still stands.

ehird wrote:IMO, my point still stands.

I don't know what your point is; I guess it's an unification of what has been expressed in this thread and the ideology around this place...

@ehird: ~ is very well allowed to take together free parts of oses to build his own, independent of what fundamentalist ideas there are attached to it, So i guess your point does not hold.

@Solar: IMO The fact that the majority does not like the idea of working on a community OS doesn't mean there are not enough people willing to take on the adventure. Surely the topic has been posted 12 times - there must be at least as many people interested in cooperating?

@~: There are better ways to get people than to post a thread inviting people. In the meantime, you may want to look around my codebase. I didn't license it public domain for nothing

If everyone wanted the same thing, we could just go hacking Linux. It was a hobby originally, and for many people it's still a hobby, even if it's grown to be big. For some reason though, people keep thinking of alternative designs.

The only way a community project of that sort would work, would be if the 'OS' would be completely made into very separate modules each developed by different people. This way, the entire group doesn't decide on a feature, but the group actually writing it. Granted, you would still have to all decide on the major issues on design and how these modules come together, but that could be figured out by a direct head, or project leader who would be someone picked out of the community to head it.

Ahem....I can't speak today. Please excuse the above if it sounds horribly confusing.

ehird wrote:HOBBY = not common, we're doing this for fun
People also have VERY different tastes in OS'.

Don't speak for the community... I personally post here out of my interest in OSDev and not in relation to any of my actual development, but my OS happens to be serious all the way and in no way a hobby.

Alboin wrote:The only way a community project of that sort would work, would be if the 'OS' would be completely made into very separate modules each developed by different people. This way, the entire group doesn't decide on a feature, but the group actually writing it. Granted, you would still have to all decide on the major issues on design and how these modules come together, but that could be figured out by a direct head, or project leader who would be someone picked out of the community to head it.

The idea is beautiful, but there is a certain problem. It is very hard to design a sound kernel to be modular enough, that you can hope to have different people implement different parts of it.

Basicly, in a typical kernel, there's the following relatively independent modules:

- kernel proper (threading/scheduling/interrupts/semaphores/timers)
- memore management (physical/virtual/kernel heap)
- IO (filesystems/IPC)
- drivers, each largely independent, though interfaces must be common
- userspace issues (processes/signals/open files)

Now, kernel proper will depend on some memory management, while everything else will depend on kernel proper (memory management possibly least). IO will depend on kernel proper and memory management. Drivers will depend on IO, memory, and kernel proper. Userspace will depend on everything.

However, the kernel proper will need to be aware of userspace issues to some extent, so in the end you end up with a system where everything depend on everything, at least a bit. That would mean you'd need to have a finished design before you can easily separate the tasks. The internal design of each module will likely have effects on what it needs from other modules.

The only thing that's pretty easy to separate is drivers. Those are also most work, if you want to support more than a little hardware. But you need to have a clean interface to write the drivers against, if you want to develop them separately.

We've had various projects about sharing drivers. The idea is nice. However, to this date, nobody has been able to come up with a good enough set of interfaces to write the drivers against. Some have been proposed, but most have been either too naive, or simply too much work to support from the kernel side. I think the only way to come up with such a design is to try to write some non-trivial driver, and then find out what it takes to port it to run under several kernels.

There are some drivers ofcourse, which there is no point of sharing. Timers and interrupt controllers are too fundamental to abstract, because not only the rest of the kernel, but every other driver will depend on them. Most will want console output available before even interrupts have been initialized, so that's another thing that needs to be special cased for each kernel (though code is mostly trivial).

The problem then, is that certain amount of completeness will be required from rest of the kernel to support any useful driver interface. Not all projects here get that far. Interrupts and timers must be available and multi-threading makes many drivers easier to write (especially as you want to do as little as possible from the interrupt handlers). Some relatively painful method for allocating memory (like malloc and friends) will be necessary for all but trivial drivers. Multi-threading drivers will depend on semaphores (more advanced locks are easy to build on those even on per-driver basis). Finally any driver will have to have some method of offering it's service, though this is best defined separately for different classes of drivers, for obvious reasons.

The day somebody manages to define a driver-interface that is complete, flexible, yet simple enough to be reasonable, I'll be the first to support it. Until then, the only way to share projects that I can see, is for someone to help someone else, either with advice, or by supplying patches (either bugfixes or new features) to a codebase that already has it's fundamental basic design up and running (it could be changed organically later, but it must be in state of having a form in code).

I have some plans on proposing certain interfaces for certain classes of drivers in the future. Display drivers, network drivers, maybe sound drivers would be nice things to share. But that'll take until I've actually tried to support some drivers in my own kernel, so that I have a better idea of the requirements for each class.

Finally, userspace issues could be separated into modules ofcourse, but writing userspace code is the 'easiest' part, and I think existing methods for writing portable software already address this.

mystran wrote:However, the kernel proper will need to be aware of userspace issues to some extent, so in the end you end up with a system where everything depend on everything, at least a bit. That would mean you'd need to have a finished design before you can easily separate the tasks. The internal design of each module will likely have effects on what it needs from other modules.

I think that there are ways around this. What about having a dependency system, which glues together the modules? So that if module A needed module B, it could ask the dependency system to load module B and any other dependencies that it may carry. With each team knowing the interface of the other teams, they could all start developing at once and assume that by the time they finish, the other team will have their dependencies finished. However, this would mean detailed planning by each group before the coding actually started.

mystran wrote:We've had various projects about sharing drivers. The idea is nice. However, to this date, nobody has been able to come up with a good enough set of interfaces to write the drivers against. Some have been proposed, but most have been either too naive, or simply too much work to support from the kernel side. I think the only way to come up with such a design is to try to write some non-trivial driver, and then find out what it takes to port it to run under several kernels.

The problem with common drivers is that since most everyone here is working on a hobby OS and many for learning purposes, I don't think they'd want to use someone else's code. At least I wouldn't.

mystran wrote:
There are some drivers ofcourse, which there is no point of sharing. Timers and interrupt controllers are too fundamental to abstract, because not only the rest of the kernel, but every other driver will depend on them. Most will want console output available before even interrupts have been initialized, so that's another thing that needs to be special cased for each kernel (though code is mostly trivial).

I wrote my 8259 Programmable Interrupt Controller as a loadable module which is abstracted using a IRQ routing system. I also did the same with the 8253 Programmable Interrupt Timer which is abstracted by clock which supports multiple clocks, and it decides which clock to use - but never the less my point is it is abstracted completely so that even the scheduler and drivers think they are dealing with just one clock at all times no matter what the implementation really is behind the scenes. And just to beep on my horn a little the clock uses the IRQ router abstraction. The only problem is having a birds eye view of the big picture.

If you would like to see my code to know what I am talking about I would not mind, and yes I currently have it working and drivers are using both abstractions.

Here is the driver for the floppy I am working on at the moment the point being it is using the abstractions, and the generic idea of it not being too complicated to get a general interface designed.

#include "shared.h"
#include <asm/io.h>
#include "dev.h"
#include "error.h"
#include "irq.h"
#include "kheap.h"

struct fdPrivData{
u16 base;
u8 state;
u8 drive;
};
typedef struct fdPrivData PRIVDATA;
typedef PRIVDATA* PPRIVDATA;

#define FD_IRQ 0x6
#define FD_BASE ((PPRIVDATA)priv->instance)->base
#define FD_STATA ( ((PPRIVDATA)priv->instance)->base+0x0)
#define FD_STATB (((PPRIVDATA)priv->instance)->base+0x1)
#define FD_DOR (((PPRIVDATA)priv->instance)->base+0x2)
#define FD_TDR (((PPRIVDATA)priv->instance)->base+0x3)
#define FD_MSR (((PPRIVDATA)priv->instance)->base+0x4)
#define FD_DRSR (((PPRIVDATA)priv->instance)->base+0x4)
#define FD_DATA (((PPRIVDATA)priv->instance)->base+0x5)
#define FD_RESERVED (((PPRIVDATA)priv->instance)->base+0x6)
#define FD_DIR (((PPRIVDATA)priv->instance)->base+0x7)
#define FD_CCR (((PPRIVDATA)priv->instance)->base+0x7)

enum DRATE{RATE1MBPS = 0,RATE500KBPS = 1,RATE300KBPS = 2,RATE250KBPS = 3};
struct{
u32 drate;
u8 drsr_bits;
u8 ccr_bits;
} precompDelays[] = {
{1000000, (0x1<<2), 0x3},
{ 500000, (0x3<<2), 0x0},
{ 300000, (0x3<<2), 0x1},
{ 250000, (0x3<<2), 0x2}
};

struct que{
u32 lba;
u32 buffer;
PPRIVATE priv;
};

static inline void setdrate(PPRIVATE priv, u8 rate){
outb((inb(FD_DRSR)&0xE)|precompDelays[rate].drsr_bits|precompDelays[rate].ccr_bits, FD_DRSR);
outb(precompDelays[RATE250KBPS].ccr_bits, FD_CCR);
}

#define CMD_READDATA 0x6
#define CMD_SENSEINTERRUPT 0x8
#define CMD_SPECIFY 0x3

static inline void specify(PPRIVATE priv){
outb(CMD_SPECIFY, FD_DATA);
outb(0xff, FD_DATA);
outb(0xff, FD_DATA);
}
static inline void readsector(PPRIVATE priv, u32 lba){
u8 head = 1;
u8 track = 1;
u8 sector = 1;
u8 drive = ((PPRIVDATA)priv->instance)->drive;


outb(0x40 | CMD_READDATA, FD_DATA);
outb(drive | (head<<2), FD_DATA);
// cylinder
outb(track, FD_DATA);
// head
outb(head, FD_DATA);
// sector R
outb(sector, FD_DATA);
// sector size 512 bytes N
outb(0x02, FD_DATA);
// eot
outb(79, FD_DATA);
// gpl
outb(128, FD_DATA);
//
outb(0xff, FD_DATA);
return;
}

static inline void senseinterrupt(PPRIVATE priv){
outb(CMD_SENSEINTERRUPT, FD_DATA);
inb(FD_DATA);
inb(FD_DATA);
}

static void InterruptThread(PPRIVATE priv){
if(((PPRIVDATA)priv->instance)->state == 0){
((PPRIVDATA)priv->instance)->state = 1;
/// some debug info for the instance of this driver
dv(((PPRIVDATA)priv->instance)->base);
dv(((PPRIVDATA)priv->instance)->state);
dv(((PPRIVDATA)priv->instance)->drive);

/// turn floppy motor on..
outb(inb(FD_DOR) | 0x10, FD_DOR);

dv(inb(FD_MSR));
/// turn DMA off.
specify(priv);

/// read sector
dv(inb(FD_MSR));
readsector(priv, 0);
dv(inb(FD_MSR));
console_write("okay waiting for floppy interrupt\x10");
return;
}
console_write("unimplemented state \x10");

dv(inb(FD_DATA));
dv(inb(FD_DATA));
dv(inb(FD_DATA));
dv(inb(FD_DATA));
dv(inb(FD_DATA));
dv(inb(FD_DATA));
dv(inb(FD_DATA));
return;
}
static i32 Interrupt(PPRIVATE priv, IRQVECTOR vector){
u32 x;
senseinterrupt(priv); /// tell the floppy controller to shut the **** up!
console_write("[FLOPPY] GOT A INTERRUPT!\x10");
////////////////////////////////////////////
dv(inb(FD_MSR));
////////////////////////////////////////////
sched_kernel_thread_create((u32)&InterruptThread, priv);
return IO_SUCCESS;
}

DRIVER floppy_generic;
i32 floppy_generic_init(void){
/// create a instance for each drive that is detected..
PDEVINSTANCE inst;
console_write("[FLOPPY] detecting controller. \x10");

if(inb(0x3f0+0x2) == 0xFF){
/// floppy drive was not detected.
console_write("[FLOPPY] failed on detected controller. \x10");
return IO_UNSUPPORTED;
}
inst = (PDEVINSTANCE)dev_CreateInstance(&floppy_generic);
inst->priv.instance = (void*)kalloc(sizeof(PRIVDATA));
((PPRIVDATA)inst->priv.instance)->base = 0x3f0;
((PPRIVDATA)inst->priv.instance)->state = 0x0;
((PPRIVDATA)inst->priv.instance)->drive = 0x0;


dv(&inst->priv);
dv(inst->priv.instance);

console_write("[FLOPPY] detecting drives \x10");
/// register interrupt
irqrouter_RegisterInterrupt(&inst->priv, &Interrupt, FD_IRQ+IRQROUTER_HWBASEVECTOR, IRQROUTER_DIRECTCALL);

/// reset controller
//outb(0x00, 0x3f0+0x2);
//outb(0x0C, 0x3f0+0x2);

/// continue initialization (select drive one)
//outb(0x10, FD_DOR);
/// try to change data rate and see if effect happens.
//setdrate(FD_BASE, RATE300KBPS);

//for(;;);
//console_write("[8259PIC] Initializing \x10");

return IO_SUCCESS;
}

static DEV_CONNECTION_MOTHERBOARD_DATA connectionDataMotherboard = {
DEV_MOTHERBOARD_FLOPPY,
&floppy_generic_init
};

DRIVER floppy_generic = {
.szName = "FLOPPYGENERIC",
.connection = DEV_CONNECTION_MOTHERBOARD,
.connectionVersion = DEV_CONNECTION_MOTHERBOARD_VERSION,
.connectionData = &connectionDataMotherboard,
.priv = 0,
.ifaceCount = 0,
.iface = {
{
.szName = "VFS.IO",
.major = DEV_VFS,
.minor = DEV_VFS_IO,
.iface = 0
}
}
};


A clock driver which has been fully, so far, abstracted by the kernel as a set of functions I call the clock since the source file is also clock.c. I think the frequency math for setting the clock is flawed which is due to the school of hard knocks. Never the less this one is pretty much finished except for fixing the bug in the set cycles per second function.

#include "shared.h"
#include <asm/io.h>
#include "dev.h"
#include "clock.h"
#include "error.h"

#define BASE 0x40
#define COUNTER0 (0x0+0x40)
#define COUNTER1 (0x1+0x40)
#define COUNTER2 (0x2+0x40)
#define CONTROL (0x3+0x40)
#define IRQ 0x0

#define MODE_INTONTERMINALCOUNT 0x0
#define MODE_SQUAREWAVE 0x3

// 1,193,181Hz or 1.193181MHz
#define FREQUENCY 1193181

/// Requested Hertz and Current Hertz show that the clock may not be able to provide the request cycles per second, but instead
/// will pick the closest value possible. These are used to track the state of the clock for now.
static CLOCKHZ curHz = 0;
static CLOCKHZ reqHz = 0;
static ICLOCK_CLOCK ifaceClockClock;

/// A quick routine that will change the clock's cycles per second.
inline void c8253_w(u8 counterIndex, u8 mode, CLOCKHZ hz){
u8 df = 0;
CLOCKHZ shz;
/*
H=F/X
X=F/H

F/H=0xFFFF
F/0xFFFF=MINH
F/1=

F/0xFFFF
F/1
*/
if(hz == 0){
hz = 1;
df = 1;
}
reqHz = hz;
shz = FREQUENCY/hz;

if(shz > 0xFFFF){
shz = 0xFFFF;
}
/// convert the actual clock speed in Hz back incase we were shz > 0xFFFF.
curHz = FREQUENCY/shz;

if(df != 1){
outb(0x30 | (counterIndex << 6) | (mode << 1), CONTROL);
outb(shz, BASE+counterIndex);
outb(shz>>8, BASE+counterIndex);
}else{
outb(0x30 | (counterIndex << 6) | (mode << 1), CONTROL);
}
return;
}

/// The interrupt handler for clock ticks, which in return calls the clock multiplexer.
i32 Interrupt(PPRIVATE priv, IRQVECTOR vector){
/// In any clock driver design this should be the last function call and no other
/// important function should reside afterwards except the normal handler function returns to unwind out of this
/// driver. This is dependant on the clock being used by the scheduler subsystem which will never allow a return
/// to this point after clock_tick, until this exact threads gets more time added to it's time slice.
clock_tick((u32)priv->instance);
return IO_SUCCESS;
}
///////////////////////////////////////////////////////////////////////////////////////////////////
//////////////////////// ENTRY ////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////////////////////////////////////////
DRIVER mobo_clock_8253;
i32 clock_8253_init(){
u32 x;
PDEVINSTANCE inst;
/// Make sure the device actually exists.
if(inb(COUNTER0) != 0xFF){
console_write("[8253PIT] driver initialized and device found. \x10");
inst = (PDEVINSTANCE)dev_CreateInstance(&mobo_clock_8253);
if(dev_res_alloc(inst->priv, DEV_RES_IO, 0x40, 0x4) != IO_SUCCESS){
/// The resource allocation failed. Exit.
console_write("[8253PIT] Resource allocation failed. \x10");
return IO_NORESOURCES;
}
/// There is no easy way to lookup the used interface from the PRIVATE structure so lets
/// hand the clock multiplexer a direct link to it with the PRIVATE structure.
/// The multiplexer will enable, disable, and change the clock hertz as needed.
inst->priv.instance = (void*)clock_register(&inst->priv, &ifaceClockClock);
/// Register our interrupt handler, which in return calls the clock multiplexer tick function.
irqrouter_RegisterInterrupt(&inst->priv, &Interrupt, IRQ+IRQROUTER_HWBASEVECTOR, IRQROUTER_DIRECTCALL);
/// Exit.
return IO_SUCCESS;
}
/// This driver was unable to detect the hardware's presence.
console_write("[8253PIT] device not detected. \x10");
return IO_DEVICENOTFOUND;
}

//////////////////////////////////////////////////////////////////////////////////////////////////
//////////////////// EXPORTS /////////////////////////////////////////////////////////////////////
//////////////////////////////////////////////////////////////////////////////////////////////////
static i32 GetInfo(PPRIVATE priv, PCLOCKINFO info){
info->curHz = curHz;
info->reqHz = reqHz;
info->minHz = FREQUENCY/0xFFFF;
info->maxHz = FREQUENCY;
return IO_SUCCESS;
}
static i32 SetHz(PPRIVATE priv, CLOCKHZ hz){
/*
Using counter zero. If other counters can be used I do not know about it.
*/
if(hz == 0){
console_write("[8253PIT] Told to disable. \x10");
c8253_w(0, MODE_INTONTERMINALCOUNT, hz);
}else{
c8253_w(0, MODE_SQUAREWAVE, hz);
}
return IO_SUCCESS;
}

/////////////////////////////////////////////////////////////////////////////////////////////////
///////////////// HEADER ////////////////////////////////////////////////////////////////////////
/////////////////////////////////////////////////////////////////////////////////////////////////
static ICLOCK_CLOCK ifaceClockClock = {
.GetInfo=&GetInfo,
.SetHz=&SetHz
};

static DEV_CONNECTION_MOTHERBOARD_DATA connectionDataMotherboard = {
DEV_MOTHERBOARD_CLOCK,
&clock_8253_init
};

DRIVER mobo_clock_8253 = {
.szName = "8253PIT",
.connection = DEV_CONNECTION_MOTHERBOARD,
.connectionVersion = DEV_CONNECTION_MOTHERBOARD_VERSION,
.connectionData = &connectionDataMotherboard,
.priv = 0,
.ifaceCount = 1,
.iface = {
{
.szName = "CLOCK.CLOCK",
.major = DEV_CLOCK,
.minor = DEV_CLOCK_CLOCK,
.iface = &ifaceClockClock
}
}
};

Alboin wrote: I think that there are ways around this. What about having a dependency system, which glues together the modules? So that if module A needed module B, it could ask the dependency system to load module B and any other dependencies that it may carry. With each team knowing the interface of the other teams, they could all start developing at once and assume that by the time they finish, the other team will have their dependencies finished. However, this would mean detailed planning by each group before the coding actually started.

You aren't thinking of OS kernels, are you? We're talking about systems that need to be combined together by the linker, so that we can have a system up and running and initialized to the point that we can actually consider supporting something you could call "loading."

Yes, you can still use dynamic interfaces, and what not. But the key issue is that it hardly makes any sense, because the problem is NOT linking the modules together. The problem is defining the interfaces between the modules before you've written any of the modules.

In userspace, you can write modules, write test cases for them, and dummies that provide hard-coded responses. You can then possibly hope to debug some of the stuff, until you hit the "integration" phase, which is going to fail, and your product is going to be crap.

The problem is even worse when you move into kernelspace. Now you can't hope to catch even the internal bugs, because you simply can't have the module running in environment where they'd matter, before you have the other modules. For this reason I believe that a kernel needs to be grown somewhat organically, and you can't do organic growth as a community project over the internet, before you have some sort of codebase to build on.

The problem with common drivers is that since most everyone here is working on a hobby OS and many for learning purposes, I don't think they'd want to use someone else's code. At least I wouldn't.

In the past there (at least in Mega-Tokyo) has been interest into common drivers.

Kevin McGuire wrote: I wrote my 8259 Programmable Interrupt Controller as a loadable module which is abstracted using a IRQ routing system. I also did the same with the 8253 Programmable Interrupt Timer which is abstracted by clock which supports multiple clocks, and it decides which clock to use - but never the less my point is it is abstracted completely so that even the scheduler and drivers think they are dealing with just one clock at all times no matter what the implementation really is behind the scenes. And just to beep on my horn a little the clock uses the IRQ router abstraction. The only problem is having a birds eye view of the big picture.

Ok fine. My current kernel doesn't (yet) bother abstracting PIC/PIT at all, but now that I think of it, yeah, I could do it as well at some point.

If you would like to see my code to know what I am talking about I would not mind, and yes I currently have it working and drivers are using both abstractions.

Would you then not mind either dropping it inside