Use of Job Queues?

[eryjus]

Hi,

I was thinking of borrowing a concept from the AS/400 (or IBM i, if you prefer) OS for work management whereby certain background processes entering the system would be managed through a queue of work to be completed. This queue of work would not be executing, but would wait in a queue to start when the processor would become "available" (in quotes since this could be defined).

One such definition *might* be only 3 of these jobs can run at a time coming through the queue called "normal background". Another definition might be that in addition to the 3 just mentioned, a "long job" queue might exist where longer running jobs get submitted and 3 additional jobs may run (subject to playing nice, of course).

Of course, not every process could qualify for this kind of treatment. However, a code management system with a number of nightly builds might be able to take advantage of this at the OS level rather than at an application/thread level.

Anyway, I was wondering about any thoughts on this concept. Is the thinking sound? Or am I being a little too grandiose for a hobby OS?


Thanks!
Adam

[Brendan]

Hi,

Is the thinking sound? Or am I being a little too grandiose for a hobby OS?

The idea of "5 completed jobs and 5 unstarted jobs is better than 10 half-finished jobs" is sound; and I don't think it's too grandiose at all.

I would suggest it needs a little refinement though. For example, rather than having a limit of 3 jobs running at a time, you could just say that a job's priority depends on when it was started (such that the job that was started soonest takes priority over jobs that were started later; but if the "soonest" jobs all block waiting for IO or something, then the latest job run until something unblocks).

Also, (in general) things like this should be extended to IO priorities too (VFS, file systems, disk drivers, networking, etc) - e.g. ensure that IO for the soonest job/s takes priority over any IO for later jobs.


Cheers,

Brendan

[turdus]

It absolutely make sense, and I'm using it in a similar fashion (that's because my design was also inspired by mainframes). What I do, is creating groups of 5 jobqueues. You have one jobqueue group for device drivers, one for services, one for normal apps etc. Within these, you'll have priority -2..2.
DRV-2, DRV-1, DRV0, DRV+1, DRV+2, APP-2, APP-1, APP0, APP+1 etc.
Driver queues has the most priority, services follows and applications. Long time queues and idle queue ends the list.
What unusual in my setup is having a placeholder entry in every queue, which refers to a lower priority queue (so jobs in lower priority queue at all will have the same amount of cpu time as every job in current queue). This guarantees priority among queues within groups (like your nice queue idea). Within a jobqueue, I use simple round-rubin.
Example: A and B are jobs in APP-1. C and D are also jobs but in APP0 (APP-1 means less nice than APP0). Jobs will be executed in order: ABCABDABCABD...

[eryjus]

Jobs will be executed in order: ABCABDABCABD...

This sounds more like a scheduling algorithm once the jobs start to process. What I am referring to is throttling the number of jobs that can execute concurrently, keeping the extra ones from starting at all until the processor(s) have "availability". Did I miss something in what you were saying?

[Owen]

Doesn't sound too different from Grand Central Dispatch

[eryjus]

Doesn't sound too different from Grand Central Dispatch

Agreed. Thanks for the link!