And ive been thinking about it for awhile, I still havent stated programming an OS yet, ive decided to some extensive research and design before starting...No, don't do this. Provide a way of calling kernel functions through interrupts.
The reason for this is that the kernel code will want to run in ring 0 (privileged) whereas user code needs to be in ring 3 (unprivileged). If ring 3 code tries to call ring 0 code the CPU will trigger a page fault or general protection fault.
Interrupts are slower than normal function calls but the protection is worth it. If you're worried about the speed decrease, consider putting more code into the kernel. For example, instead of calling the kernel once per graphics command, why not batch commands in a memory block then trigger a kernel interrupt and pass that block? Then the kernel can run through each command in the block before returning.
Ok so I therefore have a few questions...
Just what functions should be in the kernel? (im going for a monolythic kernel design) eg, it would have the schedular functions, and memory functions? but what else? I was thinking of having it so that my GUI was not hardcoded, and you could just swap in and out different GUI's... so I was thinking of having seperate gui modules... (sorta like .dll's) but how much should go in the kernel and how much in the modules? and how should the kernel interact with drivers?
Also about APIs... how would I handle these? like would I include all of the functions with the app in its address space, or have them all in the kernel/ other modules, then somehow access them?