I have a bunch of questions about this, since I don't think you've given us enough information about things, and honestly I am not sure if there isn't some confusion of ideas in all this (and if there is, whether it is on your part or mine isn't clear, either).
To start with, you might want to say more about things like your toolchain (compiler, assembler, etc - from what you've said I take it you're writing the OS in C, and presumably you aren't using GCC since that doesn't play well with real mode), your intended OS design (monolithic, microkernel, hybrid, executive monitor, what have you) and most specifically, how you mean to vector system calls (e.g., using interrupts a la MS-DOS, as simple function calls, through message passing, or something else).
I will say that, if you are planning for user processes to always use
small model (i.e., one segment for code and a second for data and stack, CS =
x and DS = SS =
y), then there is no reason for a system call to be involved at all. Real mode segments don't have selectable size boundaries (unlike segments in either 16-bit p-mode or 32-bit p-mode), meaning that you effectively are giving the entire 64KiB data and stack segment to the process regardless, unless you are intending to overlap segments for different processes for some reason (which isn't entirely unreasonable, though you'd usually do things like that with Medium or Compact mode instead, in effect making the shared data section being the equivalent of a Fortran-style COMMON block). Thus, the in-process memory management can be a local heap using
malloc() and
free() as usual.
Mind you, your diagram describes tiny model, not small, so there's a definite question there, too. While a raw binary such as a .COM file usually would have the
initialized data in the same segment as the code (since it's loaded as a single blob), in Small Model the uninitialized and dynamically allocated data memory - the .BSS area, the stack, and the heap - would be in the DS/SS segment.
However, this is also why programs in the form of .COM files were generally tiny model - for larger data areas, you'd need to explicitly manipulate the segments when addressing the initialized data so that you offset them from CS, which is easy enough in assembly but a bit awkward in C without an explicit extension and/or some shuffling by the compiler. This is part of why Microsoft introduced the .EXE format (rather late in the development of MS-DOS), and also part of why almost every other x86 OS has used more complex
Executable Formats rather than raw binaries (there are a lot of other reasons to do so though, starting with being able to link separately compiled source files into a final executable without having very different formats for
Object Files and executable files).
Further, most real mode compilers let you select a model at compile time.
However, I can only guess at whether I understood you correctly on this. Could you please elaborate?
