OSDev.org

I wanted the have a memory structure where the kernel CS, DS start from 0 and have full 4 GB. However the Kernel Stack in a separate 128 KB segment.
Thus when addressing using the SS register, the processor will add the 0x8000 to the memory location. However the same location with the DS register will add 0x0000.

The problem is that GCC is not taking into account the fact that DS =/= SS. For example, code like one shown below (and as generated by GCC) will work wrongly as the segment register is completely wrong.
Is GCC API does not allow SS to be on a different segment? I think System V API does allow that.

Please suggest.

Note: Virtual Memory is not implemened yet.

arjob wrote:I wanted the have a memory structure where the kernel CS, DS start from 0 and have full 4 GB. However the Kernel Stack in a separate 128 KB segment.

Why? Just put the stack in the same segment as everything else! This would avoid such problems.

arjob wrote:Is GCC API does not allow SS to be on a different segment? I think System V API does allow that.

No it does not. If I wasn't so lazy, I would actually find a reference for you, but all ELF operating systems expect all segment bases except those for FS and GS to be zero. And except for OpenBSD, no OS ever has anything other than a 4GB limit in these segments, either. And OpenBSD only changes the CS limit on processors that don't know the NX bit yet.

I fail to understand what it is you are even trying to do.

arjob wrote:Note: Virtual Memory is not implemened yet.

Good, because segmentation takes place before memory translation.

The idea is nice, but unfortunately not supported by primitive compilers like GCC. The primary reason you want the kernel stack in its own segment is to keep it isolated from other stacks and data. It also makes it possible to catch kernel stack overflow without wasting virtual memory space for several guard pages which is not at all as efficient either. Without stack limits in kernel, interrupt loops will lead to nasty problems that cannot be pinpointed.

The claim that no OS uses this is wrong. Every thread in my OS has a 4k selector loaded in SS with a non-zero base and a 4k limit when running in kernel mode. This means that stack problems in kernel will be caught where they happen and not as a consequence of secondary corruption of memory.

OpenWatcom supports having SS != DS, as well as writing code where CS has a non-zero base and non 4G limit. It's not really hard as it only requires the use of proper segment register overrides and converting them to 48-bit pointers when used in a different context.

Thanks for replying,

rdos wrote:The idea is nice, but unfortunately not supported by primitive compilers like GCC. The primary reason you want the kernel stack in its own segment is to keep it isolated from other stacks and data.

Yes that was the idea, however it seems that even with segmentation, we can check for Stack overflow (Expand down segment), or Stack underflow (Expand Up segment), but not both. So it is kind of useless isn't it?

OpenWatcom supports having SS != DS, as well as writing code where CS has a non-zero base and non 4G limit.

I had used OpenWatcom, but I am little reluctant to use a non standard compiler. What compiler have you used in your rdos?

arjob wrote:Thanks for replying,

rdos wrote:The idea is nice, but unfortunately not supported by primitive compilers like GCC. The primary reason you want the kernel stack in its own segment is to keep it isolated from other stacks and data.

Yes that was the idea, however it seems that even with segmentation, we can check for Stack overflow (Expand down segment), or Stack underflow (Expand Up segment), but not both. So it is kind of useless isn't it?

Can’t you do this stack overflow detection quite easily with paging? And probably an “underflow” too, though if you have an underflow I think something has gone very badly wrong, and expanding the stack isn’t going to help you.

arjob wrote:Thanks for replying,

rdos wrote:The idea is nice, but unfortunately not supported by primitive compilers like GCC. The primary reason you want the kernel stack in its own segment is to keep it isolated from other stacks and data.

Yes that was the idea, however it seems that even with segmentation, we can check for Stack overflow (Expand down segment), or Stack underflow (Expand Up segment), but not both. So it is kind of useless isn't it?

I think the kernel stack size should be static, and so if something uses the SS register faults, it should not be handled by trying to expand the stack rather handle it as a fatal error. This way, both underflows and overflows will fault.

arjob wrote:

OpenWatcom supports having SS != DS, as well as writing code where CS has a non-zero base and non 4G limit.

I had used OpenWatcom, but I am little reluctant to use a non standard compiler. What compiler have you used in your rdos?

Yes, currently everything in RDOS uses OpenWatcom (except for EFI loaders). I've used Borland C++ and Win32 emulations in the past, but right now RDOS support is integrated into the library of OpenWatcom and so I can create native applications with no dependencies.

rdos wrote:Yes, currently everything in RDOS uses OpenWatcom

It is really a shame that the most popular compilers do not support the segmented memory model. It really limits what a generation of developers can think and do. Having a separate stack segment is really a natural thought, but I am forced to implement virtual memory for the same protection.

arjob wrote:

rdos wrote:Yes, currently everything in RDOS uses OpenWatcom

It is really a shame that the most popular compilers do not support the segmented memory model. It really limits what a generation of developers can think and do. Having a separate stack segment is really a natural thought, but I am forced to implement virtual memory for the same protection.

There is also the issue of getting adaptations to compilers & libraries integrated into the mainline code of the toolchain you use. If you cannot get those accepted because they only allow these for major operating systems like Windows and Linux, then you will have to rebase the patches all the time to keep it working as other changes are done to the project. I never managed to get patches for RDOS accepted for GCC, even if there are some RDOS patches for the top-line projects (binutils). With the OpenWatcom project, it was much easier.

It seems your question has been answered a year ago, albeit in the context of 8086.

arjob wrote:It really limits what a generation of developers can think and do.

That surely has to be an exaggeration. Is segmented memory model, compared to the flatmem model, such an incredible game-changer?

arjob wrote:Having a separate stack segment is really a natural thought, but I am forced to implement virtual memory for the same protection.

Having a separate stack segment means juggling multiple ways of dealing with the same address locations. The compiler must track how a single address location is accessed, and emit instructions to adjust the offsets if the accessor-segment changes, or ensure that it doesn't change at all.

The data stored on the stack of one thread/CPU can be accessed by another thread/CPU as part of any concurrent programming paradigm. Here, a single memory location is forced to be accessed through different segments.

If the compiler chooses to or is forced to adjust the offsets, it must know how the descriptor tables are configured inside the kernel! Compiling and running of any software, now needs access to the info stored inside the descriptor tables. All usermode apps now need that info; if someone managed to corrupt the copy of that info stored in the user-space, the accesses can over/under-shoot the actual stack, thus bringing about the very situation this setup is supposed to prevent.

Edit: The info could be placed in vDSO as read-only; but the instructions to read it and adjust the offsets will be found to be sprinkled quite generously within any compiled code, resulting in noise.

Moreover, the disassembler engines, the debuggers and similar tools that work with binaries will themselves have to keep track of such memory locations if they are to be useful.

linuxyne wrote: That surely has to be an exaggeration. Is segmented memory model, compared to the flatmem model, such an incredible game-changer?

arjob wrote:Having a separate stack segment is really a natural thought, but I am forced to implement virtual memory for the same protection.

Having a separate stack segment means juggling multiple ways of dealing with the same address locations. The compiler must track how a single address location is accessed, and emit instructions to adjust the offsets if the accessor-segment changes, or ensure that it doesn't change at all.

The data stored on the stack of one thread/CPU can be accessed by another thread/CPU as part of any concurrent programming paradigm. Here, a single memory location is forced to be accessed through different segments.

If the compiler chooses to or is forced to adjust the offsets, it must know how the descriptor tables are configured inside the kernel! Compiling and running of any software, now needs access to the info stored inside the descriptor tables. All usermode apps now need that info; if someone managed to corrupt the copy of that info stored in the user-space, the accesses can over/under-shoot the actual stack, thus bringing about the very situation this setup is supposed to prevent.

You cannot do it only halfway. OTOH, you don't have to do it the way it was typically done in the 80s and 90s when large 16-bit memory models were used. These create a large amount of segment register loads that hurt performance. Instead, the compact 32-bit memory model is ideal and would call local procedures with near calls (just like any flat model). DS would be loaded with the static data selector, and so this area could also be referenced with only offsets. Inter-driver calls would be far though, and data sharing also needs to use 48-bit pointers.

I don't think that it is sound to share stack contents between threads. Why would anybody want to do this? Sharing data between threads should use either static memory or memory allocated from the heap.

A segmented OS needs to keep references to flat memory at a minimum, otherwise, the advantages will be sabotaged. At a minimum, file & disc caches will need to use flat memory as there are too few selectors to map these to selectors.

[My edit crossed with your reply].

rdos wrote: The compact 32-bit memory model is ideal and would call local procedures with near calls (just like any flat model). DS would be loaded with the static data selector, and so this area could also be referenced with only offsets. Inter-driver calls would be far though, and data sharing also needs to use 48-bit pointers.

If a pointer, to a variable that is stored on the stack, is passed to a function, how would dereferencing that pointer be accomplished inside the function?

rdos wrote: I don't think that it is sound to share stack contents between threads. Why would anybody want to do this? Sharing data between threads should use either static memory or memory allocated from the heap.

I do not think that any machine prevents us from sharing data in such a manner.

The example was given as sample of a situation where the compiler is /forced/ to adjust the offsets. One can fit in any other situation where a compiler is forced to access stack data through data segment.

---

The machines already allow running with separate ds and ss segments. It is only the ABI that's forcing them to have the same range. Since the OP feels so strongly about how much the ABI is limiting the developers from thinking and doing, the OP should submit a change proposal to them.

linuxyne wrote: The example was given as sample of a situation where the compiler is /forced/ to adjust the offsets. One can fit in any other situation where a compiler is forced to access stack data through data segment.

The compiler shouldn't try to adjust offsets, and when something is passed as a pointer then the pointer either needs to be 48-bit, or defined as based. OpenWatcom supports based pointers, but it's a bit of a hassle to handle. As a default, the compiler needs to convert a stack reference or a static data reference to 48-bit when it is assigned to a pointer.

How interesting that I found this thread just now. I've revived my old 80386DX just last week and wanted to experiment with segments in 32-bit extended DOS applications.

I did find the options for Watcom C/C++ to create a segmented executable, but as far as I can see the compiler still generates flat code. Maybe I need to use multiple OBJ-modules and/or exceed a certain minimum size of my data and code segments, I don't know at this moment.
I've also did some preliminary assembly tests but couldn't get anything to work. For example a simple "Hello World" program defined with full segment definitions and no DOS groups *STILL* generates a flat model with SS=DS=ES. The reality is that the link between generated assembly code -> object code -> executable binary is not clear enough at this moment.

Now... so far I've found that the problem with all extenders is (I believe...) the loader. Instead of creating default segments, they just fixup the segment offsets in the executable file to fit within a flat memory model (as per the specifications of most object code formats). So my guess is that somehow Watcom C/C++/Fortran must emit extra code to create new segments (and maybe change segment limits), but I haven't found it yet.

I you know more about this topic, or how to let the compiler emit code with 48-bit pointers: I'd be extremely interested to learn more!

Off topic stuff below:

I totally get the idea of using segments (again) in experimental code. Afterall, if you'd like to -say- code an old school demo and need your CPU in 32-bit protected mode but with paging disabled, then the only real protection would have to come from segment limits. I'm not going to elaborate on my fetish for segmented memory, that's for another time

rdos wrote:The idea is nice, but unfortunately not supported by primitive compilers like GCC. The primary reason you want the kernel stack in its own segment is to keep it isolated from other stacks and data. It also makes it possible to catch kernel stack overflow without wasting virtual memory space for several guard pages which is not at all as efficient either. Without stack limits in kernel, interrupt loops will lead to nasty problems that cannot be pinpointed.

The claim that no OS uses this is wrong. Every thread in my OS has a 4k selector loaded in SS with a non-zero base and a 4k limit when running in kernel mode. This means that stack problems in kernel will be caught where they happen and not as a consequence of secondary corruption of memory.

OpenWatcom supports having SS != DS, as well as writing code where CS has a non-zero base and non 4G limit. It's not really hard as it only requires the use of proper segment register overrides and converting them to 48-bit pointers when used in a different context.

arjob wrote:Thanks for replying,
Yes that was the idea, however it seems that even with segmentation, we can check for Stack overflow (Expand down segment), or Stack underflow (Expand Up segment), but not both. So it is kind of useless isn't it?

No, that's not correct. Stack over- or underflow is enforced by segment limits. I.e. a stack segment of 400h bytes will generate an exception if any reference is made outside :

[*] SS:0 and SS:3FFh for an expand-up segment (initial ESP=3ffh)
[*] SS:400h and SS:0h for an expand-down segment (initial ESP=0)

Maybe you were thinking about the expand-down trick which makes byte 0 unaccessible in expand-down segments (or the first 4096 bytes for 32-bit segments), and is sometimes used for null pointer checking.

16bitPM wrote:

arjob wrote:Thanks for replying,
Yes that was the idea, however it seems that even with segmentation, we can check for Stack overflow (Expand down segment), or Stack underflow (Expand Up segment), but not both. So it is kind of useless isn't it?

No, that's not correct. Stack over- or underflow is enforced by segment limits. I.e. a stack segment of 400h bytes will generate an exception if any reference is made outside :

[*] SS:0 and SS:3FFh for an expand-up segment (initial ESP=3ffh)
[*] SS:1 and SS:400h for an expand-down segment (initial ESP=0)

Maybe you were thinking about the expand-down trick which makes byte 0 unaccessible in expand-down segments (or the first 4096 bytes for 32-bit segments), and is sometimes used for null pointer checking.

Your examples are wrong.

intel wrote:For expand-up segments, the offset in a logical address can range from 0 to the segment limit. Offsets greater than the segment limit generate general-protection exceptions (#GP, for all segment other than SS) or stack-fault exceptions (#SS for the SS segment).

For expand-down segments, the segment limit has the reverse function; the offset can range from the segment limit plus 1 to FFFFFFFFH or FFFFH, depending on the setting of the B flag. Offsets less than or equal to the segment limit generate general-protection exceptions or stack fault exceptions. Decreasing the value in the segment limit field for an expand-down segment allocates new memory at the bottom of the segment's address space, rather than at the top. IA-32 architecture stacks always grow downwards, making this mechanism convenient for expandable stacks.