flat memory model

[clavin123]

in flat memory model for each register (stack segement, data segement etc) we provide them whole address space(i.e. they overlap)
so how does the kernel distinguish and not overwrite memory?

[Solar]

One, flat memory model does not necessarily mean that all segments provide the full address space. There are several "tricks" that can be done, like limiting code / data and stack so that they both provide zero-based addressing, but do not overlap. But that's just an aside.

Generally speaking, the kernel can do anything. In kernel space, only careful programming can keep your system from crashing. Even if you'd set up segments not to overlap, the kernel could change that any time, couldn't it? Why does it not happen? Because you simply don't do it, that's why.

The same for flat memory. You assign space for the stack, you assign space for memory management structures etc., and you take care that you use the memory regions accordingly. Yes, a wild pointer can poison your data structures, where it might have triggered a segmentation violation in segmented memory.

But what are your options, in any case? It doesn't really matter if the wild pointer triggered an exception, or just threw a fragmentation grenate in your IPC lists: Your system has a fatal bug, and the best you could do is a kernel panic anyway...

[Love4Boobies]

I think the OP may be asking about paging.

P.S.: The wiki tag links don't show as normal links in the forum's default theme.

[clavin123]

@solar
well ok i get your point that i need to handle that myself
but the JAMESM tutorial that i am following gives a full address space to ss, ds etc
and does not provide any limit
but still things work correctly. so am i missing something here ??

link
http://www.jamesmolloy.co.uk/tutorial_h ... 20IDT.html

[Solar]

but still things work correctly. so am i missing something here ??

Not sure where I was unclear.

There is no problem whatsoever with setting all segments to cover the whole address space. The kernel has to tread carefully in any case, and later on you can activate paging to have user processes protected from each other.

A somewhat more sophisticated setup (which tutorials never are) might want to discriminate, like keeping the stack segment out of the code/data segment.

I recommend studying the Intel Manuals. Volume I for an overview, volume III for the details.

[rdos]

A somewhat more sophisticated setup (which tutorials never are) might want to discriminate, like keeping the stack segment out of the code/data segment.

Not following you here. A valid flat memory model must have the same base for all selectors. You cannot set a different base for the stack, because C constructs both stack-relative and data-relative pointers, and both are assumed to be offsets only in a flat memory model. The stack can only have a differet base if the memory model is compact, meaning all data pointers are 48 bit (selector + offset), and all code pointers are near (offset only).

What can be done is to use FS/GS for some private memory area, like the TLS in the PE format. That's probably why the ability to set a non-zero base for FS/GS was retained in 64-bit mode.

[Combuster]

I suspect he meant disjoining the code and data/stack segments.

[clavin123]

ok so will be right in assuming the following-
that when we set up a flat memory model the stack, data etc are not invulnerable to overwrite
it must come in the kernel desgin.
also it depends on the language/compiler used of how they manage the memory.

[linguofreak]

A somewhat more sophisticated setup (which tutorials never are) might want to discriminate, like keeping the stack segment out of the code/data segment.

Not following you here. A valid flat memory model must have the same base for all selectors. You cannot set a different base for the stack, because C constructs both stack-relative and data-relative pointers, and both are assumed to be offsets only in a flat memory model. The stack can only have a differet base if the memory model is compact, meaning all data pointers are 48 bit (selector + offset), and all code pointers are near (offset only).

You can still have the same base but different limits and expand-up/down attributes.

[iansjack]

Talking from a 64-bit point of view, where a flat memory model is always used, this is a matter for paging, not segmenation. I give the kernel memory addresses for code, data, and stack that are so far from the user program ones that the chances of overlap are infinitesimal. All these pages are marked as acessible by supervisor mode only.

User tasks get at least four pages each - one for code, one for data, one for the stack, and one for a stack for exceptions. Again these are far apart, in discrete parts of the virtual address space. And they are the same virtual addresses for each task; that way a task cannot access another task's memory other than via the kernel. As the address spaces are discrete it is almost certain that an errant program will get a page exception rather than accessing memory that it shouldn't.

Once you start working with 64-bit you quickly become aware of just how mind-bogglingly large the virtual address space is. You may run out of physical memory but you can be free and easy with the virtual address space; you're never going to run out of addresses. As long as there are unmapped virtual addresses between your various "segments", a page exception is infinitely more likely than an incorrect address. The one exception is a null pointer; guard the first page of virtual addresses as closely as you can.

[rdos]

Talking from a 64-bit point of view, where a flat memory model is always used, this is a matter for paging, not segmenation. I give the kernel memory addresses for code, data, and stack that are so far from the user program ones that the chances of overlap are infinitesimal. All these pages are marked as acessible by supervisor mode only.

User tasks get at least four pages each - one for code, one for data, one for the stack, and one for a stack for exceptions. Again these are far apart, in discrete parts of the virtual address space. And they are the same virtual addresses for each task; that way a task cannot access another task's memory other than via the kernel. As the address spaces are discrete it is almost certain that an errant program will get a page exception rather than accessing memory that it shouldn't.

The first valid argument I've seen about a flat (64-bit) memory model having an adequate protection for faulty addresses.

Once you start working with 64-bit you quickly become aware of just how mind-bogglingly large the virtual address space is. You may run out of physical memory but you can be free and easy with the virtual address space; you're never going to run out of addresses. As long as there are unmapped virtual addresses between your various "segments", a page exception is infinitely more likely than an incorrect address. The one exception is a null pointer; guard the first page of virtual addresses as closely as you can.

This is only valid if your malloc allocates a random full page in the virtual address-space, even if the request is for 1 byte. Otherwise you are stuck with possible overwrites in your application when it exceeds the limits as usual. Segmentation is superior in this respect as it returns a selector with the exact limit, and doesn't need to allocate a full page when one byte is requested.

[iansjack]

Protection of memory at the malloc() level is certainly a problem, but I can't see that segmentation is much help here. You're hardly going to create a new segment every time you malloc() 1 byte, are you?

It's a two-fold problem. One is preventing buffer overruns, but the more serious problem is protecting the data structures that keep track of malloc() allocations. In my OS I use linked lists with the link being immediately in front of the allocated memory. But any buffer overrun here will corrupt the whole data chain. (At least it is only the data chain for the one process. You might even argue that once a buffer overrun has happened the most favourable outcome is for the process to immediately crash.)

I'm not sure of the answer. As long as the user program can access the memory used to store the links there is the potential for corruption. And in most conventional OSs it is the user runtime that manages malloc(), so the user program does have that access.

I actually make memory allocation a system call (although my motivation was the easier management of reclaiming memory from dead processes). Now this may seem horrendously inefficient but if you use the SYSCALL/SYSRET mechanism for system calls then it is really hardly any less efficient that subroutine calls. (I haven't looked at the timings - it may even be more efficient). And I feel happier that the kernel is managing memory - surely that's the job of the kernel?

I'm having thoughts of separating the data structures from the allocated memory, so that there would be linked lists in kernel data memory pointing to the allocated memory in user data memory, but I haven't finalized the details in my mind yet. That would solve the more serious problem, to my mind, by protecting the integrity of the memory allocation data structures.

As for protecting the actual memory itself - preventing one buffer from overwriting another - I don't know the answer to that. But I can't see a separate segment for every memory allocation as the answer. Surely that would be too inefficient, both in terms of speed and resources, in a real OS? And the same would go for use of the BOUND instruction for every heap memory access.

I'm certainly open to any bright ideas.

[Owen]

I actually make memory allocation a system call (although my motivation was the easier management of reclaiming memory from dead processes). Now this may seem horrendously inefficient but if you use the SYSCALL/SYSRET mechanism for system calls then it is really hardly any less efficient that subroutine calls. (I haven't looked at the timings - it may even be more efficient). And I feel happier that the kernel is managing memory - surely that's the job of the kernel?

An order of magnitude slower. Interrupts and any control transfers involving segmentation are an order of magnitude slower still.

"call" has a minimum latency of 1 cycle on most processors, often reaches that in practice, and because modern CPUs are superscalar can have a cost of 1/4th of a cycle or less. Same for ret.

The best case I've seen for syscall is about 20 cycles, similar for sysret, and this is a true cost best case, because syscall always hits microcode and can't be executed in parallel with any other instruction. Additionally, the branch predictor doesn't predict over a syscall, so it's very possible that the destination TLB entry is not present and that the instructions are not in cache. And, of course, the kernel can't trust any addresses it has been given by userspace, so there is additional overhead to deal with there.

INT 0xNN and CALL FAR via call gate both tend to be ~100 cycles, with IRET being similar. Segment loads and privilege checks are expensive operations.

Managing system memory is the kernel's job, yes, but managing a process' memory space is its own job. You can't assume one allocator is good for all cases - for example, do you think the Java Virtual Machine uses malloc/realloc/free to manage the VM heap? (The answer is no, by the way, though I expect you guessed this).

[iansjack]

Interesting information about the timing of the SYSCALL/SYSRET instructions, which provides food for thought. But I have to disagree with the TLB comment - surely the entries for the routine handling system calls will always be present as it is going to be a routine that is called fairly frequently. I would expect that, after the first SYSCALL instruction that particular page translation entry would never leave the cache. It is certainly true, as you say, that a system call via these instructions is considerably more efficient than one via segmentation. I do like the way that you can effectively ignore the segment registers in 64-bit long mode.

How important those 20 cycles are compared to the cost of the rest of the memory allocation routine is another consideration.

As I say, I'm still thinking about memory management so what you say is most useful. I think there is an elegance about the idea that the kernel handles all resource management but I see your point that it might not be the most practical or efficient idea. Rather the same arguments apply in the micro- vs. monolithic-kernel debate. Elegance and efficiency don't always go hand in hand.

To come back to the point I was originally addressing - pages or segments for memory protection - I realise now that I overlooked the fact that segments in x86 64-bit long mode don't provide limit checking. So the argument about using segments for this purpose falls flat (excuse the pun) in the 64-bit model.

[Rudster816]

You don't always have to trap\syscall into kernel land in order to be able to do all the things that need to get done. So long as you have the proper mechanism in place, you could easily put most memory allocation functions inside of the kernel, but put ones like malloc() outside of kernel space. It's still apart of the kernel in the sense that it will always "be there" for user code to call, it just doesn't run in kernel mode. Only if it NEEDS to do something inside kernel mode should it go through the rather long procedure of doing that. There are many WinAPI functions that will never call kernel level code, while others (like FileOpen) are basically 1:1 mirrors of kernel level functions.