Memory management

[ASHLEY4]

(From Tim Robinsons, Memory management 1)
Without using paging, the x86 can only address 16MB of physical memory. This comes back to the 286: as a 16-bit processor running on a 24-bit bus, it could only address 16MB. Without segmentation, you were restricted to 64KB.

Can someone explain the above please.
As i understand it ,you can access up to 4gb on a 386 and above, of physical memory, with out paging (from pmode with a20 ,gdt,etc enabled).
Or have i got it wrong?.
This is important as i am, doing my memory management with out paging.

Thanks for any help.

ASHLEY4.

[Candy]

You're right.

At least, It worked for me

[bubach]

This is important as i am, doing my memory management with out paging.

how come u r not using paging?

[Solar]

Why not? I can understand that decision, to some extend. Paging adds complexity to your kernel, and eats some percentage of your CPU performance. AmigaOS lived very well without any MMU whatsoever on systems having 1 promille of the memory today's systems enjoy.

That being said, while I will be using virtual memory for sure, the paging-to-disk will be purely optional in my kernel, too.

[bubach]

the reason i asked was becasue i have not started on memory yet, i was woundering what you guys with i should use?
i have no idea... paging is used so that you can flip stuff from ram to hd (swap file) and the other way around, or?
if i don?t use paging, how will it work then?

/ Christoffer

[Ozguxxx]

Without paging, you wont have a virtual address space, the addresses you give to program are directly mapped onto themselves in memory so the actual size of memory addressable without paging is size of your ram. (since you wont have a page translation)
With paging you have virtual memory which means more protection and according to me easier management of user address spaces.
Back to your question: If you dont have paging enabled, according to me you still need a memory manager but only a physical memory manager one will do everything for you. Also swapping is a very far consideration for people i think, since for a need of swapping you will need to eat up all the physical memory in your ram, which is a very hard thing to do I think.

[Solar]

Quick rundown. (Should this go to the FAQ?)

Physical Addresses

...is when every address "visible" to a program directly relates to an address on the address bus between CPU and RAM. If I access 0x0023 5fc4, I mean the address returned by the RAM when I apply the bit pattern 0x0023 5fc4 to the address bus.

In this memory model, every executable or library must either use PIC (position-independent code), or come with relocation tables so jump and branch targets can be adjusted by the loader.

The AmigaOS used this memory model, in absence of a MMU in early 680x0 CPUs. It is also the most efficient, but it does not allow for protecting processes from each other.

Virtual Addresses

The advent of MMUs (Memory Management Units) allowed to play tricks to the addressing. A virtual address could be mapped to any physical address. It was possible to provide each executable / library with address spaces, so each executable e.g. has its memory starting at 0x0000 0000. Since the kernel is in control of the virtual-to-physical mapping, processes cannot access each other's memory unless allowed to do so by the kernel.

Paging

Having an individual virtual-to-physical mapping for each address is of course ineffective. The traditional approach to virtual memory is to split up the available physical memory into chunks (pages), and to map virtual to physical addresses page-wise. This task is largely handled by the MMU, so the performance impact is low, and generally accepted as a low price to pay for memory protection.

Virtual Memory

The next step is, instead of reporting an "out of memory" once the physical memory runs out, is to take pages that are not actually accessed currently, and write them to hard disk (swapfile or -partition) - freeing up the physical memory page. This is referred to "paging out" memory.

This requires additional bookkeeping and scheduling, and introduces severe performance hits when a process accesses a page that's currently paged out and must be paged in again from hard drive. There is also some trickery involved, which - when not handled correctly - can severely impact your OS performance.

On the other hand, your "virtual address space" grows to whatever your CPU and hard drive can handle. In concept, CPU caches and RAM are just cache layers on top of your hard drive.

Summary

Virtual addresses and paging are A Good Thing (tm), even a requirement for every decent OS not limited to embedded systems (and sometimes even there). Virtual memory certainly is a nice-to-have (and every modern desktop OS has it), but it comes at a cost and requires much additional design and know-how to get right. AmigaOS lived fine without it, in its time, but that was before multi-megabyte applications like Word or Photoshop appeared. Or perhaps they appeared because Virtual Memory is so cheap...?

[bubach]

(Should this go to the FAQ?)

It sure should..

Thanks for all the info. What do u think? Should i skip paging for now, and go for a (easier?) "no paging" mm?
I could always implement paging later on, or would it be a pain...?

/ Christoffer

[Solar]

Unless you are aiming for a very special-purpose embedded OS, I would strongly advise to use virtual addressing (i.e., paging). A desktop OS without memory protection is seriously crippled, and virtual addressing is the key to that feature.

Moreover, having a physically adressing OS results in a significantly different design - for the messaging system, the loader, the dynamic linker... it would require a rewrite of major parts of the kernel to retrofit paging afterwards.

So yes, I suggest you use paging. Whether you support paging out (i.e., virtual memory) or not doesn't really matter. Once you have paging set up, adding the stuff required to use your hard drive as memory extension isn't exactly trivial, but rather easy to retrofit.

[Pype.Clicker]

Multi-megabyte applications haven't wait for MMU to appear ... they were just way more complicated to program (and thus Joe Average never heard of 'm).

This is one of the reason why compilers and assemblers used to do multiple pass over a file, why you had overlays to your Writing Assistant etc.

OS-assisted Swapping allows programs to write the same program regardless of how many physical memory the executing system will have. Sometimes it's a shame, sometimes it's a gift from gods (you wouldn't like to have your Lx kernel compilation or your 'save as JPEG' operation fail with a 'not enough memory', would you ?)

Same for me: this should join the FAQ...

[Solar]

It already is.

[ASHLEY4]

Thanks for the reply's.

I agree with the conclusion that for a moden desktop you should have provisions for virtual memory, but as mine is a specialist os i came to the conclusion that i could do without it, here are some of my reasons :

My os can be best described as a moden day Amiga/Dos so it is single tasking, no memory protection,pmode, ect , it is made to run from a cd (like a live linux) .
I know that this will never replace the desktop, But i belive that there is a need for such a os.
Say you want to make a mp3 player, you could put your mp3 player with my os along with your mp3 files on a cd and could run it on any x86 pc whether it had xp, linux or what have you,also if you had a old p100 you could use this as a dedicated mp3 player, as dos is 16 bit (old), linux is to complecated,and win 9x takes to long to boot up and slows your program down .

The way that i am going to do my memory management is very simple and it may not work :'( .
Basicly user memory is devided into 1mb blocks , this is the smallest allotable size, this is all mapped into a list with details of whether it is allocated ,address,user id , ect .
Programs can be given more than one block of memory , this is indicated by a bit in the list, so that when the program ends, all allocated block with the usre id are deallocated ,The size of 1mb is so that if a block is de- allocated in the middle of a allocaed blocks,that it is of a size thats useable.
There may be a number of problems that i have not thought of :-\ .
The reason for not useing paging is speed , to plot a pixel directly to memory rarther than go to a look up table etc.

Thanks again.
ASHLEY4.

[Solar]

My os can be best described as a moden day Amiga/Dos so it is single tasking...

I have to object... AmigaDOS was multitasking right from the beginning... what you are thinking about is more like MS-DOS, I beg to differ!

The way that i am going to do my memory management is very simple and it may not work :'( .

...he says, and unknowingly speaks about how just about every MMU today works. #8^)

ASHLEY4, I strongly suggest you take a look at the Intel manuals, or whatever tutorial you can get your hands on regarding paged memory support. You are trying to reinvent the wheel!

See for yourself:

Basicly user memory is devided into 1mb blocks ,

...let's call them "pages"...

...this is all mapped into a list with details of whether it is allocated ,address,user id , ect .

...let's call it "page table"...

The size of 1mb is so that if a block is de- allocated in the middle of a allocaed blocks,that it is of a size thats useable.

Look at the FAQ page where I refined what I wrote earlier in this thread, and see how I named memory fragmentation as one of the drawbacks of the (non-paging) AmigaOS memory model...

There may be a number of problems that i have not thought of :-\ .

Like, not using the part of the CPU hardware that was designed to do exactly what you are thinking about - the MMU?

The reason for not useing paging is speed , to plot a pixel directly to memory rarther than go to a look up table etc.

The things that take away the speed are the things you want to design yourself... the MMU is there to help you, and designed to do it well - including dedicated cache optimized for those table lookups, which you want to implement in main memory...

The performance hit of paging is neglectable, compared to virtual memory. Use the MMU, Luke!

[ASHLEY4]

Solar thanks for your quick reply.
I have a confession to make, i have never used AmigaOs , all my impresions of it are based on your well written information on the above OS.
I am a DOS fan at heart, but every time you write somthing about how the Amiga worked, it's as though you are reading my mind of how i want my OS to be .

So my OS is based on a protective mode DOS, the thing that is differant in my memory management ,than paging is like in dos that there is no conversion needed between virtual and physical memory so it has to be quicker or am i wrong?.

ASHLEY4.

[Tim]

You are wrong. x86-based CPUs are very good at doing virtual-to-physical translation, since they're doing it all the time. You don't think they look up the page directory and page table for every memory access, do you?

Instead, the MMU maintains the translation lookaside buffer (TLB), which acts as a cache for the page tables. Each time the MMU wants to know the physical address for a virtual address, it looks at all the locations in the TLB simultaneously, and the right one pops out. Because this process is electronic, and not in software, there is no slowdown.

Anyway, Solar makes the best points about paging/virtual memory: It makes your life easier as a programmer. You can pick addresses for things anywhere within 4GB, and as long as you set the page tables up with enough physical addresses, it will work.