How can i improve memory management?

[mariuszp]

Currently, I have a "frame bitmap" allocator. Each bit represents a 4KB frame, which also allows me to allocate multiple consecutive frames. I also have a disk cache and a file cache:

- The disk cache is per-disk and stores lists of tracks (32KB segments) from disks as big reads/writes are fastest. (Well, technically it's not a list, but rather a tree much like x86 page tables). Each segment is physically consecutive, to help with DMA.
- File cache is per-inode and stores pages (4KB) from files.

When the physical memory allocator cannot find any free frames in the bitmap, it asks the caches to free up some memory. There are some problems tho. Mainly, if the disk cache gets competly emptied, it's usually impossible for the files to be flushed, because the FS driver tries to write a file to disk, there is a disk cache miss, so the disk cache again asks the file cache to free up some memory, but it doesn't end up being 32KB consecutive, so it fails.

This system literally does not work: if the caches fill up RAM, it fails to free up any memory sooner or later.

What alternative, cache-aware PMM could be used instead?

Thanks.

[OSwhatever]

Mixing page sizes like this is usually very difficult. Fragmentation can also destroy your ability to have any 32kB page available.

A few remarks, since you always require 32kB consecutive pages, could reserve a set of them so that you always have some available?

If you don't have consecutive 32kB pages, do you have the possibility to have a fall back so that you can still use storage IO even if it becomes a little bit slower.

[mariuszp]

Mixing page sizes like this is usually very difficult. Fragmentation can also destroy your ability to have any 32kB page available.

A few remarks, since you always require 32kB consecutive pages, could reserve a set of them so that you always have some available?

If you don't have consecutive 32kB pages, do you have the possibility to have a fall back so that you can still use storage IO even if it becomes a little bit slower.

No, I don't have a fallback. Would it be better if the disk cache was in 4KB pages, and simply using a temporary 32KB buffer for the transfers?

Also, the perfomance is terrible: searching a tree to find a page that could be freed appears to be very ,very slow...

[OSwhatever]

Would it be better if the disk cache was in 4KB pages, and simply using a temporary 32KB buffer for the transfers?

You shouldn't paint yourself into a corner regarding the sector/cluster size as this size is not given by the OS but rather the storage device or the file system. There is nothing wrong having the ability to have larger cluster sizes for the swap file and several existing operating systems allow adjustable size.

Your problem is that you really need consecutive pages when you are low on memory. What system is this? Don't you have an IO MMU?

[mariuszp]

Would it be better if the disk cache was in 4KB pages, and simply using a temporary 32KB buffer for the transfers?

You shouldn't paint yourself into a corner regarding the sector/cluster size as this size is not given by the OS but rather the storage device or the file system. There is nothing wrong having the ability to have larger cluster sizes for the swap file and several existing operating systems allow adjustable size.

Your problem is that you really need consecutive pages when you are low on memory. What system is this? Don't you have an IO MMU?

It's on x86_64, and the allocator works in the way I described in the first post. I'm not sure what an IO MMU is. I could keep a 32KB track reserved for "emergency", but wouldn't that just make the whole system even slower?

[gprs]

For consecutive physical RAM you first aproximate which devices need that and how much. Allocate that very early during system boot.

If you need other sizes of consecutive physical RAM then you have to walk all linear mem chunks and see what and how you can substitute physical RAM.
You can simply traverse paging structures. Slow but doesn't require any additional permanent data structures.
On the brigth side, devices drives usually allocate consecutive physical RAM very rarely.

I use 16KB linear chunks that map to 16KB consecutive physical RAM chunks for everything. And I have separate 4KB pool of physical RAM.