OSDev.org

nullplan wrote:

nexos wrote:Shouldn't Linux just start swapping pages instead of killing QEMU? Performance would be horrible, but that makes more sense to me.

The swap space is added to physically available memory for the purpose of OOM calculation. But if both RAM and swap space have run dry, the OOM killer will run. And the OOM killer will kill the biggest offender first, and on a 4GB system that will be the QEMU instance that is consuming at least 3GB.

Interesting. But why would I run out of swap space with 200 GiB swap partition?

Greetings
Peter

PeterX wrote:200 GiB swap partition

Excuse me?

rizxt wrote:

PeterX wrote:200 GiB swap partition

Excuse me?

So what?

PeterX wrote:

rizxt wrote:

PeterX wrote:200 GiB swap partition

Excuse me?

So what?

Yeah, that's not a big deal. Its smart, as you can easily consume a huge amount of memory.

On topic, OOM might detect QEMU being a bother to performace, and hence kill it. Of course, that would be very invasive of the users decisions.

Ok, people I learned a lot. For example:
- Out of memory might kill a process without giving further error message.
- I should use "-m 1g" and definitely not "-m 4g".
- And maybe for some reason my swap is turned off?
- GDB used to analyze Qemu could perhaps help to narrow it down. (If only I weren't kind of a "real programmer" when it comes to debugging...)

Greetings
Peter

If you are allocating 4GB to QEMU, that's four gigabytes. Other programs also need to use RAM, too!

Peter is your host CPU 32bit or was the 4GB restriction due to some other system limitation?

PeterX wrote:- And maybe for some reason my swap is turned off?

Maybe 200GB of swap is too much and it just plain doesn't work. Maybe QEMU is doing something that can't be swapped out, so it gets killed for using too much physical memory.

Why do you need 200GB of swap, anyway?

rizxt wrote:If you are allocating 4GB to QEMU, that's four gigabytes. Other programs also need to use RAM, too!

Yeah, we already got that topic. You remember?

PeterX wrote:200 GiB swap partition?

The kernel needs a certain amount of RAM to manage the swap. I forget how much it is in Linux, but it's proportional to the size of the swap space. It might be 1 byte per kilobyte for x86. Not sure how amd64 compares... pages in Linux are twice as large and addresses twice as long.... Does that make it the same? (I've been up all night.)

(At least we're not in the ancient era when Linux would have severe problems when swap size was more than twice RAM size. That must have been the early/mid-90s; it was over by the first version I used: 2.0.14. The book I got with it, a gigantic paperback called Dr. Linux, contained dire warnings that you must never make swap more than twice the RAM size. I followed the warning with awful results because my machine only had 4MB RAM. Everything crashed under X. Then I very nervously increased swap to 12MB and everything got much better!)

eekee wrote:

PeterX wrote:200 GiB swap partition?

The kernel needs a certain amount of RAM to manage the swap. I forget how much it is in Linux, but it's proportional to the size of the swap space. It might be 1 byte per kilobyte for x86. Not sure how amd64 compares... pages in Linux are twice as large and addresses twice as long.... Does that make it the same? (I've been up all night.)

(At least we're not in the ancient era when Linux would have severe problems when swap size was more than twice RAM size. That must have been the early/mid-90s; it was over by the first version I used: 2.0.14. The book I got with it, a gigantic paperback called Dr. Linux, contained dire warnings that you must never make swap more than twice the RAM size. I followed the warning with awful results because my machine only had 4MB RAM. Everything crashed under X. Then I very nervously increased swap to 12MB and everything got much better!)

Ok, thank you! Added to my "learned lessons" list and my "Todo list".

Greetings
Peter

PeterX wrote:

eekee wrote:

PeterX wrote:200 GiB swap partition?

The kernel needs a certain amount of RAM to manage the swap. I forget how much it is in Linux, but it's proportional to the size of the swap space. It might be 1 byte per kilobyte for x86. Not sure how amd64 compares... pages in Linux are twice as large and addresses twice as long.... Does that make it the same? (I've been up all night.)

Ok, thank you! Added to my "learned lessons" list and my "Todo list".

Quite apart from anything else, it's nice to have swap on an SSD, and putting aside soooo much expensive FLASH just to accommodate bad programming seems ... expensive. The only reason you'd so much swap is to swap memory that is most likely leaked and therefore literally useless data.

thewrongchristian wrote:Quite apart from anything else, it's nice to have swap on an SSD, and putting aside soooo much expensive FLASH just to accommodate bad programming seems ... expensive. The only reason you'd so much swap is to swap memory that is most likely leaked and therefore literally useless data.

Err... use of valgrind & other memory use debuggers is very common these days. I think some well-designed programs can't avoid heavy memory use, depending on what they need to do. However, I think most complex programs today allocate a lot of memory just to speed things up, whether in caches or something else. I think that includes automatic garbage collectors in the runtime of many languages from Emacs Lisp to .NET. On topic, Virtualizers and emulators certainly can't avoid heavy memory use for allocated RAM and the framebuffer if there is one. They may have reason to cache disk access. Non-raw disk image formats must require some memory. VMware, being a professional product which existed long before CPUs had virtualization support, might very well cache translated binaries.

eekee wrote:

thewrongchristian wrote:Quite apart from anything else, it's nice to have swap on an SSD, and putting aside soooo much expensive FLASH just to accommodate bad programming seems ... expensive. The only reason you'd so much swap is to swap memory that is most likely leaked and therefore literally useless data.

I think some well-designed programs can't avoid heavy memory use, depending on what they need to do. However, I think most complex programs today allocate a lot of memory just to speed things up, whether in caches or something else. I think that includes automatic garbage collectors in the runtime of many languages from Emacs Lisp to .NET. On topic, Virtualizers and emulators certainly can't avoid heavy memory use for allocated RAM and the framebuffer if there is one. They may have reason to cache disk access. Non-raw disk image formats must require some memory. VMware, being a professional product which existed long before CPUs had virtualization support, might very well cache translated binaries.

You have three scenarios, when you're actively using this much swap:

• You have very heavy memory use, all still managed and referenced, and periodically accessed, but working set doesn't exceed RAM.
• You have very heavy memory use, all still managed and referenced, and periodically accessed, but working set does exceed RAM.
• You have very heavy memory leakage, and the swapped out memory is no longer referenced but is taking up space.

If you have the first, and memory is being cycled between RAM and swap, typically because you have very heavy VM images, and your working set doesn't exceed the amount of RAM you have, then your application isn't utilising memory very efficiently. It's caching stuff that isn't referenced, so what's the point?

If you have the second, and memory is being cycled between RAM and swap, typically because you have very heavy VM images, and your working set exceeds the amount of RAM you have, then you'll start thrashing and your only solution is to install more RAM, or shrink the required RAM (by reducing cache sizes or migrating VMs to another machine.)

If you have the third, then you're just wasting disk space with badly behaving applications, but active working set may not exceed the amount of RAM.

The idea of a cache is to keep data on a quicker, transient storage medium. In RAM, you cache stuff that is on disk. On disk, you cache stuff that lives remotely (such as an HTTP caching proxy.) If your RAM cache ends up in swap, your cache is oversized.

It's possible for memory to be managed and never leaked as such, but still wastefully allocated. (That's what I was trying to say.) For a very simple example, (perhaps too simple,) every integer in Python needs another integer to hold its type and reference count. Functional programming also tends to lead to a lot of memory use where imperative programming can use less. This all might end up being a problem for me; some of my ideas would go down that road.