OSDev.org

Hello,

I've been moving into the memory management area of kernel development with my personal project recently, and I'm at the point where I can obtain a memory map from GRUB (or another compliant bootloader) and now need to process this map in order to get an idea of what the computer's memory looks like and prepare it for management. Unfortunately, as some of you may know, bochs, the emulator I'm using to test my kernel, makes very clean and orderly memory maps when it simulates a PC, and in reality, the memory of a PC isn't just a bunch of simple, ordered, sections like bochs makes it out to be. This is normally a good aspect of an emulated PC, but in order to properly test my memory map processing code, I need to make sure it works on non-ideal cases such as those presented by real hardware.

I have a real computer I can test my kernel on (besides my dev machine), but the process of compiling a kernel, creating a boot-able USB drive, moving that drive to the computer, rebooting from the drive, every time I want to check if my code works, is pretty complex and time consuming, plus, there's not much in the way of debugging apart from printing info onto the screen (Bochs has the handy debug command line and magic break instruction). Is there a way to ask bochs to "Mess up" the memory maps it emulates, so as to provide a more realistic memory map I can work with, or do I have to test my algorithm by other means? If it possible to develop a plugin for bochs which might provide that functionality (and if so, how difficult and time consuming would that be)?

In summary, the general goal here is to save time by avoiding working with a real machine, and working instead with a virtual one, but I need that virtual machine to provide a more realistic (by realistic, I mean non-contiguous and possibly unordered) memory map. Is there a way for this to happen, and would it be faster than the overhead from just testing on a real PC?

Thanks for reading,
- Mikumiku747

You could design a Bochs state that contains memory from real machines or from a page file (or a memory image after running Windows 98 or FreeDOS) and then load it. You could also run DOS for a while (make your tests and compile your kernel under the emulator) and then reboot to load it.

What's good about it is that it will allow you to get a healthy circular development process where your OS will tend to become more stand-alone than in other ways (you will need to know more tools, tricks and will keep better track of your code and progress, packed in one same stable environment to allow you to standarize and reuse your code and efforts).

Also, there's a point where you would be able to test part of your functions from the OS you normally use (at the very least by creating test native applications that contain your library functions, test them with the attitude to assume you are under raw hardware and then include them in your OS when they are debugged enough).

Mikumiku747 wrote:Is there a way to ask bochs to "Mess up" the memory maps it emulates, so as to provide a more realistic memory map I can work with, or do I have to test my algorithm by other means?

Of course, use "other means". In your case it's a very simple filtering algorithm with memory map data structure as input and your OS data structure as output. There's no need in Bochs or any hardware to test such very simple algorithm, just plain "Hello world" case for test driven development. It's better to try it, and sooner you'll start the better result you'll get.

~ wrote:Also, there's a point where you would be able to test part of your functions from the OS you normally use (at the very least by creating test native applications that contain your library functions, test them with the attitude to assume you are under raw hardware and then include them in your OS when they are debugged enough).

Of course, use "other means". In your case it's a very simple filtering algorithm with memory map data structure as input and your OS data structure as output.

Thanks for the help, I'll probably go with this option, since it seems to be the easiest to implement for now, and my project was never meant to be an operating system so much as a platform closer to the real hardware which I can play around in (Read: I never plan on getting to the point of compiling my 'OS' from inside my own OS, it could happen, but probably not). I know it's usually wiser to do things the 'right' way, but

Just as a learning experience, what would the first option involve? I'm not really super experienced with bochs (mostly just configuring it for basic use and debugging tools), but are you talking about the Suspend feature which saves machine state to a folder? If so, just so I can investigate this in future, how do I go about doing this (customising a bochs state with info I got from a real machine)? I think it's important I learn the solution in case something similar comes up again, plus, being able to create a custom machine state for bochs to load sounds like a good way to make some development shortcuts or tricks.

I'm not sure where ~ got the idea, but rebooting from a running machine doesn't affect the memory map.

The memory map depends only on the hardware and the firmware. In order to get Bochs to provide a less-convenient memory map, you'll have to modify its firmware. However, if you want to simulate extreme cases of strange memory maps, you may also need to modify Bochs itself into providing virtual hardware that better fits your chosen memory map.

The memory map sanitizer is something you can run outside your OS, so the easiest way to test it is a standalone version, as already suggested.

For me, the Microsoft Hyper-V seems to provide a pretty realistic system, but it's of course virtualization and not emulation.

The way I would go about it would be to isolate the code that processes the memory map, and unit test it. That's pretty easy if it's C, somewhat trickier with assembly but still possible.

OK, Thanks for the responses everyone, it's been helpful. Since my kernel is mostly C (with a bit of assembly where needed) it should be pretty easy to write the algorithm into a normal linux application, write down the results of GRUB2's 'lsmmap' command, and then feed that to the algorithm. Thanks again for all your help.