x86 32-bit paging management (Pillar Subject)

[~]

I created a new Administrator user, so the resource usage is almost nothing (the CPU cores are also running to 3% or 1% and up).

Having the full native power is well worth it (more than 4GB of RAM, 16/32/64-bit emulators, BIOS, DOS, DPMI, VGA, VESA, Sound Blaster, DirectX, Win16, ZSNES, NO$GMB, Z26, Nesticle, Turbo C, Pascal, all of the old demos).

But the question is, even with that much RAM, does Windows let so much fragmentation happen without swapping out as to have to close running windows programs to reserve 40 contiguous Megabytes? It looks like having that amount of tolerance to fragmentation would be enough most of the time for a simple paged OS.

The screenshot is Resource Hacker from SourceForge. 1.6GB is only 23% of usable PAE RAM. If I open 32 YouTube tabs in Firefox, it goes beyond 4GB:

[LtG]

According to Wikipedia 32-bit Windows limit RAM to 4GiB, reason was that drivers didn't work well with more, alternatively it was due to licensing.
https://en.wikipedia.org/wiki/Physical_ ... ft_Windows

You have a 64-bit CPU, why not use 64-bit OS?

As far as the memory allocation in half-life failing, have you disabled swapping? I would imagine that if swapping is enabled, then Windows will swap enough RAM to disk to make room for the 40MiB allocation. If all RAM is used and swapping is disabled, then all allocations will fail.

[~]

The swap file is 4GB and enabled. Maybe it should be 16GB for 12GB.

The 16-bit and DPMI help me make most of my OS. I can't abandon all the code for learning, 64-bit OSes turn into an unbearable desert when I want to learn more tricks for the lack of PC-specific code and hardware compatibility.

[LtG]

The 16-bit and DPMI help me make most of my OS. I can't abandon all the code for learning, 64-bit OSes turn into an unbearable desert when I want to learn more tricks for the lack of PC-specific code and hardware compatibility.

I don't understand what you are trying to say here.

Personally I prefer for my host OS to have all the resources it can (64-bit OS using all of 64GiB of RAM), and then I start a VM (Virtual Machine, like VMWare, Qemu, etc) that can have as little RAM as I want, and do there whatever I want.

So why don't you use a 64-bit OS, start a VM and do your 16-bit DPMI stuff there?

[~]

Every emulator layer has all the physical reserved Gigabytes less of RAM, so it's bad just for that when tiny things can be run in the real machine.

I have like 75 old books with floppy/CDs and other PDFs/ZIPs/ISOs that work under DOS or Win16 (VxDs, fonts, SVGA, 3D, file formats, WinAPI, MFC).

I have mirrors from programmersheaven, planetsourcecode and many FTP repositories that have Gigabytes of interesting tricks but in 16-bit mode.

I don't find material that teaches the basics from the mining principles for the components in 64-bit mode or in well-finished platforms like pure Linux or Windows, etc. I can learn in 16/32-bit environments and port to 64-bit or JavaScript. I can't run the code to see how the real hardware behaves with Win64.

I like to handle VGA/VESA graphics. With that, DPMI, my emulation, and Sound Blaster, Bochs, and some old Pentiums, I can implement my OS under Windows 7 without rebooting. What depends on the host OS can be replaced by ring 0 assembly components later, but an OS with GUI, manual VGA and SVGA for the current card, indirect network through special files, and native APIs can be done in Win32. I could add a 64-bit emulator for 32-bit CPUs, and a 16-bit one for 64-bit ones.
----------------------------------------------------------
----------------------------------------------------------
----------------------------------------------------------
----------------------------------------------------------
Skeleton of Page-Disabled Functions

Code: Select all

OPCODE__CPU_x86_32__page_disable_function_skeleton:
;Save flags and temporary registers,
;disable interrupts and paging,
;add the custom function code:
;;
 pushfwide
 push wideax
 ;;
 cli
 call OPCODE__CPU_Read_Default_Paging
 push wideax
 
 call OPCODE__CPU_Disable_Default_Paging
 ;;











 ;;
 pop wideax
 call OPCODE__CPU_Restore_Default_Paging
 ;;



 pop wideax
 popfwide
;Enable paging and restore temporary registers
;(restoring FLAGS will restore interrupts
;to their intended state):
;;
retwide

Skeleton of Page-Disabled Functions for cdecl/Local Variables



Skeleton of Page-Disabled Functions for stdcall/Local Variables



Get CR3 Value From CR3 Array Index

Code: Select all

      ;Get the base address of the specified
      ;page directory in WIDEDX:
      ;;
       mov widedx,.CR3_array_index
       mov widedx,[CR3_array+(widedx*wideword_sz)]

[Ethin]

I'm quite confused on how an emulator/virtual machine can't satisfy your needs. (Then again, I'm quite confused on why anyone would ever want to do what your doing, too.) A VM can run in 16-bit mode, 32-bit mode, and 64-bit mode. An emulator can, too. Do all your tricks and hacks and other "learning" in that sandboxed environment. So, explain (again) how a VM or emulator can't do what you want to do, and how physical hardware can, somehow, do what a VM can't?
I would understand if your testing your OS on physical hardware to fix bugs and to make workarounds for buggy hardware, but this does not seem like it at all.

[~]

I can perfectly do the following with this library version (moving big physically fragmented blocks that don't fit for expansion and shrinking is left to do):

Code: Select all

tmprealloc=malloc(655360);
realloc(tmprealloc,1310720);
free(tmprealloc);

https://sourceforge.net/projects/lowest-kernel/files/

I have to test allocating/reallocating up and down/freeing blocks that span several page directory entries, like 40MB blocks.

[~]

What should happen if I allocate, say, a 32768-byte buffer ad get a ptr value (ptr=malloc(32768)) and then I do a free with a pointer N bytes later (ptr+=N; free(N))?

[reapersms]

if ptr+N happens to end up as another pointer you got from malloc(), it will probably work. If it is anything else, it will probably fail horribly.

The math to do that is technically undefined, given the rules about what you can point to, and the limits of pointer arithmetic, but those don't really apply to actually implementing malloc.

Most malloc implementations store the book keeping and management information in-line with the memory blocks, just before and after the block itself, and use the pointer passed in to find it. If you pass a random pointer in, it will look for that data just before the value you pass in, find who knows what, and then promptly corrupt your heap.

Some paranoid ones could protect from that with some canary values, sanity checks, or explicitly walking the allocation list to find the block. Those are rare in practice, as that can make free() rather slow, or increase the allocation overhead unreasonably.

There are some allocators that keep their bookkeeping data out of line, but they tend to not use quite the same interface as malloc/free, other than, say, page allocators.

[nullplan]

Some paranoid ones could protect from that with some canary values, sanity checks, or explicitly walking the allocation list to find the block. Those are rare in practice, as that can make free() rather slow, or increase the allocation overhead unreasonably.

Well, musl's malloc isn't that slow, and it keeps a tiny bit of safety information: The size of the chunk is saved in a header as well as a footer, and if they don't match in free(), then musl will deliberately crash. This basically just fell out of the design of having all memory chunks in the same block as part of a doubly linked list.