mmap/mumap and files ...

[distantvoices]

Very Recently, I've completed some features in the dynamic memory allocation stuff, aka morecore (wraps around mmap), mmap - which I've without thinking christened do_sbrk - moron thou' I am if low on coffee and high on need of sleep *gg* - mumap is still missing. But this one is to come too.

Now, in relation to mmap, something bugs me: mmap of a file.

Let's draw a rough sketch of how it *might work*:

1. Process calls mmap (filepointer,adress,length,mapped_file);

2. mm service adds an entry to the process adress space tree, marks the entry as "mapped file" and saves the filepointer.

3. Process goes on and strolls throu the file as if it were an array: while(mapped_file[ i ]!=0)printf("%c",mapped_file);

4. Upon access of not present page in adress space, the pager asks the mmservice: whats up with that pagefault.

5. MMservice checks the faulting process adress space, finds the entry of the mapped file: it tells the pager to *add* one page to the adress space, and then it tells the fs_service to load the next 4096 kb of the mapped file to the start of the recently acquired page. So, the file would be loaded into memory in chunks of 4096 kb while the userprocess walks throu' it. - until it is loaded completely and we have reached EOF.

MUMAP simply removes the mapping from the adress space and causes MM service inform the pager to free/clean the allocated pages.

would this work? or is it completely *rubbish*?

thanks for feedback!

stay safe

[Pype.Clicker]

basically, yes. That's it. Though you have the free option of pre-loading pages that haven't faulted yet (taking benefit of track-caching at the disk if your file is made of contiguous runs),j which will greatly improve performances if disk is not much used by other applications atm, since it means the CPU will not have to wait the disk.

Another thing that could be funny for non-linear-accessed files would be to use an oracle saying "you should pre-load page X, based on the current access history".

Hidden Markov Models might also be helpful here, allowing the OS to collect access patterns in a compact way in order to use them later to guess what's the next page that is likely to be requested, and with what probability...

[Candy]

Now, in relation to mmap, something bugs me: mmap of a file.

Let's draw a rough sketch of how it *might work*:

1. Process calls mmap (filepointer,adress,length,mapped_file);

2. mm service adds an entry to the process adress space tree, marks the entry as "mapped file" and saves the filepointer.

I would design it by making the mmap handler tell the page fault handler for the appropriate space (kernel or user space) add that series of pages to be handled to/from that file, and then letting the page fault mechanism handle it.

3. Process goes on and strolls throu the file as if it were an array: while(mapped_file[ i ]!=0)printf("%c",mapped_file);

4. Upon access of not present page in adress space, the pager asks the mmservice: whats up with that pagefault.

I get it, microkernel . How do you do that with numerous user spaces and one service taking all the page fault hits? Doesn't that limit your memory amount?

5. MMservice checks the faulting process adress space, finds the entry of the mapped file: it tells the pager to *add* one page to the adress space, and then it tells the fs_service to load the next 4096 kb of the mapped file to the start of the recently acquired page. So, the file would be loaded into memory in chunks of 4096 kb while the userprocess walks throu' it. - until it is loaded completely and we have reached EOF.

you can write mmapped files beyond EOF. They grow a page at a time.

MUMAP simply removes the mapping from the adress space and causes MM service inform the pager to free/clean the allocated pages.

would this work? or is it completely *rubbish*?

Translating all your microkernel concepts to monolithic concepts, it's identical to what I am planning, and afaik Windows and Linux do. I don't think that's rubbish.

There's even an at least 95% chance you're using a program using Windows MMAP functions to access this page, theoretically

[distantvoices]

@candy:
I will of course grow short on physical memory as long as there is no swap thing present.

All multiple user processes and all the pagefaults going to one service: mm service is multithreaded: one thread is responsible for pagefault checking and mmap/mumap and the other thread takes the fork,exec,exit (and other) calls - which, you 've got it, are actuallys messages.

FS service will be multithreaded too - in the near future, as soon as I find the time to implement it: each file system gets a dedicated thread in the fs service - and talks with the devices it is responsible for, and a main thread just fetches messages and queues them to the work horse threads (kinda IO queueing). This design is very likely to change for I smell the scent of bad things (tm) in it. So to say: still tinkering with it.

@pype: random, non linear acces to the file implies translating the start of the missing page into an offset into the file -> the blocks to fetch next from disk. This requires a *seek*.

STill learning I am and neverending the secrets of this lore are ... arent they my preciousss? *gg*

OT: a few weeks ago I've done myself a treat and purchased Lord Of the Rings Part II Extended Edition. Looked as if it 's been one of the last pieces in stock.

[Candy]

All multiple user processes and all the pagefaults going to one service: mm service is multithreaded: one thread is responsible for pagefault checking and mmap/mumap and the other thread takes the fork,exec,exit (and other) calls - which, you 've got it, are actuallys messages.

FS service will be multithreaded too - in the near future, as soon as I find the time to implement it: each file system gets a dedicated thread in the fs service - and talks with the devices it is responsible for, and a main thread just fetches messages and queues them to the work horse threads (kinda IO queueing). This design is very likely to change for I smell the scent of bad things (tm) in it. So to say: still tinkering with it.

In a way I'm starting to compare microkernels with a circus parade in my head. To make the parade more simple you transform all animals into superintelligent shades of the color blue, so you can then use a simple way of determining which animal is which... it's a way to do it, it looks really cool (I like blue ) but it's so incredibly overdone and not appropriate.

@candy:
I will of course grow short on physical memory as long as there is no swap thing present.

Implement mmapping in terms of swapping. A mmapped file is a region that is permanently a swap space for a certain area in virtual memory. A normal swap file is a region that is temporarily (or even permanently, you could do that) a swap space for a certain area in virtual memory. Use the parallelism between the two to use a single solution, giving you higher performance... at least, that's the way I see it. The only exceptions are the growing of the file (which is dangerous! think overlapping other things!) for which you need a special page fault handler.

The idea in more concrete terms:

Code: Select all

struct swap_page {
  uint32 hash; // for quick lookups of a page
  uintn virt; // native int (32/64 bit) for the address
  uintn swap_addr; // address in the swap file
  inode_t swap_file;  // the file that swaps it
  uint16 flags; // permanent store, temporary store
} swaps[some_amount];

enum swap_page_flags {
  SPF_PERMANENT = 0x00000001
};

@pype: random, non linear acces to the file implies translating the start of the missing page into an offset into the file -> the blocks to fetch next from disk. This requires a *seek*.

sorry if I wasn't supposed to reply:
nonrandom linear access of a file that's fragmented still results in seeks, and even linear access of linear files is probably going to get you seeking (considering having other files open). Swap is not fast, mmapped files are not fast. Point is, they're both equally fast, and you'd be using swap anyways, so you're in the same situation.

STill learning I am and neverending the secrets of this lore are ... arent they my preciousss? *gg*

OT: a few weeks ago I've done myself a treat and purchased Lord Of the Rings Part II Extended Edition. Looked as if it 's been one of the last pieces in stock.

I do seem to have a nice avatar ;D

[distantvoices]

Hmmm ... I didn't refer to the head movement and the picking of block numbers out of the inode/single/double/triple indirection blocks to fed them to the driver, but to change the actual position in the file as indicated in the file pointer,to which an inode(or what soever in FAT) usually is attached, because from this position, the block which is to be retrieved is calculated by turning it into a series of indices into the inode/indirection blocks.

I'm of course aware of the possibility to say: Read Ahead -> send a bunch of block requests to the driver which executes them one after each other.

re avatar: Oh yess, gollum gollum, these bagginsess have it, my preciousss ... *ggg*