Physical Memory Management : Two Unsuccessful Scenario

[RezaNoei]

Hello,

my os developmend has began with Nick Blundell's operating System development textbook, where he placed the kernel at 0x7e00.
since his tutorial was limited to loading a simple kernel, i have searched for a better tutorial. i have found "brokenthorn os development series".

scenario 1 :

where i load the kernel at 0x7e00 i want to place a bitmap for taking control over the all memory block at physical level.
my kernel size was 0x4210 and the base address as i said was 0x7e00,
then my bitmap as in brokenthorn tutorial has been placed right after the kernel ( in 0xc010 ).
problem :
pmmngr_init will write value 0xff to 0xc010 - 0xd003 to say all blocks of memory is in use.
when this function suppose to write on 0xc142, everything gets crazy like those time that there was an
unhandled interrupts.

Is the value of 0xc142 used for special purpose, then i can't write to that ?


scenario 2 :

Loading the kernel in 0x100000 :

So, I have been decided to load the kernel at 0x100000 but CopyKernelImage Doesn't work Correctly like an infinite loop :

Code: Select all

CopyKernelImage:
  	 mov	        eax, dword [ImageSize]
  	 mov         ebx, 4
  	 div           ebx
   	 cld
   	 mov         esi, 0x7e00
   	 mov	        edi, 0x100000
   	 mov	        ecx, eax
   	 rep	 movsd   

please help me to find the problem

[mariuszp]

According to the x86 memory map, that address should be free for use. Exactly what interrupts are you getting?

Did you enable the A20 line?

Have you checked (e.g. with the Bochs debugger) if ImageSize was loaded correctly? Do registers have the expected values just before rep movsd?

Side note: There is a simpler (and faster) way to divide by 4. Instead of:

Code: Select all

      mov           eax, dword [ImageSize]
      mov         ebx, 4
      div           ebx

You could have just written:

Code: Select all

CopyKernelImage:
    mov ecx, [ImageSize]
    shr ecx, 2                     ; divide by 2^2 (=4)
    cld
    mov esi, 0x7e00
    mov edi, 0x100000
    rep movsd

[RezaNoei]

According to the x86 memory map, that address should be free for use. Exactly what interrupts are you getting?

Did you enable the A20 line?

Have you checked (e.g. with the Bochs debugger) if ImageSize was loaded correctly? Do registers have the expected values just before rep movsd?

Side note: There is a simpler (and faster) way to divide by 4. Instead of:

Code: Select all

      mov           eax, dword [ImageSize]
      mov         ebx, 4
      div           ebx

You could have just written:

Code: Select all

CopyKernelImage:
    mov ecx, [ImageSize]
    shr ecx, 2                     ; divide by 2^2 (=4)
    cld
    mov esi, 0x7e00
    mov edi, 0x100000
    rep movsd

I Know That, I said that was similar to an unhandled interrupt not a real interrupt,

About A20 Line, Yes, i will not allow to the kernel to work if it is disable

i'm not using of bochs but i believe that the kernel has loaded correctly. about registers i have written some value directly to check that, and it doesn't work as explained before.

thanks,

[embryo2]

i'm not using of bochs

It's very bad practice. You aren't see what you're doing, like a blind man.

but i believe that the kernel has loaded correctly

Any belief is bad when you are going to program something useful. Only the facts that you can see can be your ground. Until you can't see the actual register values your belief just deceives you.

about registers i have written some value directly to check that, and it doesn't work as explained before.

It means you have made an error and now just can't find it. Yes, it's a normal practice, but how are you going to fight it? With your beliefs only? Most probably you'll be hunting the error(s) much longer than it takes to install Bochs and to get used to it.

[neon]

Hello,

We are a little uncertain about the provided code given the reference to the series. The series boot loader and kernel define imageSize and kernelSize as the number of sectors rather than the number of bytes. The presented CopyKernelImage as well as what you expected appear to be inconsistent with the series. Please compare your presented code with the series stage2.asm and main.c (Specifically CopyImage and how we call pmm_initialize.)

If this is a custom boot loader, please let us know. If it is only modified, please try the original version. However we suspect it is an incorrectly modified version of the boot loader resulting in an error during loading.

[Brendan]

Hi,

So, I have been decided to load the kernel at 0x100000 but CopyKernelImage Doesn't work Correctly like an infinite loop :

Code: Select all

CopyKernelImage:
  	 mov	        eax, dword [ImageSize]
  	 mov         ebx, 4
  	 div           ebx
   	 cld
   	 mov         esi, 0x7e00
   	 mov	        edi, 0x100000
   	 mov	        ecx, eax
   	 rep	 movsd   

please help me to find the problem

This DIV instruction divides the 64-bit value in EDX:EAX by the 32-bit value in EBX and stores the 32-bit result in EAX (and the 32-bit remainder in EDX). It's entirely possible for the value in EDX:EAX to be so large that the result will not fit in 32 bits (e.g. 0x1234567800000004 / 4 = 0x48D159E00000001 = too big for 32 bits) and in this case you get a "divide error" exception. Because you don't set EDX to anything I assume this is the cause of your problem.

Also note that if "ImageSize" isn't a multiple of 4 (e.g. 10003 bytes) you don't copy the last few bytes of the image.

For larger copies (e.g. multiple cache lines or maybe 256 bytes or more) the CPU optimises string instructions to work on entire cache lines and not individual bytes/words/dwords. For this reason "REP MOVSB" is likely to be just as fast as "REP MOVSD"; which means that it'd be more efficient (and fix both of the bugs mentioned above at the same time) to do this instead:

Code: Select all

CopyKernelImage:
  	 mov ecx, dword [ImageSize]
   	 cld
   	 mov esi, 0x7e00
   	 mov	edi, 0x100000
   	 rep movsb

Finally; you should expect that (eventually) "ImageSize" will grow to several MiB. This means you're probably (eventually) going to have to load a small part and copy it above 1 MiB, then load the next small part and copy it above 1 MiB, and so on. If you're sure that the entire image will actually fit in the limited amount of RAM that can be accessed in real mode, then there's no point copying it in the first place (given that you should be enabling paging and mapping it into virtual memory, which makes its physical address mostly irrelevant) .


Cheers,

Brendan

[RezaNoei]

i'm not using of bochs

It's very bad practice. You aren't see what you're doing, like a blind man.

but i believe that the kernel has loaded correctly

Any belief is bad when you are going to program something useful. Only the facts that you can see can be your ground. Until you can't see the actual register values your belief just deceives you.

about registers i have written some value directly to check that, and it doesn't work as explained before.

It means you have made an error and now just can't find it. Yes, it's a normal practice, but how are you going to fight it? With your beliefs only? Most probably you'll be hunting the error(s) much longer than it takes to install Bochs and to get used to it.

yes, I am a blind man , but Im using qemu emulator for testing.
Im using print function calls for seeing some information about currently executed code.
thank you, I am in a wrong way.

[RezaNoei]

Hello,

We are a little uncertain about the provided code given the reference to the series. The series boot loader and kernel define imageSize and kernelSize as the number of sectors rather than the number of bytes. The presented CopyKernelImage as well as what you expected appear to be inconsistent with the series. Please compare your presented code with the series stage2.asm and main.c (Specifically CopyImage and how we call pmm_initialize.)

If this is a custom boot loader, please let us know. If it is only modified, please try the original version. However we suspect it is an incorrectly modified version of the boot loader resulting in an error during loading.

yes, it was me A man with alot of baby mistakes. i'm using real flash memory instead of a floppy image. there was a problem in mapping CHA to LBA that lead me to avoid from FAT12 and when there is'nt any FAT then There isn't any value such as number of sectors of the kernel file.
i have decided to avoid from using FAT12 in brokenthorn os development series and i know that i should solve many problem in every steps.

my mistake : (excuse me friends, that was a really baby mistake, i have tried to copy kernel file into 0x100000 in the real mode so the result was a core dump.

know i have another problem ( that i called him the lake of information ) i must find the entry point of the kernel in elf_i386 format specification.

thanks

[BASICFreak]

Spoiler:

ELFHead->e_entry

I think the offset from the beginning of the ELF is 24 (for elf-i386) (I did not look up, just checked my boot code...DWORD [EDI + 24])

* but for a Kernel you should really put a wrapper that is at .text 0 with address at 0x100000 and then just jump to 1MB

[neon]

Hello,

It appears the problem has been resolved then. I would not recommend avoiding something due to a problem that can not be resolved; it is better to overcome the problems and support it if necessitated by your system requirements. With that said however, we do typically use fat32 hard disk volumes in our system and provide support for those that want to deviate from the legacy fat12 media. We also provide copies of our boot records if needed for these file systems.

As noted above, the boot loader you presented has a number of problems that should be addressed later on, most notably that it does not support loading large files (which requires swapping between real and protected/long modes.) I encourage looking into expanding the software later on since the boot loader can easily be itself a large project. Since your boot loader appears to be somewhat based off the one in the series, it should be noted that it will be unable to load images that exceed 65k without, eventually, corruption, do to readSectors not supporting loading across segment boundaries. We can provide our updated version if requested, but you will still be limited by the real mode address space.

Finally, I highly recommend learning the Bochs debugger. It is your only means of debugging (next to displaying contents on screen, dumping data to serial port and redirecting it to file, and trial and error) until you are able to port or write a kernel symbolic debugger. Besides, you will need to be able to run and test your software in a lot of different environments; just because it runs in one emulator does not guarantee (and often it won't) work in another. The only way to catch these types of bugs is by running the software in a lot of different architecture environments; either real or emulated.

[RezaNoei]

Hello,

It appears the problem has been resolved then. I would not recommend avoiding something due to a problem that can not be resolved; it is better to overcome the problems and support it if necessitated by your system requirements. With that said however, we do typically use fat32 hard disk volumes in our system and provide support for those that want to deviate from the legacy fat12 media. We also provide copies of our boot records if needed for these file systems.

As noted above, the boot loader you presented has a number of problems that should be addressed later on, most notably that it does not support loading large files (which requires swapping between real and protected/long modes.) I encourage looking into expanding the software later on since the boot loader can easily be itself a large project. Since your boot loader appears to be somewhat based off the one in the series, it should be noted that it will be unable to load images that exceed 65k without, eventually, corruption, do to readSectors not supporting loading across segment boundaries. We can provide our updated version if requested, but you will still be limited by the real mode address space.

Finally, I highly recommend learning the Bochs debugger. It is your only means of debugging (next to displaying contents on screen, dumping data to serial port and redirecting it to file, and trial and error) until you are able to port or write a kernel symbolic debugger. Besides, you will need to be able to run and test your software in a lot of different environments; just because it runs in one emulator does not guarantee (and often it won't) work in another. The only way to catch these types of bugs is by running the software in a lot of different architecture environments; either real or emulated.

Hi,
that was very useful to me,
maybe one of the reasons that i don't like to spend more time on FAT12 was the limitation of it, indeed i would highly prefer to implement FAT32 instead of FAT12 to work with the real hardware. i will try to use bochs as you said for many reasons.

[RezaNoei]

Hello,

Finally I loaded the kernel at 0x100000,
my kernel will work just like before, but there is still the same problem,

When memset tries to write in 0x104142, one page fault interrupt cause to stop everything.

where this page fault came from ?

it seems this address has been overlapped with my interrupt handlers ...

[BASICFreak]

Well a page fault is int 0x0E and if you did not remap the PIC, a floppy INT is also int 0x0E (highly unlikely)

Other than that, what does your memset look like?
Are you sure it is a #PF?
Have you enabled paging?
Have you mapped the page? Present and writable?

If it overlaps your int handlers it shouldn't cause a #PF unless the page is marked RO. (not saying it should be allowed to overlap it...)





As suggested before learn to (fully / properly) use Bochs or maybe QEMU - it really helps in debugging.
I know Bochs allows one to transverse the TLB (I do not use QEMU so I cannot speak for it).

[RezaNoei]

Well a page fault is int 0x0E and if you did not remap the PIC, a floppy INT is also int 0x0E (highly unlikely)

Other than that, what does your memset look like?
Are you sure it is a #PF?
Have you enabled paging?
Have you mapped the page? Present and writable?

If it overlaps your int handlers it shouldn't cause a #PF unless the page is marked RO. (not saying it should be allowed to overlap it...)





As suggested before learn to (fully / properly) use Bochs or maybe QEMU - it really helps in debugging.
I know Bochs allows one to transverse the TLB (I do not use QEMU so I cannot speak for it).

im sorry but i think, i must migrate from a usb based development to floppy based one (because i used some tricks for taking more sectors from a cylinder, may be you cant believe it, but i have loaded more than 50 sector with int 0x13 while it must be lower than 18 in count) that was because i had some problem with mapping LBA to CHS and now i must resolve that.
i must implement floppy driver and some other things that i have been ignored them.
until then, i cant forgive myself for discussing bad questions and taking your time, maybe there are real problems that must solve them but before solving those problems i must sync my development with my references,

i will back in less than one week to report the results and if there was any problem.

thank you,

[RezaNoei]

Hello Firends,

I'm returned, with new problems.

i leaved my os, and i began to read some other tutorials like JamesMolloy and "The little book about os development".
now i'm using bochs as my debugger and i work with GRUB bootloader instead of my own bootloader,

i'm using boot.s file from bare_bones tutorial : http://wiki.osdev.org/Bare_Bones
but i added this line, right before calling kernel_main()

Code: Select all

pushl %ebx

because i want to get multiboot header from this loader.

and i imported multiboot.h from this address : http://www.gnu.org/software/grub/manual ... 002eh.html
(good formed structures and different than JamesMolloy one's) to my project.
i have got my floppy image from jamesmolloy tutorial and i'm not depend on that (if needed i can create my own),
but jamesmolloy's multiboot structure is not similar to my multiboot structure.

Question : is my multiboot structure placed in wrong place ? then i mustn't use jamesMolloy's structure ?

How can I get multiboot header to work in a correct way ?