Memory Heap Value Seems Incorrect

[TheGameMaker90]

So for this one, I am trying to get started with modifying the heap I have in place for a proprietary heap implementation. My problem is that I seem to be getting the wrong address back from my initialization code.

Example:

Code: Select all

uint32_t *test = (uint32_t *)0xC010000;
printf("Address: 0x%X.\n", &test);

Output:
Address: 0x10DC48

...

I believe I pointed the address to 0xC0100000, as indicated as the start address of a higher half kernel. Is this the intended result or am I missing something here?

[iansjack]

Did you mean to print the address of the variable test rather than the pointer value that you initialized it with? What you print will be wherever on the stack the variable is located. What is the address range of your stack?

[TheGameMaker90]

Did you mean to print the address of the variable test rather than the pointer value that you initialized it with? What you print will be wherever on the stack the variable is located. What is the address range of your stack?

Goodness, I'm still figuring all of this stuff out. Do you mean in the bootloader? Not sure. It's very simple but the bootloader stack is defined like this:

Code: Select all

.section .bss
.align 16
stack_bottom:
     .skip 16384
stack_top:

Also I'm using GRUB, and in the linker it starts at the . = 1M; (So 1MB) boundary. Also all sections are set using BLOCK(4K) : ALIGN(4K). (in the linker that is.)

Hope all of that is useful to help me.

[foliagecanine]

I believe this is the intended result.

Code: Select all

uint32_t *test = (uint32_t *)0xC010000;

initializes a new 32-bit variable (likely located somewhere in the data segment, which makes sense with the address of 0x10DC48) with the value 0xC010000. Pointers are just like any other variable stored in memory. The only difference is how they are used in the programming language.

Code: Select all

printf("Address: 0x%X.\n", &test);

takes the address of the new 32-bit variable (test) and prints it out. It happens to be placed at 0x10DC48 by the compiler.

If you use

Code: Select all

printf("Address: 0x%X.\n", test);

you will find that it prints out "Address: 0xC010000"

Although it will complain about implicitly casting from a pointer to an integer unless you do something like this:

Code: Select all

printf("Address: 0x%X.\n",(uint32_t)(intptr_t)test);

------------------------------------------

Goodness, I'm still figuring all of this stuff out. Do you mean in the bootloader? Not sure. It's very simple but the bootloader stack is defined like this:

Code: Select all

.section .bss
.align 16
stack_bottom:
     .skip 16384
stack_top:

No, that is not the bootloader. That code is just the start code that first runs when the bootloader (GRUB) loads your OS. However, that (along with your linker) does show where your stack is.

[TheGameMaker90]

I believe this is the intended result.

Code: Select all

uint32_t *test = (uint32_t *)0xC010000;

initializes a new 32-bit variable (likely located somewhere in the data segment, which makes sense with the address of 0x10DC48) with the value 0xC010000. Pointers are just like any other variable stored in memory. The only difference is how they are used in the programming language.

Code: Select all

printf("Address: 0x%X.\n", &test);

takes the address of the new 32-bit variable (test) and prints it out. It happens to be placed at 0x10DC48 by the compiler.

If you use

Code: Select all

printf("Address: 0x%X.\n", test);

you will find that it prints out "Address: 0xC010000"

Although it will complain about implicitly casting from a pointer to an integer unless you do something like this:

Code: Select all

printf("Address: 0x%X.\n",(uint32_t)(intptr_t)test);

------------------------------------------

Goodness, I'm still figuring all of this stuff out. Do you mean in the bootloader? Not sure. It's very simple but the bootloader stack is defined like this:

Code: Select all

.section .bss
.align 16
stack_bottom:
     .skip 16384
stack_top:

No, that is not the bootloader. That code is just the start code that first runs when the bootloader (GRUB) loads your OS. However, that (along with your linker) does show where your stack is.

Oh if only it were as simple as that... If you can believe it, I get a completely different result. No such complaint and it prints out 0x3FF00000 now. That is without the address of (&) operator. At least I now know why I was getting a page fault at that address before. I suppose though, I should remove the current heap implementation and paging first though...

Aaaaand... No such luck. I thought that it might have been a virtual address location designated by the current heap implementation. I can't understand why it would be so off...

[foliagecanine]

If you could supply your code (i.e. on GitHub) for us to look at, that would be helpful for diagnosing the problem.
Specifically, if you are willing, please post the linker script and the file in which this code is running.

It may be other code around it or the linker causing the problem, since compiling

Code: Select all

#include <stdio.h>
#include <stdint.h>

int main() {
    uint32_t *test = (uint32_t *)0xC010000;
    printf("Address: 0x%X.\n",(uint32_t)(intptr_t)test);
    return 0;
}

with a C compiler and run it, it outputs "Address: 0xC010000."

Make sure you aren't modifying the value of test anywhere else as well.

[TheGameMaker90]

If you could supply your code (i.e. on GitHub) for us to look at, that would be helpful for diagnosing the problem.
Specifically, if you are willing, please post the linker script and the file in which this code is running.

It may be other code around it or the linker causing the problem, since compiling

Code: Select all

#include <stdio.h>
#include <stdint.h>

int main() {
    uint32_t *test = (uint32_t *)0xC010000;
    printf("Address: 0x%X.\n",(uint32_t)(intptr_t)test);
    return 0;
}

with a C compiler and run it, it outputs "Address: 0xC010000."

Make sure you aren't modifying the value of test anywhere else as well.

I wish I could post the link, but it's on a private repo for a reason... It's privileged code only for my company Psionix Softworks, despite the fact that I work "alone" on it. I can provide snippets here and there, but most things are renamed to protect actual source base. I have created a function called "test_code" for the purposes of testing before implementation. So no, it isn't used elsewhere. Here's my linker (temporary, with unimportant things left out/omitted):

Code: Select all

...
. = 1M;
.text BLOCK(4K) : ALIGN(4K)
{
    *(.multiboot)
    *(.text)
}
.data BLOCK(4K) : ALIGN(4K)
{
     *(.data)
}
.rodata BLOCK(4K) : ALIGN(4K)
{
    *(.rodata)
}
.bss BLOCK(4K) : ALIGN(4K)
{
    *(COMMON)
    *(.bss)
}
kernel_end = .;
...

As for the code that the address value is assigned, it's literally after the initialization of the other modules with my memory module and paging code commented off as to not interfere.

Paraphrased in my main module:

Code: Select all

<kernel-type> kmain(void)
{
    ...
    test_code();
}

static inline void test_code(void)
{
    uint32_t *test = (uint32_t *)0xC0100000;
    printf("Address: 0x%X.\n", test); // Also tried "&test"
}

Literal Output:
...
Address: 0x3FF00000. // Without '&'

And:
...
Address: 0x10DC4C. // With '&'

Nothing too special really. This code was tested with minimal modules to avoid conflicts.
Also I noticed tou missed a zero. It's a uint32_t, as such, it should have 8 digits in the hex value: 0xC010 0000.

Edit:
I feel that I should also mention that this is in an attempt to replace my current (working) heap. I borrowed it from elsewhere and now want to replace it with my own code. As such, I am in the process of building my heap from the ground up. So this location in memory should not be influenced by other code since it is most definitely independent. And in Visual Studio 2017 I got the same result as you foliagecanine. I just can't understand why all of a sudden the addresses are all so far off. Let me try something else...

Edit2:
No luck. I thought that if I pulled the address of an array index it would work. Yesterday I was using smaller values like 0x4000 and it was working. Now it seems to not work. I can't remember if it was because I removed the old heap or left it in, but for some reason it doesn't seem to want to work right...

Okay so I tested with just 0xC010, could it be a "number too large" case? I feel like that's what it is. test[0] = 0xC010, test[1] = C014, which makes sense if it's aligned by 4 bytes. How would I fix that if that's what it is?

Edit3:
Okay so here's something interesting. I decided to test a different way...

Code: Select all

if (&test[0] == 0xC0100000)
    printf("&test = 0xC0100000");

At which point it printed:
&test = 0xC0100000

Perhaps it's the way large numbers are being represented in hexadecimal. Maybe indeed...
I just tried something:

Code: Select all

printf("%d\n", 0xFFFFFFFF);

And it prints: -1...
Hmm. That can't be normal...

Figured it out actually... It wasn't that the address was misplaced, it was how I printed hex values to the screen in my print function. Now it prints the correct address. However, without the array index it still prints a 6 digit hex value starting at 10XXXX. Is that normal?
Also, would it be a bad idea to consider starting at 0x7FFFFFFF?

[Octocontrabass]

However, without the array index it still prints a 6 digit hex value starting at 10XXXX. Is that normal?

According to your linker script, yes.

Also, would it be a bad idea to consider starting at 0x7FFFFFFF?

The typical higher-half example places the kernel's code and data in the 0xC0000000-0xFFFFFFFF range to make page table management easier, since that address range is covered by a single PDPTE. If you think reserving a different portion of the address space for the kernel makes more sense, you can do that instead.

[TheGameMaker90]

However, without the array index it still prints a 6 digit hex value starting at 10XXXX. Is that normal?

According to your linker script, yes.

Also, would it be a bad idea to consider starting at 0x7FFFFFFF?

The typical higher-half example places the kernel's code and data in the 0xC0000000-0xFFFFFFFF range to make page table management easier, since that address range is covered by a single PDPTE. If you think reserving a different portion of the address space for the kernel makes more sense, you can do that instead.

Fair enough, but what part of my linker script dictates that.

And That makes sense. Here's something I've also been wondering: Is the kernel heap itself supposed to be the max size of RAM? Like I have 16GB of RAM in my (custom built) computer and was hoping to be able to allocate up to that 16GB for later use if possible. Perhaps even when I finally get to UserLand.
If not, then how (besides in 16 bit real mode since I'm using GRUB as a bootloader) would I detect how much RAM my computer has. Or is it already stored somewhere I can pull it from. Is it something I should worry about now or later?

[Octocontrabass]

Is the kernel heap itself supposed to be the max size of RAM?

Typically, no. The kernel heap is usually built on top of another allocator specifically for physical memory in page-sized units (sometimes called "physical memory manager" or "PMM"). The wiki has several pages on memory management.

If not, then how (besides in 16 bit real mode since I'm using GRUB as a bootloader) would I detect how much RAM my computer has. Or is it already stored somewhere I can pull it from.

You're using GRUB, which means you're using Multiboot or Multiboot2, which means GRUB has prepared a memory map for you. The Multiboot and Multiboot2 specifications explain how to get the memory map.

Is it something I should worry about now or later?

You can't allocate memory if you don't know where memory is.

[TheGameMaker90]

Is the kernel heap itself supposed to be the max size of RAM?

Typically, no. The kernel heap is usually built on top of another allocator specifically for physical memory in page-sized units (sometimes called "physical memory manager" or "PMM"). The wiki has several pages on memory management.

If not, then how (besides in 16 bit real mode since I'm using GRUB as a bootloader) would I detect how much RAM my computer has. Or is it already stored somewhere I can pull it from.

You're using GRUB, which means you're using Multiboot or Multiboot2, which means GRUB has prepared a memory map for you. The Multiboot and Multiboot2 specifications explain how to get the memory map.

Is it something I should worry about now or later?

You can't allocate memory if you don't know where memory is.

I've been on that page. As we speak I'm looking at "Pancake's" examples on heap implementations. Hopefully after enough studying I'll be able to do it the way I have in mind.

Okay, I'll have to look into that. In truth, GRUB is (hopefully) temporary, and I hope to write my own bootloader at some point. I've already created a separate project (literally called OtherOS) for the purpose of testing it. It works,m but has flaws and is missing a good chunk of things I know Grub has. Putting that aside, I will definitely check it out. Perhaps you could provide a link??

Haha, this is true. I kind of felt stupid asking this one, but I like to have a thorough understanding of things. Looks like I have a bit of homework to do...

Oh and one other thing: is it possible (in the off chance it's needed) to go beyond the address 0xFFFFFFFF? I know that's the max unsigned integer value, but perhaps virtual memory could be mapped beyond that point?

[Octocontrabass]

Perhaps you could provide a link??

Here is the Multiboot specification and here is the Multiboot2 specification. Without seeing your code, I can't tell you which one you're using right now.

Oh and one other thing: is it possible (in the off chance it's needed) to go beyond the address 0xFFFFFFFF? I know that's the max unsigned integer value, but perhaps virtual memory could be mapped beyond that point?

In protected mode, virtual addresses have 32 bits. In long mode, virtual addresses have 64 bits, but only 48 of those bits are usable on current CPUs.

In both modes, physical addresses may have up to 52 bits. In protected mode, you must use PAE to access all 52 bits.

[TheGameMaker90]

Perhaps you could provide a link??

Here is the Multiboot specification and here is the Multiboot2 specification. Without seeing your code, I can't tell you which one you're using right now.

Oh and one other thing: is it possible (in the off chance it's needed) to go beyond the address 0xFFFFFFFF? I know that's the max unsigned integer value, but perhaps virtual memory could be mapped beyond that point?

In protected mode, virtual addresses have 32 bits. In long mode, virtual addresses have 64 bits, but only 48 of those bits are usable on current CPUs.

In both modes, physical addresses may have up to 52 bits. In protected mode, you must use PAE to access all 52 bits.

Sort of makes sense, but how can you have 64 bits on a 32 bit OS?

[iansjack]

Have you read the Intel Programmer's Manual? If this is a project you are doing for your employer then you really need to do your homework. Learn how to program in C, in particular be sure you fully understand pointers, and learn about the hardware you are working with.

[TheGameMaker90]

Have you read the Intel Programmer's Manual? If this is a project you are doing for your employer then you really need to do your homework. Learn how to program in C, in particular be sure you fully understand pointers, and learn about the hardware you are working with.

No I haven't in detail. And employer?? I believe I mentioned "my company." I'm the CEO and this is (until it gets into a usable state) a personal project.