Can't read USTAR filesystem

[22OsC]

I tried to implement the USTAR filesystem driver for my os and for some reason is not working as I expected it to.

I read this osdev wiki page about it and seems very easy to implement. I tried mapping the tar file sections, adding the tar file to limine config, mapping the tar from limine, the same result.
Trying to read for signature or name will return random data from the memory.

It's because I use Stivale2 PMR?
If is, how I can fix it?

Code: Select all

struct file_header_t
{
    char name[100];
    char mode[8];
    char uid[8];
    char gid[8];
    char size[12];
    char mtime[12];
    char chksum[8];
    char typeflag[1];
    char link[100];
    char signature[6];
    char version[2];
    char owner[32];
    char group[32];
    char dev_maj[8];
    char dev_min[8];
    char prefix[155];
    char pad[12];
};

...

extern uint64_t _binary_initrd_tar_gz_start, _binary_initrd_tar_gz_end, _binary_initrd_tar_gz_size;

...

        for (uintptr_t k = (uintptr_t)&_binary_initrd_tar_gz_start; k < (uintptr_t)ALIGN_UP(&_binary_initrd_tar_gz_end, PAGE_SIZE); k += PAGE_SIZE)
        {
            MapMemory((void *)k, (void *)k - FVM_OFFSET, ReadWrite);
        }

        struct file_header *headers[32];
        uint64_t address = &_binary_initrd_tar_gz_start;
        uint64_t i;
        for (i = 0;; i++)
        {
            struct file_header *header = (struct file_header *)address;
            if (header->name[0] == '\0')
                break;
            uint64_t size = getsize(header->size);
            headers[i] = header;
            DEBUG("%d:%s", i, header->name);
            address += ((size / 512) + 1) * 512;
            if (size % 512)
                address += 512;
        }

...

The output for DEBUG("%d:%s...); is this:


I tried to do my own research but found nothing that could help me.

[BenLunt]

Code: Select all

        for (uintptr_t k = (uintptr_t)&_binary_initrd_tar_gz_start; k < (uintptr_t)ALIGN_UP(&_binary_initrd_tar_gz_end, PAGE_SIZE); k += PAGE_SIZE)

Just a quick look, may or may not be the issue, but you realize that

Code: Select all

k += PAGE_SIZE

actually adds PAGE_SIZE * sizeof(uintptr_t) to k, not PAGE_SIZE as you expect?

Again, may or may not be the issue.

Ben
- https://www.fysnet.net/osdesign_book_series.htm

[nullplan]

Code: Select all

        for (uintptr_t k = (uintptr_t)&_binary_initrd_tar_gz_start; k < (uintptr_t)ALIGN_UP(&_binary_initrd_tar_gz_end, PAGE_SIZE); k += PAGE_SIZE)

Just a quick look, may or may not be the issue, but you realize that

Code: Select all

k += PAGE_SIZE

actually adds PAGE_SIZE * sizeof(uintptr_t) to k, not PAGE_SIZE as you expect?

No, it doesn't. k is declared as a uintptr_t, which is an unsigned integer type of pointer width, not a pointer type, so arithmetic is integer arithmetic, not pointer arithmetic.

No, I would attempt to look at the start and end values to see if they make sense.

[BenLunt]

Code: Select all

        for (uintptr_t k = (uintptr_t)&_binary_initrd_tar_gz_start; k < (uintptr_t)ALIGN_UP(&_binary_initrd_tar_gz_end, PAGE_SIZE); k += PAGE_SIZE)

Just a quick look, may or may not be the issue, but you realize that

Code: Select all

k += PAGE_SIZE

actually adds PAGE_SIZE * sizeof(uintptr_t) to k, not PAGE_SIZE as you expect?

No, it doesn't. k is declared as a uintptr_t, which is an unsigned integer type of pointer width, not a pointer type, so arithmetic is integer arithmetic, not pointer arithmetic.

No, I would attempt to look at the start and end values to see if they make sense.

Unless I am mistaken, since uintptr_t is not declared anywhere in his/her code, I took it as

Code: Select all

unsigned int *

If I have the following code:

Code: Select all

  unsigned int *p = 0x00000000;
  
  p += 1;

  printf("  %p  \n", p);

The value printed will be sizeof(unsigned int).

Therefore,

Code: Select all

  p += PAGESIZE;

will indeed add sizeof(unsigned int) times PAGESIZE to the pointer value.

Am I wrong?

Ben

[klange]

Unless I am mistaken, since uintptr_t is not declared anywhere in his/her code, I took it as ...
Am I wrong?

uintptr_t is a standard-defined unsigned integer type, large enough that any void pointer can be cast back and forth to it.

[BenLunt]

Optionally defined standard. Keyword is optionally. :-)

Since there was no reference to <stdint.h> in the original post and a lot of us tend to define our own types when creating OS code, I wanted to make sure they understood that it may not be what they wanted it to be. Also, if I am not mistaken, and if my memory serves me correctly, that type can actually be of a different size that void*.

Anyway, after reading over my previous reply, it might have sounded rude. Sorry, it was not my intent.

Ben

[Octocontrabass]

The output for DEBUG("%d:%s...); is this:

Are you sure that output is correct? Even if you're reading random garbage instead of your tape archive, you should see either different random garbage each time or a null string from reading uninitialized memory (which is zero-filled in typical VMs).

I've seen some tutorials that have extremely wrong code for accessing variadic function arguments...

[22OsC]

Unless I am mistaken, since uintptr_t is not declared anywhere in his/her code, I took it as

Code: Select all

unsigned int *

my implementation for uintptr_t is

Code: Select all

typedef unsigned long int uintptr_t;

The output for DEBUG("%d:%s...); is this:

Are you sure that output is correct? Even if you're reading random garbage instead of your tape archive, you should see either different random garbage each time or a null string from reading uninitialized memory (which is zero-filled in typical VMs).

I've seen some tutorials that have extremely wrong code for accessing variadic function arguments...

I noticed that. not sure why the string output is the same. The strange thing is that the code has the same results with or without memory mapping and the page fault is occurred only after like 5 minutes of infinite loop.

My guess is that I need some specific compiler arguments for it? With PSF font objects are working without any issues. The memory is already mapped for some reason even that I applied the new PML4

[Octocontrabass]

my implementation for uintptr_t is

Code: Select all

typedef unsigned long int uintptr_t;

Your implementation for uintptr_t should be

Code: Select all

#include <stdint.h>

My guess is that I need some specific compiler arguments for it?

Probably not. Most likely there's a bug somewhere else in your code, but it's hard to say for sure without either debugging it some more or seeing the rest of your code.

Have you tried something like this?

Code: Select all

DEBUG("%d:%s", 1234, "test string");

[22OsC]

my implementation for uintptr_t is

Code: Select all

typedef unsigned long int uintptr_t;

Your implementation for uintptr_t should be

Code: Select all

#include <stdint.h>

My guess is that I need some specific compiler arguments for it?

Probably not. Most likely there's a bug somewhere else in your code, but it's hard to say for sure without either debugging it some more or seeing the rest of your code.

Have you tried something like this?

Code: Select all

DEBUG("%d:%s", 1234, "test string");

I discovered that it's not a memory mapping issue nor a wrong uintptr_t implementation. I can't load the tar at all. I put the code in the kernel main and the output is the same. With PSF fonts I don't have any issues, only with this tar archive. tar file size is 540,8 kb.
I use this command to create the tar file:

Code: Select all

tar czf initrd.tar.gz -C bin/ ./ --format=ustar

and then convert it to elf object using

Code: Select all

%.o: %.tar.gz
	objcopy -O elf64-x86-64 -I binary $< $@

And for

Code: Select all

DEBUG("%d:%s", 1234, "test string");

the output is

Code: Select all

1234:test string

[Octocontrabass]

Code: Select all

tar z

Compressing the archive compresses everything, including the headers. Decompress it first and then try reading the headers.

[22OsC]

Code: Select all

tar z

Compressing the archive compresses everything, including the headers. Decompress it first and then try reading the headers.

Oh, thanks for that! Now it works perfectly. I didn't know that it compresses the headers too.