OSDev.org

The Place to Start for Operating System Developers
http://f.osdev.org/

Can't read USTAR filesystem

http://f.osdev.org/viewtopic.php?t=56131

Page 1 of 1

Can't read USTAR filesystem

Posted: Thu Feb 17, 2022 9:33 am
by 22OsC
I tried to implement the USTAR filesystem driver for my os and for some reason is not working as I expected it to.

I read this osdev wiki page about it and seems very easy to implement. I tried mapping the tar file sections, adding the tar file to limine config, mapping the tar from limine, the same result.
Trying to read for signature or name will return random data from the memory.

It's because I use Stivale2 PMR?
If is, how I can fix it?

Code: Select all

struct file_header_t
{
    char name[100];
    char mode[8];
    char uid[8];
    char gid[8];
    char size[12];
    char mtime[12];
    char chksum[8];
    char typeflag[1];
    char link[100];
    char signature[6];
    char version[2];
    char owner[32];
    char group[32];
    char dev_maj[8];
    char dev_min[8];
    char prefix[155];
    char pad[12];
};

...

extern uint64_t _binary_initrd_tar_gz_start, _binary_initrd_tar_gz_end, _binary_initrd_tar_gz_size;

...

        for (uintptr_t k = (uintptr_t)&_binary_initrd_tar_gz_start; k < (uintptr_t)ALIGN_UP(&_binary_initrd_tar_gz_end, PAGE_SIZE); k += PAGE_SIZE)
        {
            MapMemory((void *)k, (void *)k - FVM_OFFSET, ReadWrite);
        }

        struct file_header *headers[32];
        uint64_t address = &_binary_initrd_tar_gz_start;
        uint64_t i;
        for (i = 0;; i++)
        {
            struct file_header *header = (struct file_header *)address;
            if (header->name[0] == '\0')
                break;
            uint64_t size = getsize(header->size);
            headers[i] = header;
            DEBUG("%d:%s", i, header->name);
            address += ((size / 512) + 1) * 512;
            if (size % 512)
                address += 512;
        }

...
The output for DEBUG("%d:%s...); is this:
Image

I tried to do my own research but found nothing that could help me.

Re: Can't read USTAR filesystem

Posted: Thu Feb 17, 2022 2:53 pm
by BenLunt
22OsC wrote:

Code: Select all

        for (uintptr_t k = (uintptr_t)&_binary_initrd_tar_gz_start; k < (uintptr_t)ALIGN_UP(&_binary_initrd_tar_gz_end, PAGE_SIZE); k += PAGE_SIZE)
Just a quick look, may or may not be the issue, but you realize that

Code: Select all

k += PAGE_SIZE
actually adds PAGE_SIZE * sizeof(uintptr_t) to k, not PAGE_SIZE as you expect?

Again, may or may not be the issue.

Ben
- https://www.fysnet.net/osdesign_book_series.htm

Re: Can't read USTAR filesystem

Posted: Thu Feb 17, 2022 9:38 pm
by nullplan
BenLunt wrote:
22OsC wrote:

Code: Select all

        for (uintptr_t k = (uintptr_t)&_binary_initrd_tar_gz_start; k < (uintptr_t)ALIGN_UP(&_binary_initrd_tar_gz_end, PAGE_SIZE); k += PAGE_SIZE)
Just a quick look, may or may not be the issue, but you realize that

Code: Select all

k += PAGE_SIZE
actually adds PAGE_SIZE * sizeof(uintptr_t) to k, not PAGE_SIZE as you expect?
No, it doesn't. k is declared as a uintptr_t, which is an unsigned integer type of pointer width, not a pointer type, so arithmetic is integer arithmetic, not pointer arithmetic.

No, I would attempt to look at the start and end values to see if they make sense.

Re: Can't read USTAR filesystem

Posted: Thu Feb 17, 2022 11:03 pm
by BenLunt
nullplan wrote:
BenLunt wrote:
22OsC wrote:

Code: Select all

        for (uintptr_t k = (uintptr_t)&_binary_initrd_tar_gz_start; k < (uintptr_t)ALIGN_UP(&_binary_initrd_tar_gz_end, PAGE_SIZE); k += PAGE_SIZE)
Just a quick look, may or may not be the issue, but you realize that

Code: Select all

k += PAGE_SIZE
actually adds PAGE_SIZE * sizeof(uintptr_t) to k, not PAGE_SIZE as you expect?
No, it doesn't. k is declared as a uintptr_t, which is an unsigned integer type of pointer width, not a pointer type, so arithmetic is integer arithmetic, not pointer arithmetic.

No, I would attempt to look at the start and end values to see if they make sense.
Unless I am mistaken, since uintptr_t is not declared anywhere in his/her code, I took it as

Code: Select all

unsigned int *
If I have the following code:

Code: Select all

  unsigned int *p = 0x00000000;
  
  p += 1;

  printf("  %p  \n", p);
The value printed will be sizeof(unsigned int).

Therefore,

Code: Select all

  p += PAGESIZE;
will indeed add sizeof(unsigned int) times PAGESIZE to the pointer value.

Am I wrong?

Ben

Re: Can't read USTAR filesystem

Posted: Thu Feb 17, 2022 11:28 pm
by klange
BenLunt wrote:Unless I am mistaken, since uintptr_t is not declared anywhere in his/her code, I took it as ...
Am I wrong?
uintptr_t is a standard-defined unsigned integer type, large enough that any void pointer can be cast back and forth to it.

Re: Can't read USTAR filesystem

Posted: Fri Feb 18, 2022 7:26 am
by BenLunt
Optionally defined standard. Keyword is optionally. :-)

Since there was no reference to <stdint.h> in the original post and a lot of us tend to define our own types when creating OS code, I wanted to make sure they understood that it may not be what they wanted it to be. Also, if I am not mistaken, and if my memory serves me correctly, that type can actually be of a different size that void*.

Anyway, after reading over my previous reply, it might have sounded rude. Sorry, it was not my intent.

Ben

Re: Can't read USTAR filesystem

Posted: Fri Feb 18, 2022 4:32 pm
by Octocontrabass
22OsC wrote:The output for DEBUG("%d:%s...); is this:
Are you sure that output is correct? Even if you're reading random garbage instead of your tape archive, you should see either different random garbage each time or a null string from reading uninitialized memory (which is zero-filled in typical VMs).

I've seen some tutorials that have extremely wrong code for accessing variadic function arguments...

Re: Can't read USTAR filesystem

Posted: Sun Feb 20, 2022 10:15 am
by 22OsC
BenLunt wrote: Unless I am mistaken, since uintptr_t is not declared anywhere in his/her code, I took it as

Code: Select all

unsigned int *
my implementation for uintptr_t is

Code: Select all

typedef unsigned long int uintptr_t;
Octocontrabass wrote:
22OsC wrote:The output for DEBUG("%d:%s...); is this:
Are you sure that output is correct? Even if you're reading random garbage instead of your tape archive, you should see either different random garbage each time or a null string from reading uninitialized memory (which is zero-filled in typical VMs).

I've seen some tutorials that have extremely wrong code for accessing variadic function arguments...
I noticed that. not sure why the string output is the same. The strange thing is that the code has the same results with or without memory mapping and the page fault is occurred only after like 5 minutes of infinite loop.

My guess is that I need some specific compiler arguments for it? With PSF font objects are working without any issues. The memory is already mapped for some reason even that I applied the new PML4

Re: Can't read USTAR filesystem

Posted: Sun Feb 20, 2022 5:10 pm
by Octocontrabass
22OsC wrote:my implementation for uintptr_t is

Code: Select all

typedef unsigned long int uintptr_t;
Your implementation for uintptr_t should be

Code: Select all

#include <stdint.h>
22OsC wrote:My guess is that I need some specific compiler arguments for it?
Probably not. Most likely there's a bug somewhere else in your code, but it's hard to say for sure without either debugging it some more or seeing the rest of your code.

Have you tried something like this?

Code: Select all

DEBUG("%d:%s", 1234, "test string");

Re: Can't read USTAR filesystem

Posted: Sun Feb 20, 2022 6:24 pm
by 22OsC
Octocontrabass wrote:
22OsC wrote:my implementation for uintptr_t is

Code: Select all

typedef unsigned long int uintptr_t;
Your implementation for uintptr_t should be

Code: Select all

#include <stdint.h>
22OsC wrote:My guess is that I need some specific compiler arguments for it?
Probably not. Most likely there's a bug somewhere else in your code, but it's hard to say for sure without either debugging it some more or seeing the rest of your code.

Have you tried something like this?

Code: Select all

DEBUG("%d:%s", 1234, "test string");
I discovered that it's not a memory mapping issue nor a wrong uintptr_t implementation. I can't load the tar at all. I put the code in the kernel main and the output is the same. With PSF fonts I don't have any issues, only with this tar archive. tar file size is 540,8 kb.
I use this command to create the tar file:

Code: Select all

tar czf initrd.tar.gz -C bin/ ./ --format=ustar
and then convert it to elf object using

Code: Select all

%.o: %.tar.gz
	objcopy -O elf64-x86-64 -I binary $< $@
And for

Code: Select all

DEBUG("%d:%s", 1234, "test string");
the output is

Code: Select all

1234:test string

Re: Can't read USTAR filesystem

Posted: Sun Feb 20, 2022 6:48 pm
by Octocontrabass
22OsC wrote:

Code: Select all

tar z
Compressing the archive compresses everything, including the headers. Decompress it first and then try reading the headers.

Re: Can't read USTAR filesystem

Posted: Mon Feb 21, 2022 8:50 am
by 22OsC
Octocontrabass wrote:
22OsC wrote:

Code: Select all

tar z
Compressing the archive compresses everything, including the headers. Decompress it first and then try reading the headers.
Oh, thanks for that! Now it works perfectly. I didn't know that it compresses the headers too.
All times are UTC-06:00
Page 1 of 1

Powered by phpBB® Forum Software © phpBB Limited