Temporary buffers without memory management implementation

[austanss]

I have continued development on my OS.
repo: https://github.com/microNET-OS/microCORE

One of the bugs I had to fix was a completely broken terminal. I mean, it could output text and in color, but it couldn't scroll. I fixed this issue by implementing an array shift to shift the buffer's array 80 characters to the left (out of the array). This does work, however it is slow. The function is as follows:
from Terminal.cpp

Code: Select all

void Terminal::shift()
{
    for (int i = 0; i < 80; i++)
    { 
        buffer[i] = vga_entry(0, 0);
    }

    /* DO() is a macro for a `for` loop */
    DO(VGA_WIDTH) // shift VGA_WIDTH (80) times
    {
        for (int i = 0; i < (VGA_HEIGHT * VGA_WIDTH); i++) {
            buffer[i] = buffer[i + 1]; // shift vga buffer to the left
        }
    }

    if (staticLogo)
    {
        for (int i = 0; i < (VGA_WIDTH * 15); i++)
        {
            buffer[i] = vga_entry(0, 0);
        }

        size_t rowtemp = row;
        size_t coltemp = column;
        setCursor(0, 0);
        Terminal::instance() << logo;
        setCursor(coltemp, rowtemp);
    }
    row--;
}

This tearing is extremely annoying when I'm trying to get debugging information for my broken interrupts. I attempted to implement a temporary buffer, and assign the temporary buffer to the new buffer when the function completes. Code:
from new Terminal.cpp

Code: Select all

void Terminal::shift()
{
    uint16_t *tbuffer = reinterpret_cast<uint16_t *>(0xe9500);

    for (int i = 0; i < 80; i++)
    {
        tbuffer[i] = vga_entry(0, 0);
    }

    DO(VGA_WIDTH) // shift VGA_WIDTH (80) times
    {
        for (int i = 0; i < (VGA_HEIGHT * VGA_WIDTH); i++) {
            tbuffer[i] = tbuffer[i + 1]; // shift vga buffer to the left
        }
    }

    if (staticLogo)
    {
        for (int i = 0; i < (VGA_WIDTH * 15); i++)
        {
            tbuffer[i] = vga_entry(0, 0);
        }

        size_t rowtemp = row;
        size_t coltemp = column;
        setCursor(0, 0);
        Terminal::instance() << logo;
        setCursor(coltemp, rowtemp);
    }

    *buffer = *tbuffer;

	row--;
}

This just doesn't work at all. The screen just doesn't update. Does this have to do with my lack of memory management? Or is it something else?

I appreciate any help.

[Octocontrabass]

I fixed this issue by implementing an array shift to shift the buffer's array 80 characters to the left (out of the array). This does work, however it is slow.

It's slow because you move each character one position to the left, and you do that 80 times. You're reading and writing the entire frame buffer 80 times to scroll one line! Reading and writing the VGA frame buffer is very slow, so the easiest way to speed it up is to avoid unnecessary reading and writing.

If you moved each character 80 positions to the left, you would only need to read and write the whole frame buffer once to scroll one line.

Be careful not to read or write outside the boundaries of your frame buffer.

I attempted to implement a temporary buffer, and assign the temporary buffer to the new buffer when the function completes.

If fixing the above issue isn't enough, you might look into having a (permanent) back buffer holding the same contents as the VGA buffer, so you can avoid reading the VGA buffer.

Does this have to do with my lack of memory management? Or is it something else?

I think you might be missing one or more items on this list.

[bloodline]

This just doesn't work at all. The screen just doesn't update. Does this have to do with my lack of memory management? Or is it something else?

I appreciate any help.

Hi rizxt, I'm really struggling to follow your code here. So I'm going to post my BIOS text buffer scrolling code, hopefully it might be helpful to you.

Code: Select all

void textbuffer_scroll_up_one_line(){
    uint16_t index = 0;
    
    for(size_t y=0;y<HEIGHT - 1;y++){
        for(size_t x=0;x<WIDTH;x++){
            text_buffer[index] = text_buffer[index+WIDTH];
            index++;
        }
    }

    // clear the bottom row
    for(size_t x=0;x<WIDTH;x++){
        text_buffer[index] = ' ';
        index++;
    }
    
}

I also think you might be making more work for yourself using C++ at this time. For all this low level stuff, plain C is easier to follow and you can (with appropriate function mangling attributes) call C functions from C++ at your higher levels.

[nullplan]

Why do people just love to write loops? Assuming text_buffer is an array of char, bloodline's code can be simplified down entirely like this:

Code: Select all

void textbuffer_scroll_one_line(void) {
  memmove(text_buffer, text_buffer + WIDTH, (HEIGHT - 1) * WIDTH);
  memset(text_buffer + (HEIGHT - 1) * WIDTH, ' ', WIDTH);
}

And if it isn't, then the first line remains almost valid (requiring a multiplication with the size of text_buffer's base type in the size argument), and the second line does have to be replaced with that loop. But at least the memmove is out of the way.

I also think you might be making more work for yourself using C++ at this time.

Erm, you know you can write exactly the same thing in C++, and exactly the same criticism is valid there, right? Besides, if your code is C and not C++, you should look into the difference between declaring a function with an empty argument list, and with a void argument. In C++ there is no difference, and indeed some people are pushing empty argument lists into style guides, but in C there is one, and not knowing it can break your code.

[crosssans]

Looking at your code for your `Terminal` class implementation, I can see that there are some issues - let me tell the important ones as I do a quick analysis of your code:

• Line 6: You seem to have declared an `enum` called `vga_color`, but this shouldn't be in the implementation file as it breaks implementation <-> interface isolation (see "why is it advantageous to separate function/class declarations from their implementations into .h and .cpp files?") - I suggest you to move this to the corresponding `Terminal.hpp` header so that `vga_color`
  • is accessible from any other header; this may be useful if you want to implement more complex classes and methods (such as a pretty kernel logger that output colorful messages!)
  • can be used in functions such as `put_entry_at` or `put_char` - you could make a small structure that contains two fields `background:4` & `foreground:4` (the `4` prefix indicates that these two fields combine only occupy 1 byte, see https://en.cppreference.com/w/cpp/language/bit_field)
• Line 36: Macros should be used only for very specific occasions in C++ as they may cause strange behaviour compilation-wise - It's better if they are avoided altogether (see this to know why it's considered as bad practice in C++ code) - but mandatory macros such as header guards are fine of course!
• Line 95: I don't understand the point of the `size` argument in this function, as it is not used anywhere in the function's code - so you could just remove this argument and the extra `write` function that ommits `size`
• Line 150: I think this function should be moved to the "string operations" category, as it seamingly does what an `itoa` function would do.. to some extent: the implementation is very odd and I can't seem to understand the logical reasoning behind it
• Line 180: The actual problematic function - I think it would be easier to just blindly copy from `0xb8000 + 1 * 80` to `0xb8000` & fill the last line with 80 words with `0x2000` (since x86 processors are little endian, the character & attributes value "fields" will be flipped when set to the framebuffer). You could develop optimized functions that use under their hood the `movs.` & `stos.` instructions combined with `rep` so that terminal scrolling is really fast (but that's an extreme solution of course )

As other have pointed out, you seem to lack some knowledge in C++/kernel development; maybe you missed to read Beginner Mistakes & Required Knoledge?

I recommend you to do simpler C++ projects first before jumping straight to kernel development, as it is a VERY hard task to complete especially if you're going to use an object oriented programming language that requires runtime support. In general, it's a better idea to use C instead as it is way easier to use when it comes to freestanding projects, since it requires little to almost no setup (it's just as easy as doing `extern ...` + `call ...` if you're going to not put extra arguments in your kernel's start function). This will not prevent you to implement higher-level components in C++ of course! - but I highly recommend you to read the Writing a kernel in C++ & C++ Kernel articles that explains what you are required to provide if you're still going to use this language after gaining experience. We'll be glad to help on the forum once you've matured your skills!

[...] If your code is C and not C++, you should look into the difference between declaring a function with an empty argument list, and with a void argument. In C++ there is no difference, and indeed some people are pushing empty argument lists into style guides, but in C there is one, and not knowing it can break your code.

I've found an article about this that explains the issue of not using `void` in functions that don't take any arguments in C. I'm quoting the important part just in case if this website goes down in the future so that people that may find this answer don't get frustrating of not being able to see what I was referring to:

[...]
A simplified explanation provided by K&R (pgs- 72-73) for the above stuff:
Furthermore, if a function declaration does not include arguments, as in `double atof();`, that too is taken to mean that nothing is to be assumed about the arguments of `atof`; all parameter checking is turned off. This special meaning of the empty argument list is intended to permit older C programs to compile with new compilers. But it's a bad idea to use it with new programs. If the function takes arguments, declare them; if it takes no arguments, use `void`.

[bloodline]

Why do people just love to write loops? Assuming text_buffer is an array of char, bloodline's code can be simplified down entirely like this:

Code: Select all

void textbuffer_scroll_one_line(void) {
  memmove(text_buffer, text_buffer + WIDTH, (HEIGHT - 1) * WIDTH);
  memset(text_buffer + (HEIGHT - 1) * WIDTH, ' ', WIDTH);
}

And if it isn't, then the first line remains almost valid (requiring a multiplication with the size of text_buffer's base type in the size argument), and the second line does have to be replaced with that loop. But at least the memmove is out of the way.

In my defence, the function I posted was written before I had written any memory manipulation functions, but you are right, it does make sense to use them, if you have them... I suspect the OP does not have them at this time.

Also I thought OP might benefit from seeing the more verbose algorithm.

I also think you might be making more work for yourself using C++ at this time.

Erm, you know you can write exactly the same thing in C++, and exactly the same criticism is valid there, right?

I wasn't criticising his choice of language, rather that using an OOP design pattern is going to take more effort to get right at this stage, and with very little gain. I think he might learn more easily working closer to the way the hardware actually works.

Besides, if your code is C and not C++, you should look into the difference between declaring a function with an empty argument list, and with a void argument. In C++ there is no difference, and indeed some people are pushing empty argument lists into style guides, but in C there is one, and not knowing it can break your code.

Well, the function I posted is actually the definition, the declaration is in the header file, but you do make a good point! I do sometimes mix up C and C++ language features.

[austanss]

I really don't understand the hate for inexperienced programmers writing OSes. My OS is just an ambitious hobby project. I am creating it purely to gain experience, and there is only a very slight possibility I actually use it. I can't just magically become an experienced programmer in a few months. I personally have a talent to learn and gain knowledge extremely quickly. It has only taken me 7 months for my C# knowledge to gain to an amount where it would be useful in the job world. I use C++ as I find it much more friendly than C. Most C libraries (std functions as well) are designed with 6 letter function names due to the technical limits of old C. C++ had and has no restriction, giving way to much better names. I refuse to actively work with C due to the cryptic namings (i.e. sprintf, strcmp, strstr. As well, my kernel main function is written in C++ and is externed as C, so no struggles are there. I'm creating an OS to gain knowledge, throwing myself into the deep end to learn how to swim. The main things I plan on gaining with this project is:

- Operable ASM (nasm) knowledge
- Further knowledge of C++
- Enhanced knowledge of low-level concepts

There is no way I can even imagine (other than driver development, which is significantly harder IMO due to you having to use other people's code) to gain the third option. The OS might be shitty, but at least I will have some experience. Please stop hating on intermediate programmers' attempts to make an OS. I understand the difficulties and frustrations that come with OS development. I know this isn't easy. I know I'm not making the next Linux. I know all of this. I just want help.

[bloodline]

I really don't understand the hate for inexperienced programmers writing OSes.

Be careful not to confuse criticism for hate. Criticism is a good thing, as it shows us where we need to develop.

I know I'm not making the next Linux. I know all of this. I just want help.

I have found everyone on this site to be extremely helpful and patient. But don't throw people's advice back in their faces.

To make an Operating System, you are going to need to read (and understand) a lot of example code, if you don't like C, you might be in trouble.

[zaval]

oh, well, "hate" again... something is telling me, bloodline, yet one advice of you to a newcomer, and you'll get your fanclub in form of a bunch of crazy screaming girls.

rizxt, the choice of language for your project is solely your privilege, I want to just make a tiny note, and believe me, it in no way comes from any bad intention - short names of functions of the C standard library, you don't love. you aware, you can create your internal kernel API with any style you want, right? the C world doesn't circle aroung C std library. no matter how important one may consider it, you, in the kernel, may not use it at all.

[foliagecanine]

To make an Operating System, you are going to need to read a lot of example code, if you don't like C, you might be in trouble.

I wouldn't say you have to like C, but you definitely need to be able to understand it, especially pointers (which C++ generally handles automatically) and structs. (I know both of these things are in C++, just used more in C)

Most C libraries (std functions as well) are designed with 6 letter function names due to the technical limits of old C. C++ had and has no restriction, giving way to much better names.

You [are] aware you can create your internal kernel API with any style you want, right?

That's exactly what I was about to say, Zaval (you beat me to it )
The thing with an OS is, you can DEFINE your own libraries. No one can force you to name your functions the same as the standard libc.
Note:{I'm not saying you (rizxt) should use C instead of C++, I'm just saying the sky's the limit with OSDev }
--------------------------------------
I think I managed to create a solution:

Code: Select all

for (int i = 0; i < (int)(VGA_HEIGHT * VGA_WIDTH); i++) {
        buffer[i] = i>((VGA_HEIGHT-1) * VGA_WIDTH) ? 0 : buffer[i+VGA_WIDTH];
}

This can easily be stretched out to multiple lines, but I tested it with your OS code and it worked.
(To use this, put this in place of lines 187-192 in Terminal.cpp)

[austanss]

To make an Operating System, you are going to need to read a lot of example code, if you don't like C, you might be in trouble.

I wouldn't say you have to like C, but you definitely need to be able to understand it, especially pointers (which C++ generally handles automatically) and structs. (I know both of these things are in C++, just used more in C)

Most C libraries (std functions as well) are designed with 6 letter function names due to the technical limits of old C. C++ had and has no restriction, giving way to much better names.

You [are] aware you can create your internal kernel API with any style you want, right?

That's exactly what I was about to say, Zaval (you beat me to it )
The thing with an OS is, you can DEFINE your own libraries. No one can force you to name your functions the same as the standard libc.
Note:{I'm not saying you (rizxt) should use C instead of C++, I'm just saying the sky's the limit with OSDev }
--------------------------------------
I think I managed to create a solution:

Code: Select all

for (int i = 0; i < (int)(VGA_HEIGHT * VGA_WIDTH); i++) {
        buffer[i] = i>((VGA_HEIGHT-1) * VGA_WIDTH) ? 0 : buffer[i+VGA_WIDTH];
}

This can easily be stretched out to multiple lines, but I tested it with your OS code and it worked.
(To use this, put this in place of lines 187-192 in Terminal.cpp)

Thanks foliage, that worked. The scroll is much less trippy and significantly readable.
However, although it fixes my problems, it doesn't answer my real question:

Is it possible to write data to arbitrary memory addresses, without a memory management system?

EDIT: Looking back I noticed that I didn't really state the question that well. I merely gave an example, but too much of it. I would like to clarify. That is my question [stated above].

[foliagecanine]

Is it possible to write data to arbitrary memory addresses, without a memory management system?

I don't quite understand exactly what you are asking.

If you are using a physical memory layout (no paging) then it is as simple as doing

Code: Select all

type_t * somewhere = (type_t *)0x12345678

where 0x12345678 is the address of the memory you want to access, and type_t is the type of data (i.e. int, char, etc.)

If you are using a virtual memory layout (paging), then it is a bit more complicated.
First, you must find a free virtual page in your page tables, then map that page to the physical memory address you want to access.
Then you can use the above code, making sure you are accessing the virtual address (which can be the same as the physical address, but isn't always).
This in itself is a form of memory management (Page Allocation).

Are you asking whether it's possible to access a physical address with paging on without mapping it in the page tables?
The answer to that question is no. If you have paging on, then you need to write a page allocation memory manager.

Technically, you can identity map everything, and therefore would need no memory manager, but that kinda defeats the point of having paging.

If you are asking about whether you can access arbitrary addresses without something like malloc, then the answer is yes.

[AndrewAPrice]

I wonder if what you mean is that you refer to the temporary memory buffer by an ID instead of a pointer, so the OS or some other system can move the object around or potentially deallocate it?

The API could look something like:

Code: Select all

// Writes memory to a buffer or does nothing of ID or index is invalid.
void WriteBufferMemory(int buffer_id, int index, int value);
// Reads memory from a buffer or returns 0 if ID or index is invalid.
int GetBufferMemory(int buffer_id, int index);

Or you could use locks, and while locked the object can't move, but it can when unlocked.

Code: Select all

// Returns a buffer, or nullptr if the ID is invalid.
Buffer* LockBuffer(int buffer_id);
// Unlocks a buffer. Does nothing if the buffer is already unlocked. The pointer returned by LockedBuffer is no longer safe.
void UnlockBuffer(int buffer_id);

The latter API would make it trivial to implement a readers-writers lock.

Referring to objects by an ID instead of memory address (only grabbing a memory address temporarily when you want to work with the object) makes it super simple to serialize the world. This is the basis of many entity component systems.

[bloodline]

Is it possible to write data to arbitrary memory addresses, without a memory management system?

Yes.

Code: Select all

// To write an unsigned byte to memory you use pointers
//
// Here, I write the byte value of 42 to the address 0xDFF180

#include <stdint.h>

//define a pointer to some specific address
uint8_t* unsigned_byte_pointer = 0xDFF180;

//to access the value at that address, derefrence the pointer.
*unsigned_byte_pointer = 42;

//or you can index it
unsigned_byte_pointer[0] = 42;

[austanss]

Thank you everyone for your advice! This information was very insightful! Thank you very much!