OSDev.org

Posted: **Sun Jan 19, 2020 9:51 am**

I am building a real mode OS as a hobby project. As my first OS project I wanted to begin at the beginning and so the choice to build a Real Mode Operating System. Having said that, I am not following any OS in particular, but have ideas from many sources. I think more Real Mode as a scratch pad sort of mode where given the hardware, software can be anything with very little intervention by the processor.

Coming to the problem, I am building a

Code: Select all

alloc()

like system call which the user programs can call when they want more memory. I understand a brk() like system call would be more simple to implement but then there would anyways be a

Code: Select all

malloc()

anyways.

A user program in this OS would be a simple COM file, with no header and with statically allocated data and code together. However the dynamically allocated memory will reside after the Code and Stack Segments.

: mem_2.png (30.06 KiB) Viewed 2242 times

The A, AE, B, BE, C, CE are addresses in the memory model. Note that such an memory arrangement is needed to conform with the SMALL memory model (because most compilers assume such a model).

Now that there is some context, I plan to implement the dynamic memory allocation via a table of the sort below.

: mem_1.png (64.61 KiB) Viewed 2242 times

The 'Offset' column is the start of allotted memory after the 'C' address. When more memory is requested OS will allocate memory by either finding a large enough block from the table which is currently free or allocate more memory after the end of the last 'Offset + Size' in the table.

Consequently, a free() system call will free one or more a blocks when done.

My question is:

1. Is such a model of dynamic memory allocation is feasible? And why such a model is not used?
2. Should such system calls 'alloc' and 'free' are better to of in a library rather than as a system call?

Posted: **Sun Jan 19, 2020 1:28 pm**

I have a bunch of questions about this, since I don't think you've given us enough information about things, and honestly I am not sure if there isn't some confusion of ideas in all this (and if there is, whether it is on your part or mine isn't clear, either).

To start with, you might want to say more about things like your toolchain (compiler, assembler, etc - from what you've said I take it you're writing the OS in C, and presumably you aren't using GCC since that doesn't play well with real mode), your intended OS design (monolithic, microkernel, hybrid, executive monitor, what have you) and most specifically, how you mean to vector system calls (e.g., using interrupts a la MS-DOS, as simple function calls, through message passing, or something else).

I will say that, if you are planning for user processes to always use small model (i.e., one segment for code and a second for data and stack, CS = x and DS = SS = y), then there is no reason for a system call to be involved at all. Real mode segments don't have selectable size boundaries (unlike segments in either 16-bit p-mode or 32-bit p-mode), meaning that you effectively are giving the entire 64KiB data and stack segment to the process regardless, unless you are intending to overlap segments for different processes for some reason (which isn't entirely unreasonable, though you'd usually do things like that with Medium or Compact mode instead, in effect making the shared data section being the equivalent of a Fortran-style COMMON block). Thus, the in-process memory management can be a local heap using malloc() and free() as usual.

Mind you, your diagram describes tiny model, not small, so there's a definite question there, too. While a raw binary such as a .COM file usually would have the initialized data in the same segment as the code (since it's loaded as a single blob), in Small Model the uninitialized and dynamically allocated data memory - the .BSS area, the stack, and the heap - would be in the DS/SS segment.

However, this is also why programs in the form of .COM files were generally tiny model - for larger data areas, you'd need to explicitly manipulate the segments when addressing the initialized data so that you offset them from CS, which is easy enough in assembly but a bit awkward in C without an explicit extension and/or some shuffling by the compiler. This is part of why Microsoft introduced the .EXE format (rather late in the development of MS-DOS), and also part of why almost every other x86 OS has used more complex Executable Formats rather than raw binaries (there are a lot of other reasons to do so though, starting with being able to link separately compiled source files into a final executable without having very different formats for Object Files and executable files).

Further, most real mode compilers let you select a model at compile time.

However, I can only guess at whether I understood you correctly on this. Could you please elaborate?

Posted: **Sun Jan 19, 2020 2:20 pm**

Schol-R-LEA wrote:I have a bunch of questions about this, since I don't think you've given us enough information about things, and honestly I am not sure if there isn't some confusion of ideas in all this (and if there is, whether it is on your part or mine isn't clear, either).

To start with, you might want to say more about things like your toolchain (compiler, assembler, etc - from what you've said I take it you're writing the OS in C, and presumably you aren't using GCC since that doesn't play well with real mode), your intended OS design (monolithic, microkernel, hybrid, executive monitor, what have you) and most specifically, how you mean to vector system calls (e.g., using interrupts a la MS-DOS, as simple function calls, through message passing, or something else).

I am build the kernel and the drivers in assembly (1. To gain experience and 2. To avoid compilers as most do not support real mode well enough). However I will setup the process memory model to that user programs can be written in compilers that support real mode. System calls are handled via interrupts - There is a system call dispatcher routine will setup some registers and call the appropriate routine based on a call table)

Schol-R-LEA wrote:I will say that, if you are planning for user processes to always use small model (i.e., one segment for code and a second for data and stack, CS = x and DS = SS = y), then there is no reason for a system call to be involved at all. Real mode segments don't have selectable size boundaries (unlike segments in either 16-bit p-mode or 32-bit p-mode), meaning that you effectively are giving the entire 64KiB data and stack segment to the process regardless, unless you are intending to overlap segments for different processes for some reason (which isn't entirely unreasonable, though you'd usually do things like that with Medium or Compact mode instead, in effect making the shared data section being the equivalent of a Fortran-style COMMON block). Thus, the in-process memory management can be a local heap using malloc() and free() as usual.

Mind you, your diagram describes tiny model, not small, so there's a definite question there, too. While a raw binary such as a .COM file usually would have the initialized data in the same segment as the code (since it's loaded as a single blob), in Small Model the uninitialized and dynamically allocated data memory - the .BSS area, the stack, and the heap - would be in the DS/SS segment.

You are right, I am sorry, user programs will have 'TINY' model. Rest of the OS is build as modules (separate binary files, like SYS in DOS). These are loaded by the OS into separate segments when OS is loading.

Schol-R-LEA wrote:However, this is also why programs in the form of .COM files were generally tiny model - for larger data areas, you'd need to explicitly manipulate the segments when addressing the initialized data so that you offset them from CS, which is easy enough in assembly but a bit awkward in C without an explicit extension and/or some shuffling by the compiler. This is part of why Microsoft introduced the .EXE format (rather late in the development of MS-DOS), and also part of why almost every other x86 OS has used more complex Executable Formats rather than raw binaries (there are a lot of other reasons to do so though, starting with being able to link separately compiled source files into a final executable without having very different formats for Object Files and executable files).

After the basic OS is ready, I do want to have a linkable executable format.

Schol-R-LEA wrote:Further, most real mode compilers let you select a model at compile time.

The kernel memory model is rather strange (CS = DS <> SS). I had tried BSS, SmallerC, sourcery etc, none really worked for me. I finally found openwatcom that supported this model. But I realised that it will be lot less pain if wrote the drivers and kernel in assembly. But will setup TINY memory model for user processes, so that they can be written in C.

Schol-R-LEA wrote:However, I can only guess at whether I understood you correctly on this. Could you please elaborate?

Posted: **Sun Jan 19, 2020 6:04 pm**

my first operating system was also in real mode but as it was also monotask. I did not bother to create a memory allocation system, the system had the memory space from 0h to 4FFFFh, the program had the space from 50000h to 8FFFFh and the queue from 90000h to 9FFFFh but as soon as I needed to multitask, allocating / freeing memory was the first thing to do

1. Is such a model of dynamic memory allocation is feasible? And why such a model is not used?

I have absolutely no idea if such a model exists but if it works then why not use it? I use another way but I'm not sure it works better

2. Should such system calls 'alloc' and 'free' are better to of in a library rather than as a system call?

system call, I consider that the libraries should only concern optional things

Posted: **Mon Jan 20, 2020 2:58 pm**

arjob wrote:As my first OS project I wanted to begin at the beginning and so the choice to build a Real Mode Operating System. [...] I think more Real Mode as a scratch pad sort of mode where given the hardware, software can be anything with very little intervention by the processor.

Real mode is pretty significantly different from the other processor modes, and also significantly different from other processors in general. It's fine if you're just doing it for fun, but keep in mind that you'll be solving a lot of problems that aren't applicable elsewhere.

arjob wrote:1. Is such a model of dynamic memory allocation is feasible? And why such a model is not used?

Your model is already malloc(), so of course it's feasible. Off the top of my head, I can think of a few reasons why OSes typically don't work this way. The first is complexity: it takes less code to manage memory when your model can use the CPU's memory management hardware. The second is security: simpler code means a lower attack surface for exploits, and an easier time locating and mitigating them. The third is flexibility, which brings me to your second question.

arjob wrote:2. Should such system calls 'alloc' and 'free' are better to of in a library rather than as a system call?

It's better to put functions like malloc() in a library to allow software to replace it with its own memory management. Some software has very particular requirements on memory management, and needs to provide specialized or optimized memory management routines. Non-C language runtimes are common users of alternative memory management algorithms.

Posted: **Mon Jan 20, 2020 6:15 pm**

arjob wrote: My question is:

1. Is such a model of dynamic memory allocation is feasible? And why such a model is not used?
2. Should such system calls 'alloc' and 'free' are better to of in a library rather than as a system call?

Generally, the OS reserves some part of memory for itself, and gives the rest of memory to the program and its libraries, and then the language runtime implements something like malloc() within the space the OS handed to the program (the OS also generally has some sort of malloc() implementation for its own internal use in its reserved area).

On modern systems with memory management hardware, the above statement means that the OS creates a virtual address space for each process in which only the program, its libraries, and the OS exist (other programs exist elsewhere in *physical* memory, but the OS lays out virtual memory so that the currently running program thinks it has the computer all to itself). The OS has to have a method of selecting unused physical pages to be assigned to a virtual address space, but it doesn't generally provide anything like malloc() to programs.

On older systems that used real mode (or the equivalent on other architectures), the system was generally single-tasking, so except for things like DOS TSRs (which, depending on what they did, could either be considered analogous to drivers on a modern system, or to userspace libraries), the program generally *did* have the system all to itself, so, once again, the OS didn't really provide any malloc() type functionality to programs.

There were some exceptions, where a real-mode OS provided "multitasking*", and in that case the OS would need to provide some sort of malloc() type functionality to programs.

*Note that on a real-mode system, any program can usurp the position of the OS and take over the computer for itself, so that even if the OS provides facilities for multitasking, the system only remains a multitasking system if all the programs play nice.

Posted: **Wed Jan 22, 2020 8:02 am**

Remember that you can use 4GB of RAM in real mode, so you need to design your OS to support that.

First enable Unreal Mode (see the attached file; you still need to enable line A20):
UR86.ASM

Code: Select all

;Print it to study... it works in the 386DX after commenting out the stack code
;and after using a simple RET instead of INT 20h or INT 21h to terminate the program.


;;Protected mode enabling/disabling sample for DOS.
;;On termination, it leaves the
;;processor into Unreal Mode.
;;
;; Characteristics:
;;
;;   DOS-Free Ready? [???]
;;;

org 100h    ;DOS Load Base
bits 16     ;Our program is 16-bit


cli

;Save return address to DOS shell:
;;
pop gs


push cs
pop ax
mov ds,ax
mov es,ax
mov fs,ax


mov ss,ax
mov esp,Local_Stack_END

;Put the return address to DOS shell into the new stack:
;;
 push gs





sti


 mov dx,strAppName
 mov ah,9
 int 0x21


 ;;Prepare our Protected-Mode-Ready GDTR and selectors
 ;;to DOS/Real Mode addressing for load and set easy
 ;;32-bit base addresses selectors for using like in
 ;;DOS.
 ;;;
   call ConfigGDTselectors




 cli               ;Disable our interrupts...


;;;INIT: Here we will finally enter full 32-bit Protected Mode
;;;INIT: Here we will finally enter full 32-bit Protected Mode
;;;INIT: Here we will finally enter full 32-bit Protected Mode

  ;;Save our Real Mode Segment Registers:
  ;;Save our Real Mode Segment Registers:
  ;;Save our Real Mode Segment Registers:
  ;;;
   push ds
   push es
   push fs
   push gs
   mov bp,ss

   mov word[cs:_rm_final_far_seg],cs




  ;;Load our GDT:
  ;;Load our GDT:
  ;;Load our GDT:
  ;;;
   lgdt [GDT]



  ;;Enable PE Bit and Thus, Initial Protected Mde:
  ;;Enable PE Bit and Thus, Initial Protected Mde:
  ;;Enable PE Bit and Thus, Initial Protected Mde:
  ;;;
   mov eax,cr0
   inc ax
   mov cr0,eax


  ;;Enter to Full Basic Protected Mode using a DOS Base-addressed Code Selector:
  ;;Enter to Full Basic Protected Mode using a DOS Base-addressed Code Selector:
  ;;Enter to Full Basic Protected Mode using a DOS Base-addressed Code Selector:
  ;;;
   jmp SELCod32DOS:bits32


  bits 32  ;Now we use 32-bit instructions
  bits32:  ;This is our label for 32-bit PM Entry Point



   ;;;INIT: Protected Mode CPU Configuration Examples
   ;;;INIT: Protected Mode CPU Configuration Examples
   ;;;INIT: Protected Mode CPU Configuration Examples
   ;;;INIT: Protected Mode CPU Configuration Examples
   ;;;INIT: Protected Mode CPU Configuration Examples
   ;;;INIT: Protected Mode CPU Configuration Examples
   ;;;INIT: Protected Mode CPU Configuration Examples
   ;;;INIT: Protected Mode CPU Configuration Examples
   ;;;INIT: Protected Mode CPU Configuration Examples
   ;;;INIT: Protected Mode CPU Configuration Examples
     ;;This is a sample of how to load 32-bit
     ;;data segments with a DOS Base-addressed
     ;;Data Selector:
     ;;;
      mov ax,SELDat32
      mov ds,ax
      mov es,ax
      mov fs,ax
      mov gs,ax
      mov ss,ax




     ;;Now we jump to a 16-bit Protected Mode code
     ;;area, for preparation to return to Real Mode,
     ;;using a selector base Address according to where
     ;;DOS put us:
     ;;;
      jmp SELCod16DOS:bits16


  bits 16  ;Now we use 32-bit instructions
  bits16:  ;This is our label for 32-bit PM Entry Point

   ;;;END:  Protected Mode CPU Configuration Examples
   ;;;END:  Protected Mode CPU Configuration Examples
   ;;;END:  Protected Mode CPU Configuration Examples
   ;;;END:  Protected Mode CPU Configuration Examples
   ;;;END:  Protected Mode CPU Configuration Examples
   ;;;END:  Protected Mode CPU Configuration Examples
   ;;;END:  Protected Mode CPU Configuration Examples
   ;;;END:  Protected Mode CPU Configuration Examples
   ;;;END:  Protected Mode CPU Configuration Examples
   ;;;END:  Protected Mode CPU Configuration Examples



   ;;;INIT: At this stage we must not make any further calls or stack use
   ;;;INIT: At this stage we must not make any further calls or stack use
   ;;;INIT: At this stage we must not make any further calls or stack use
   ;;;INIT: At this stage we must not make any further calls or stack use
   ;;;INIT: At this stage we must not make any further calls or stack use
   ;;;INIT: At this stage we must not make any further calls or stack use
   ;;;INIT: At this stage we must not make any further calls or stack use
   ;;;INIT: At this stage we must not make any further calls or stack use
   ;;;INIT: At this stage we must not make any further calls or stack use
   ;;;INIT: At this stage we must not make any further calls or stack use

     ;;Here we will leave our code segments with
     ;;32-bit addressable data segments. Without
     ;;it, we would return to Real Mode and not
     ;;to Unreal Mode if we used "SELDat16" or
     ;;"SELDat16DOS" instead of SELDat32:
     ;;;
;      mov ax,SELDat32
;      mov ds,ax
;      mov es,ax
;      mov fs,ax
;      mov gs,ax

     ;Set the stack segment as 16-bit data:
     ;;
      mov ax,SELDat16
      mov ss,ax


  ;;Disable PE Bit and thus, go back to (Un)real Mode:
  ;;Disable PE Bit and thus, go back to (Un)real Mode:
  ;;Disable PE Bit and thus, go back to (Un)real Mode:
  ;;;
   mov eax,cr0
   dec ax
   mov cr0,eax





   ;;Attemtp to further flush the Protected Mode
   ;;Instruction Queue to stabilize (Un)real Mode:
   ;;;
   ;jmp segment16:offset16
    db 0xEA
                      dw exit_   ;far offset
    _rm_final_far_seg dw 0       ;far segment
    exit_:


   ;;;END:  At this stage we must not make any further calls or stack use
   ;;;END:  At this stage we must not make any further calls or stack use
   ;;;END:  At this stage we must not make any further calls or stack use
   ;;;END:  At this stage we must not make any further calls or stack use
   ;;;END:  At this stage we must not make any further calls or stack use
   ;;;END:  At this stage we must not make any further calls or stack use
   ;;;END:  At this stage we must not make any further calls or stack use
   ;;;END:  At this stage we must not make any further calls or stack use
   ;;;END:  At this stage we must not make any further calls or stack use
   ;;;END:  At this stage we must not make any further calls or stack use



  ;;Restore our Real Mode Segment Registers:
  ;;Restore our Real Mode Segment Registers:
  ;;Restore our Real Mode Segment Registers:
  ;;;
   mov ss,bp
   pop gs
   pop fs
   pop es
   pop ds



 sti          ;Enable our interrupts...






 mov dx,rmpmurmsg
 mov ah,9
 int 0x21




;Return to DOS.
;;
 ret
; mov ax,4C00h
; int 21h


;;;END:  Here we will finally enter full 32-bit Protected Mode
;;;END:  Here we will finally enter full 32-bit Protected Mode
;;;END:  Here we will finally enter full 32-bit Protected Mode









;;INIT: Data/Constants definitions and Arch. Structures
;;INIT: Data/Constants definitions and Arch. Structures
;;INIT: Data/Constants definitions and Arch. Structures
   ;INIT: GDT
   ;INIT: GDT
   ;INIT: GDT
     GDT:
      SELNull equ 0    ;As you may see, we have used
        GDT_size:      ;the Null Selector as the GDT
          dw GDTsize   ;pointer to load with LGDT.
        GDT_actualPtr: ;
          dd GDT       ;You can see that the WORD at
          dw 0x0000    ;the end of these 8 bytes are
                       ;2 padding bytes with 0x0000.

      SELCod32DOS equ 8
      SelCod32DOS:
        dw 0FFFFh       ; bits 0-15 length (Bytes 0-1)
        dw 00000h       ; bits 0-15 base addr (Bytes 2-3)
        db 0            ; bits 16-23 base addr (Byte 4)
        db 10011010b    ; bits 0-3 Type, 4 DT, 5-6 DPL & 7 P (Byte 5)
        db 11001111b    ; bits 16-19 length into 0-3, 6 D & 7 G (Byte 6)
        db 0            ; bits 24-31 base addr (Byte 7)

      SELDat32 equ 16
      SelDat32:
        dw 0FFFFh       ; bits 0-15 length (Bytes 0-1)
        dw 00000h       ; bits 0-15 base addr (Bytes 2-3)
        db 0            ; bits 16-23 base addr (Byte 4)
        db 10010010b    ; bits 0-3 Type, 4 DT, 5-6 DPL & 7 P (Byte 5)
        db 11001111b    ; bits 16-19 length into 0-3, 6 D & 7 G (Byte 6)
        db 0            ; bits 24-31 base addr (Byte 7)

      SELCod16DOS equ 24
      SelCod16DOS:
        dw 0FFFFh       ; bits 0-15 length (Bytes 0-1)
        dw 00000h       ; bits 0-15 base addr (Bytes 2-3)
        db 0            ; bits 16-23 base addr (Byte 4)
        db 10011010b    ; bits 0-3 Type, 4 DT, 5-6 DPL & 7 P (Byte 5)
        db 00000000b    ; bits 16-19 length into 0-3, 6 D & 7 G (Byte 6)
        db 0            ; bits 24-31 base addr (Byte 7)

      SELDat16 equ 32
      SelDat16:
        dw 0FFFFh       ; bits 0-15 length (Bytes 0-1)
        dw 00000h       ; bits 0-15 base addr (Bytes 2-3)
        db 0            ; bits 16-23 base addr (Byte 4)
        db 10010010b    ; bits 0-3 Type, 4 DT, 5-6 DPL & 7 P (Byte 5)
        db 00000000b    ; bits 16-19 length into 0-3, 6 D & 7 G (Byte 6)
        db 0            ; bits 24-31 base addr (Byte 7)

      GDT_end:

      GDTsize equ (GDT_end-GDT)-1
   ;END:  GDT
   ;END:  GDT
   ;END:  GDT
;;END:  Data/Constants definitions and Arch. Structures
;;END:  Data/Constants definitions and Arch. Structures
;;END:  Data/Constants definitions and Arch. Structures

























ConfigGDTselectors:
 push eax

 xor eax,eax
 mov ax,ds
 shl eax,4

 add eax,GDT
 mov [GDT_actualPtr],eax



 xor eax,eax
 mov ax,cs
 shl eax,4

 mov [SelCod32DOS+2],ax
 mov [SelCod16DOS+2],ax

 shr eax,16


 mov [SelCod32DOS+4],al
 mov [SelCod16DOS+4],al


 mov [SelCod32DOS+7],ah
 mov [SelCod16DOS+7],ah





 pop eax
ret
























align 16
times 128 db 0
Local_Stack_END:



strAppName db "UR86_1.COM: $"

rmpmurmsg db "Test Message - Enabling 386-grade Unreal Mode...",0x0D,0x0A,'$'

Posted: **Thu Jan 23, 2020 10:17 am**

You know, ~, it would help if you actually read these threads before adding your two cents.

OSDev.org

Memory Allocation on a Real Mode Operating System

Memory Allocation on a Real Mode Operating System

Re: Memory Allocation on a Real Mode Operating System

Re: Memory Allocation on a Real Mode Operating System

Re: Memory Allocation on a Real Mode Operating System

Re: Memory Allocation on a Real Mode Operating System

Re: Memory Allocation on a Real Mode Operating System

Re: Memory Allocation on a Real Mode Operating System

Re: Memory Allocation on a Real Mode Operating System