protected mode or drivers first?

[JohnpaulTH]

Hi All,
I decided to make a unikernel or library operating system after spending a long time developing applications for real mode that used BIOS interrupts.
(An example of my absolutely flawless top-notch real-mode programs can be found here:
https://gitlab.com/jhumphrey/tty-pre-alpha)
I came to myself, repented of my misdeeds, and decided that I would experience the joys of writing my own drivers instead of having the BIOS do it. Additionally, I wanted to have my programs support multiple alphabets and "successfully" made a driver to handle 8x8 text in mode 11h. I know that it is possible to not use BIOS interrupts, and write your own drivers in real mode, but I figure that eventually, I might want to get into protected mode.
First, because I don't know what kinda programs I am going to try to do.
I am going to port blender to bare metal, with a java VM and windows emulation in 1 meg!!!
Second, I don't know if the CPUs of the future will still have a real mode.
Third, y'all seem to frown on real mode, even though Lou Dite (Ludite) told me something about real mode being the only valid mode, but I do want to use halal osdev practices.

My question is, should I spend my time implementing drivers in REAL mode and then "upgrade" to protected or long mode, or should I concentrate on getting to protected or long mode first, and then implementing drivers? it would seem that it could be hard to port RM code to PM or LM because of different memory schemes.

Recently I have been trying to get into protected mode, but my heart is really more in the drivers than the memory. It might be fun writing the same driver twice, but ultimately it would be a waste of time. I have encountered a difficulty enabling the A20 gate. (The VM crashes) I am sure that it is a simple oversight that I will eventually find (like every other of the countless oversights.) but here is the code for laughs:

Code: Select all

; nasm -f bin 
BITS 16
ORG 0x7c00

cli
call a20enable
;nothing gets executed beyond here
mov ebx, 0xb8000
mov byte [ebx], 'X'

jmp $

a20enable:
	push ax
		call a20test
		call a20bios
		call a20test
		call a20kbd
		call a20test
		call a20fast
		call a20test
		;it didn't work
		mov dword [ebx],":0(0"
		hlt
	.enabled:
	pop ax
ret

a20test:
	mov ax, 0x0000
	mov es, ax
	mov byte [es:0x500], 0x00
	mov ax, 0xFFFF
	mov gs, ax
	mov byte [gs:0x510], 0xFF
	mov al, byte [es:0x500]
	cmp al, 0
	je a20enable.enabled
ret

a20bios:
	sti
		mov ah, 0x24
		mov al, 0x01
		int 15h
	cli
ret

a20kbd:
	call kbdcmdwait
	mov al, 0xAD ;PS2 port 1 -> OFF	
	out 0x64, al
	
	call kbdcmdwait
	mov al, 0xD0 ;request output rom controller
	out 0x64, al	

	call kbddatawait ;wait for it....
	in al, 0x60 ;here we go!
	or al, 2 ;A20 -> ON
	push ax

	call kbdcmdwait
	mov al, 0xD1 ;write out
	out 0x64, al

	pop ax
	out 0x60, al

	call kbdcmdwait
	mov al, 0xae	;PS2 port 1 -> ON
	out 0x64, al
ret

kbdcmdwait:
	in al, 0x64
	test al, 1
	jnz kbdcmdwait
ret

kbddatawait:
	in al, 0x64
	test al, 2
	jnz kbddatawait
ret

a20fast:
	in al, 0x92
	or al, 2
	out 0x92, al
ret

boot:
TIMES 510-($-$$) db 0
dw 0xAA55

And here is the error:

Code: Select all

qemu: fatal: Trying to execute code outside RAM or ROM at 0x000a0000
EAX=00007b0c EBX=00000000 ECX=00000000 EDX=00000080
ESI=00000000 EDI=00000000 EBP=00006ef0 ESP=00006ef0
EIP=0009ffbc EFL=00000016 [----AP-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 0000ffff 00009300
CS =0000 00000000 0000ffff 00009b00
SS =0000 00000000 0000ffff 00009300
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =ffff 000ffff0 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT=     00000000 00000000
IDT=     00000000 000003ff
CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=00000000 DR1=00000000 DR2=00000000 DR3=00000000 
DR6=ffff0ff0 DR7=00000400
CCS=00007b0c CCD=00007b7b CCO=ADDB    
EFER=0000000000000000
FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80
FPR0=0000000000000000 0000 FPR1=0000000000000000 0000
FPR2=0000000000000000 0000 FPR3=0000000000000000 0000
FPR4=0000000000000000 0000 FPR5=0000000000000000 0000
FPR6=0000000000000000 0000 FPR7=0000000000000000 0000
XMM00=00000000000000000000000000000000 XMM01=00000000000000000000000000000000
XMM02=00000000000000000000000000000000 XMM03=00000000000000000000000000000000
XMM04=00000000000000000000000000000000 XMM05=00000000000000000000000000000000
XMM06=00000000000000000000000000000000 XMM07=00000000000000000000000000000000
Aborted (core dumped)

I am still developing my debugging skills, so apologize if this is mundane.
I am not sold on protected mode (though knowing I have more than 1MB could be comforting.) so I feel like "is it really worth the effort to get into protected mode?" but then part of me feels like, "you have to be able to figure it out!"

Thanks,
Johnpaul

P.S: I apologize for not replying to my previous thread, I forgot to check the notify box!
I checked it this time.

[nullplan]

Second, I don't know if the CPUs of the future will still have a real mode.

Yeah, Schol-R-lea keeps making noises in that direction, but I doubt real mode will be removed. In order to do that, they would first have to invent a way to boot secondary CPUs in something other than real mode, and that (AFAICS) hasn't happened yet. Real mode is probably here to stay as well, if only as a stepping stone. Now, real mode BIOS, that is a completely different question. That one is already on its way out.

Third, y'all seem to frown on real mode, even though Lou Dite (Ludite) told me something about real mode being the only valid mode, but I do want to use halal osdev practices.

Oh, you can do whatever you want; if you want to remain in real mode, that is your choice. However, real mode is extremely limited, in that it can only address 1MB of address space (plus 64KB, yes I know). Some hardware is literally unsupportable in real mode, as it requires accessing memory close to the 4GB limit, and some even more modern hardware requires accessing memory beyond the 4GB barrier.

My question is, should I spend my time implementing drivers in REAL mode and then "upgrade" to protected or long mode, or should I concentrate on getting to protected or long mode first, and then implementing drivers? it would seem that it could be hard to port RM code to PM or LM because of different memory schemes.

Start with long mode. That way there are no switching pains. So many people here have related their difficulties in converting their OSes from 32-bit to 64-bit, and I don't doubt it. But since long mode is here to stay, you probably should just start with it. Also, you have to get your environment figured out before you can work on your drivers, because your drivers will need to call back into the kernel to do things like allocate memory, register interfaces, &c., so not having everything figured out from the start makes it just that much harder.

Most of all, figure out for yourself what you want your kernel to do. Long mode requires paging, which is less of a hassle than it seems, especially when you know the environment you want in the end. I just end up mapping all physical memory to 0xFFFF'8000'0000'0000 and have more than enough memory left over to map my kernel like a normal ELF executable starting at 0xFFFF'FFFF'8000'0000. Make a plan, then it becomes easier to execute it.

In regards to your code, you are not setting up the stack but are using it anyway (push and call instructions). You are also accessing memory without having set up the DS register. The error you were getting is indicative of something walking off the end of the program. Maybe place a magic breakpoint before the boot signature?

[JohnpaulTH]

Thanks!
I appreciate the recommendation for long mode.
I will look into what you were saying about the stack.
In my other version, I have a GDT and everything (and I do set up the stack), but in this version all the code does is enable A20. this is to demonstrate that the problem lies in the A20 routine. additionally, jmp $ should keep it from walking off the end, shouldn't it?
Also, the character is never displayed on the screen. which means that if it is walking, it is doing so within the functions. I'll look into it more. Thanks for the tip!

[JohnpaulTH]

because your drivers will need to call back into the kernel to do things like allocate memory, register interfaces, &c., so not having everything figured out from the start makes it just that much harder.

This is a library or unikernel, so it is more like a collection of drivers accessible by the main program but since long mode requires paging, I probably have to have a bit more robust kernel.

[Octocontrabass]

When you call "a20test", you push a return address onto the stack, but "a20test" doesn't always pop that value from the stack. This mismatch between push and pop causes later instructions to pop unexpected values. One of those later instructions is RET, and the nonsense it pops must be sending the CPU off into the abyss.

[8infy]

Second, I don't know if the CPUs of the future will still have a real mode.

Yeah, Schol-R-lea keeps making noises in that direction, but I doubt real mode will be removed. In order to do that, they would first have to invent a way to boot secondary CPUs in something other than real mode, and that (AFAICS) hasn't happened yet. Real mode is probably here to stay as well, if only as a stepping stone. Now, real mode BIOS, that is a completely different question. That one is already on its way out.

Nope, it has been invented already. New ACPI specification describes a way to boot APs directly into long mode. (Multiproccessor Wakeup 5.2.12.19 in ACPI 6.4)

For Intel processors, the execution environment is:
Interrupts must be disabled.
RFLAGES.IF set to 0.
Long mode enabled.
Paging mode is enabled and physical memory for
waking vector is identity mapped (virtual address
equals physical address)
Waking vector must be contained within one physical
page.
Selectors are set to flat and otherwise not used.

[kzinti]

Nope, it has been invented already. New ACPI specification describes a way to boot APs directly into long mode. (Multiproccessor Wakeup 5.2.12.19 in ACPI 6.4)

https://uefi.org/specs/ACPI/6.4/05_ACPI ... -structure

[nullplan]

Nope, it has been invented already. New ACPI specification describes a way to boot APs directly into long mode. (Multiproccessor Wakeup 5.2.12.19 in ACPI 6.4)

Ah, cool. Well if that's the case, then not only is there nothing in the way of removing real mode, I'm looking forward to it. But it will take some time for this new version of ACPI to become ubiquitous.

[JohnpaulTH]

When you call "a20test", you push a return address onto the stack, but "a20test" doesn't always pop that value from the stack. This mismatch between push and pop causes later instructions to pop unexpected values. One of those later instructions is RET, and the nonsense it pops must be sending the CPU off into the abyss.

That is very helpful! If I am understanding correctly when I send a "call" instruction. the Instruction pointer is pushed onto the stack. and ret pops it. since I instead do a jump if it is enabled, that causes an imbalance in the instruction pointer. I could tell from the error that it was with a jump, but I could not see where I went wrong! I will see about reshaping the code to address that.
Thanks!

[JohnpaulTH]

That jmp was my problem!
I added an extra pop instruction, and it performs as expected. Thanks! On to the next area.
I am very interested in the new ACPI specification, (though I doubt QEMU supports it yet)

[rdos]

Nope, it has been invented already. New ACPI specification describes a way to boot APs directly into long mode. (Multiproccessor Wakeup 5.2.12.19 in ACPI 6.4)

Ah, cool. Well if that's the case, then not only is there nothing in the way of removing real mode, I'm looking forward to it. But it will take some time for this new version of ACPI to become ubiquitous.

Probably a very long time. Seems like Linux still uses ACPI that is quite outdated (alternatively has branched off from ACPICA). ACPI also gets more and more unnecessary as more and more hardware operates with MSI or MSI-X, and the only reason you need ACPI is to figure out IRQ routings. And to pretend you are a recent version of Windows.

A more decent place to put this in would be UEFI.

[bzt]

I am not sold on protected mode (though knowing I have more than 1MB could be comforting.)

You can use protected mode addressing in real mode, it's called "unreal" mode. Basically you fill up the GDT with prot-mode descriptors and limits, but with 16-bit code segment. Then you set up the segment shadow registers and switch back to real mode. Here's an example code how to do it.

so I feel like "is it really worth the effort to get into protected mode?"

Definitely. Or even more try long-mode.

Cheers,
bzt

[nullplan]

ACPI also gets more and more unnecessary as more and more hardware operates with MSI or MSI-X, and the only reason you need ACPI is to figure out IRQ routings.

That's a bit reductive. I also need ACPI to figure out:

• what legacy devices are present (I really don't like probing unknown ports),
• how to power-manage (standby, reboot, power-off),
• how to get inputs (like power-button, lid button, temperature alerts, etc.),
• how to do certain other things (like read AC status, read battery health info, ...),
• what other CPUs are present (MADT for the win)

But yeah, there is a lot of useless stuff in there. The xHCI spec concludes with an appendix containing an ACPI listing that tells the OS what internal slot connects to what external slot on the device case. I read it and constantly thought why I would ever care about this. I don't think even Windows uses that feature, except maybe to nag users into plugging their SS device into the SS slot.

It seems likely that mainboard manufacturers would want to continue allowing for both approaches with AP startup (both MADT and that mailbox system). With MADT, it does not matter what the firmware makes an AP do, the INIT IPI will clear that out, and the startup IPI will put the AP under OS control from the first instruction. Only non-revertible changes e.g. to the MSRs would remain.

From a purely mechanical perspective, the difference between AP startup with MADT and with mailbox isn't that big. With MADT, you allocate a page, copy your PML4 there, identity map low memory (easy for me, since I map all physical memory linearly, so adding the identity map is just "pml4[0] = pml4[256]"), allocate a page in low memory, copy the trampoline there, set up its variables, and then do the IPI dance. With the mailbox system you need a different trampoline, and are not constrained to low memory when locating a page for it. And don't need to do the IPI dance.

[JohnpaulTH]

I was aware of unreal mode, but I hadn't seriously considered it.
In your opinion is protected mode better than unreal mode?
Also is long mode better than protected mode?
I know you have paging in protected mode, and I also doubt that any of my programs are going to use 4GB memory, so maybe it is not worth the effort of getting into long mode, however, if there comes a point in the future where long mode is the only mode (or the first ode) then it will be good to start there.
I suppose no mode is ultimately "better" than another, they just have different pros and cons.
With unreal you would be doing 16bit code (which takes less space) and have access to bios interrupts, (but that doesn't matter, because I am leaving BIOS interrupts behind.)
With protected mode you have paging. Also of interest is the ability to leverage other cores. Even for a unikernel, that could be very cool. But I am not sure I see any advantage of long over protected mode.

[nullplan]

In your opinion is protected mode better than unreal mode?

Yes. Protected mode already frees you from that weird segment-based addressing real mode does. Unreal mode only gets rid of the segment limit, but you still have segment bases to deal with.

Also is long mode better than protected mode?

Holy hell yes. In long mode, you have 16 registers rather than eight, and all of them are 64 bits wide. Protected mode limits you to one quarter the register capacity of long mode. While 16 is not as much as 32 (which for example PowerPC has), it is already enough to pass parameters entirely in registers. Many functions in long mode can be implemented entirely with registers, never needing auxiliary storage at all.

however, if there comes a point in the future where long mode is the only mode (or the first ode) then it will be good to start there.

That is also a good point. Besides, there is already some hardware that has 64-bit base address registers, and in future, firmware may come along that places these devices into memory beyond the 4GB.

With unreal you would be doing 16bit code (which takes less space)

I would dispute that. It very much depends on the use case. A CRC32 function is going to be smaller in 32-bit mode than in 16-bit mode, merely because you don't need the operand size override prefixes. And the dimensions we're talking about are so small, frankly it doesn't matter outside of boot sectors.

and have access to bios interrupts,

I would dispute that as well. It is entirely possible for parts of BIOS to be implemented in protected mode. See SeaBIOS for an instructive example. It is entirely possible that when leaving protected mode, it will restore the limits it assumes were in effect previously. So calling a BIOS function from unreal mode may very well undo unreal mode. In fact, so may interrupt handling. And suddenly unreal mode loses a lot of appeal.

But I am not sure I see any advantage of long over protected mode.

Look at it this way: When changing from real to protected mode, you already need a major overhaul of your existing code. Switching to long mode at a later date means repeating that process. Why not skip the pain once and directly go to long mode? That way, you only need to overhaul your kernel once instead of twice.

Other than that: More registers, wider registers, more address space (not just RAM) --> more devices you can talk to.