Long Mode, Documentation and more Questions

[FallenAvatar]

Hey everyone,

Been lurking around here for a while. I've read atleast half of the wiki (probably more), and have read tons of documentation online. I have started a "version 0" OS, basically a OS to play with/test different things (build system, testing/debugging, emulators) to figure out how I should organize things once I actually get going.

My main goal is to have a "modern" OS. Namely, Long mode only, SMP "almost only" (Designed for SMP, but I would like to make it work on a single core), High resolution GUI, etc. I know this is going to be hard. I'm not dumb enough to even claim I will have this done in a year, hell, I'll be happy with 5 years .

And, I have a couple questions for you guys, if you have a few minutes

1) When it comes to Long Mode, and SMP actually, is there any emulator that can "closely" resemble real hardware? (Long Mode support, multiple cores? SSE3+ extensions?) Or am I going to have to use real hardware to actually test?

2) I have downloaded and briefly looked at several hardware manuals from AMD and Intel. There are a ton! I realize, that eventually I will probably have to read all of them to understand everything I need, but can anyone offer a good starting point? (Maybe something like, for solid Task Switching on SMP you need to read..., etc.)

3) Plug and Play: (http://wiki.osdev.org/PNP_Calls_In_Protected_Mode) Can PnP calls be made the same way in Long Mode?

4) BIOS functions: Since v8086 does not work in Long Mode, if a BIOS function needs to be called, I need to go all the way back to real mose to make these calls? Or is there a new "v8086" method for Long mode?

5) CPU "rings": Everything I read says to use ring 0 and ring 3. Do modern processors have a ring 1 and 2? Do they offer anything different?

The following questions have a much more intimate relation to my desired implementation. So, I am aiming to have a high security, high performance OS. My GOAL (NOT expectations) is to replace Windows as the gaming OS.

So for performance, I am mainly looking at it as, everything else was designed so long ago, something designed specifically for modern hardware will perform better, because it will take advantage of a ton of new features all the way through the OS.

For security, I am looking to require user-land code to be delivered (from CD, Network, or Internet) as a IL executable, and will use Install-time-compilation to achieve better performance. This also helps with security, because I can determine at install time every function call, etc. Obviously pointer arithmetic will not be allowed, unless I can figure out a safe way to do it. Since I can guarantee a lot using this method, I can "cut" a lot of corners.

6) Given the above, I can make syscalls not require a context switch. I can add code into every "secure" kernel method (a macro probably) that determines where the call came from (by walking up the stack to find the "return" address of the calling function). Shouldn't this be sufficient for security?

7) Taking a step back, is this even something I need to worry about? What kind of overhead do existing syscall implementations cause? Am I over-/pre-optimizing this?

Data Execution Prevention: From what I understand, I can use CS and DS to say this section is code, and this section is data. Do modern processors actually enforce this distinction? Will they fault if a buffer overflow happens, and a function returns to somewhere in the DS?

I think that is it for right now. I know I could potentially find out all of this information through either a ton of googling (I have tried basic googling to find the info), or through trial and error. But, I was hoping that someone had either run into these same situations, and already had the answer, and could save me, and anyone else after me with the same questions, significant time.

I got to say, I love this forum and wiki so far. There is so much info up for grabs here already, and I hope I can add my knowledge some time in the future.

Sincerely,
- Monk

[thepowersgang]

1) Bochs has very good support for most features, but rule 1 of testing is to test on everything you can.

2) Start with tutorials to find your feet, then use the manuals to locate features you want and errors you encounter.

3) PnP is virtually useless nowadays, but I expect they are real mode only. So, no.

4) Using VM8086 is a hack, if you really need access to BIOS functions in long mode... then do them before switching, or use an emulator. (e.g. emu86)

5) Rings 1 and 2 were present in Protected mode, but have been deprecated (and iirc, are not avaliable) in long mode.

6) There are a lot topics here about reducing context switches, but modern processor designs have striven to make them as inexpensive as possible, so attempting to avoid them can quite often lead to less efficent code. However, check for yourself if your design will be better (I believe the AMD manuals have instruction timing, so use that as a guide)

7) See above

8) Long mode only allows a flat memory model (code and data segments both have an offset of zero, and a limit of 0xFFFFFFFFFFFFFFFF)


- Disclaimer: This post was written from the top of my head, so may not be 100% accurate.

[Velko]

4) BIOS functions: Since v8086 does not work in Long Mode, if a BIOS function needs to be called, I need to go all the way back to real mose to make these calls? Or is there a new "v8086" method for Long mode?

By interpreting or JIT-ing Real Mode code.

For security, I am looking to require user-land code to be delivered (from CD, Network, or Internet) as a IL executable, and will use Install-time-compilation to achieve better performance. This also helps with security, because I can determine at install time every function call, etc. Obviously pointer arithmetic will not be allowed, unless I can figure out a safe way to do it. Since I can guarantee a lot using this method, I can "cut" a lot of corners.

6) Given the above, I can make syscalls not require a context switch. I can add code into every "secure" kernel method (a macro probably) that determines where the call came from (by walking up the stack to find the "return" address of the calling function). Shouldn't this be sufficient for security?

It is not going to be sufficient. If you do an install-time compilation, somebody could tamper with executables after they were installed. It may work though if you do run-time in-memory compilation (JIT-ing) every time a program is loaded (I believe this is how .NET works).

Still, if executables may be unsafe, one could mock up a stack to look like a syscall came trough "safe and validating" library functions and then jmp right in the middle of function that makes syscalls. Returning to right place should not be a problem as well. In fact, you should not worry where the syscall came from, validate the arguments instead.

Data Execution Prevention: From what I understand, I can use CS and DS to say this section is code, and this section is data. Do modern processors actually enforce this distinction? Will they fault if a buffer overflow happens, and a function returns to somewhere in the DS?

In Long Mode CS and DS are set up starting from 0 to end of available address space. Since they overlap completely, there's no way to enforce it on per-segment level.

Long Mode paging offers another way, however. Execute Disable bit can be set for every page (or PT, or PD, or...).

[FallenAvatar]

2) Start with tutorials to find your feet, then use the manuals to locate features you want and errors you encounter.

I've read quite a few already, and have a semi-functioning kernel that I am playing with, changing, and testing ideas already. I was looking more for a next step.

3) PnP is virtually useless nowadays, but I expect they are real mode only. So, no.

PnP is useless? How do you go about finding out which hardware the system has then?

...

Thanks, that is some useful info

4) BIOS functions: Since v8086 does not work in Long Mode, if a BIOS function needs to be called, I need to go all the way back to real mose to make these calls? Or is there a new "v8086" method for Long mode?

By interpreting or JIT-ing Real Mode code.

I'm not sure what you mean here? How can you JIT Real mode code? Wouldn't you need to actually jump to Real Mode to run certain BIOS functions and such?

For security, I am looking to require user-land code to be delivered (from CD, Network, or Internet) as a IL executable, and will use Install-time-compilation to achieve better performance. This also helps with security, because I can determine at install time every function call, etc. Obviously pointer arithmetic will not be allowed, unless I can figure out a safe way to do it. Since I can guarantee a lot using this method, I can "cut" a lot of corners.

6) Given the above, I can make syscalls not require a context switch. I can add code into every "secure" kernel method (a macro probably) that determines where the call came from (by walking up the stack to find the "return" address of the calling function). Shouldn't this be sufficient for security?

It is not going to be sufficient. If you do an install-time compilation, somebody could tamper with executables after they were installed. It may work though if you do run-time in-memory compilation (JIT-ing) every time a program is loaded (I believe this is how .NET works).

Still, if executables may be unsafe, one could mock up a stack to look like a syscall came trough "safe and validating" library functions and then jmp right in the middle of function that makes syscalls. Returning to right place should not be a problem as well. In fact, you should not worry where the syscall came from, validate the arguments instead.

I was thinking about signing executables to prevent this.

Data Execution Prevention: From what I understand, I can use CS and DS to say this section is code, and this section is data. Do modern processors actually enforce this distinction? Will they fault if a buffer overflow happens, and a function returns to somewhere in the DS?

In Long Mode CS and DS are set up starting from 0 to end of available address space. Since they overlap completely, there's no way to enforce it on per-segment level.

Long Mode paging offers another way, however. Execute Disable bit can be set for every page (or PT, or PD, or...).

Paging I don't see that bit on the wiki?

Thanks,
- Monk

[AJ]

Hi,

PnP is useless? How do you go about finding out which hardware the system has then?

ACPI

Wouldn't you need to actually jump to Real Mode to run certain BIOS functions and such?

No - if you have a real mode emulator, you can run the BIOS code in that.

I was thinking about signing executables to prevent this.

Even doing that, if someone gains physical access to the machine, you've had it!

Paging I don't see that bit on the wiki?

Have a look at the intel software developer manual Volume 3A - System Programming Guide.

Cheers,
Adam

[FallenAvatar]

Hi,

PnP is useless? How do you go about finding out which hardware the system has then?

ACPI

Bugger, my brain saw power management, and skipped "configuration standard". *sigh* Thanks

Wouldn't you need to actually jump to Real Mode to run certain BIOS functions and such?

No - if you have a real mode emulator, you can run the BIOS code in that.

Somehow I hadn't run across real mode emulators while looking around the internet. Will start doing some research, Thanks.

I was thinking about signing executables to prevent this.

Even doing that, if someone gains physical access to the machine, you've had it!

Eh. If someone has physical access, they can do whatever they like. Obviously, I can't really do anything about it, heh. And I know I will have to watch for rootkits and the like, if possible. But that much is for later.

I'm more worried about a virus or something that gets run by an unsuspecting user. This actually has a nice benefit of preventing self-mutating viruses and the like. TBH, I plan on preventing creation of executables at all on the OS. Only the IL binaries can be "created" and then installed. This allows me to make sure they don't use any high security stuff, by just not supporting those "opcodes" in the compiler.

Paging I don't see that bit on the wiki?

Have a look at the intel software developer manual Volume 3A - System Programming Guide.

Thanks, I will do some reading, then see if I can find an equivalent for AMD. If not, I guess I'm sol on that.

- Monk

[AJ]

Hi,

Thanks, I will do some reading, then see if I can find an equivalent for AMD. If not, I guess I'm sol on that.

You will find an AMD equivalent, but the paging stuff should be the same for both. The bit (bit 63 of the PTE) is called the NX bit. The feature is known as XD by intel and something else by AMD, though.

Cheers,
Adam

[Owen]

AMD's documentation of long mode is far better than Intel's. I recommend their manuals for the 64-bit specific parts of x86.

[FallenAvatar]

Found the NXE bit in the manuals.

I also noticed the SCE bit (System call extensions) in the AMD docs, which determines if the contained instructions can syscall/sysret. This is kind of interesting. Trying to fin something similar in the intel docs...

Hmm, it is only mentioned once in the Intel Docs, in section 5.13.1 as part of an EFER.

I guess I will have to do some more research. And look into syscall and sysret, to see how they perform/what exactly they do.

- Monk

[Tobba]

For BIOS functions, during the loading process you could keep yourself in legacy protected mode for a while to use BIOS calls as needed

If you need to call a BIOS function after the system has fully booted up, you are doing something wrong

[turdus]

legacy protected mode for a while to use BIOS

Wtf? Think it again. There're a few protmode BIOS calls, sure, but neither useful nor bugfree in most cases...

For the OP: you can switch to real mode from long mode directly, call the BIOS and return into long again. There're examples on the wiki and on this forum if you don't want to struggle with sw emulation.
http://forum.osdev.org/search.php?keywo ... =titleonly

[Tobba]

legacy protected mode for a while to use BIOS

Wtf? Think it again. There're a few protmode BIOS calls, sure, but neither useful nor bugfree in most cases...

For the OP: you can switch to real mode from long mode directly, call the BIOS and return into long again. There're examples on the wiki and on this forum if you don't want to struggle with sw emulation.
http://forum.osdev.org/search.php?keywo ... =titleonly

In protected mode you can still use v8086 to do BIOS calls, once you're in long mode you cant do this anymore

[turdus]

In protected mode you can still use v8086 to do BIOS calls, once you're in long mode you cant do this anymore

Still don't get it. Why is it better to switch to protected mode and use a virtual real mode instead of switching to real mode in the first place?!?

[AJ]

Hi,

Because switches from PMode to v8086 mode are faster than switches from PMode to Real Mode, you still get paging and you still get memory protection.

Cheers,
Adam

[turdus]

Because switches from PMode to v8086 mode are faster than switches from PMode to Real Mode

Yes, except we are not in protected mode... From the OP:

4) BIOS functions: Since v8086 does not work in Long Mode, if a BIOS function needs to be called, I need to go all the way back to real mose to make these calls? Or is there a new "v8086" method for Long mode?

So switching directly to real mode from long mode seems to me a better solution. By the way performance is not an issue here, we're talking about a very rare situation (probably only one, changing video mode).