OS program execution in depth.

[loopie69]

Hi,
I have been learning computer architecture for a while to know the inner working of a computer. But I think its leading me no where. For example, I do not know the following or even a hint about it:

When I start computer, how does my bios loads boot menu, then how does it loads a bootloader sothat I can switch OS. Then when in desktop, I write some c program and compile it with a compiler. It compiles an exe. What is in the exe? My machine is same but why a linux executables do not run on windows. Also how comes a program which I compiled for my machine works on others?? When I double click on exe, how does exactly everthing run under the hood? Shed some light, please.

[Artlav]

Try to think it through.
Levels of abstraction.
You start from transistors.
They make logic gates.
Logic gates are put together into an instruction set and an architecture that takes commands and executes them.

Now we are in the software land.
A computer runs programs.
There are plenty of common tasks that programs do.
Such tasks were always grouped into libraries.
BIOS on the lower level.
OS a bit above.
dlls/sos on the user level - sets of code that different programs reuse.

A computer have to be able to run several programs at once.
So there is task switching - a timer which saves the state of one program, and restores the state of the other.

There is storage. The programs are stored in there.
One OS store tham in a format of exe, other - in ELF.

Different OSes have different sets of libraries, so even if you can read the format, the program written for one set of libraries won't run on another.
It's like trying to put a train engine into a ship - it won't fit.

Thinks like that.
Your question is very wide, and is covered over a few years of courses or experience.

[onlyonemac]

You're asking a lot of complicated questions all at once, but in general I think what you're wanting to know mainly is how "data" (e.g. an exe file) can be "run" and make stuff happen. I'm not going to explain it all here as it's long and complicated, but in that case you probably want to research such concepts as machine code and the way in which CPUs execute code (maybe look at the design of a simple CPU as well).

Your other question about Linux vs Windows executable files seems to relate mostly to the concept of an API. Basically what this means is that a particular operating system provides various "utility" functions to programs that are running under it - you've probably worked with a few under Windows, such as to create a new window, place a button in it, and respond to the button being clicked, for example. There are also functions to open files and read from or write to them, and another set of functions for working with networking devices, and so on. These functions are accessed through a means that is consistent among all programs running under that operating system, such as by jumping to the code at a particular memory address (see previous paragraph) or by calling a particular software interrupt (a complex thing that's kind of like calling a function in your C program, but at a lower level). So now the difference between e.g. Linux and Windows is simply that different operating systems provide differing sets of API functions (which is what really sets them apart, from a programmer's perspective) and provide differing ways of accessing those functions, so a program designed for one operating system will not compile or execute under a different one (although there are often fairly equivilent functions, meaning that a program designed for one operating system may be modified slightly to work on another - called "porting" an application).

[loopie69]

Thanks for replay and suggestions. Hmm, looks like I have a huge conceptual gap. May be I should introduce myself a bit clearer sothat you guys can help me more precisely.

I came off from a different world totally not related to IT/CSE. I was a passionate gamer and love of gaming is what brought me into the game development. I write games using popular game engines such as unity, unreal etc. The thing is, we create arts, then we program in a very very high level sphere to create gameplay using javascript/c#/C++/node based tools. And then some low level rendering access to make shaders and materials for games using hlsl,glsl, cg; assuming that engine's shader compiler or driver will do the dirty inner works. Then we hit cook/export/build, some internal tools automate themselves and make a clean executable of games. Previously I worked with c/c++ native windows programs as well. Had some months around learning GCC too. But it was not very deep. Whenever a new processor or console or new hardware tech arrives I used to hear all those buzz words, "cache", "pipelines" etc etc I felt that I lack in depth of hardware. If I know how my processor works, then probably I will know how things works under the hood. So I started computer architecture thing and trying to educate myself 8086 thing. I took help from here.
https://www.youtube.com/playlist?list=P ... U1GwkXAJC-

Completed lecture 3, but these questions are killing me much!

You know, its not easy seeing things run around me, yet pretending and make myself believe that, those run by somehow. I want to know that "somehow". I like sweets, but its locked under a several level of cruel cryptos. I shall decode them all to make everything clean. Help with you light sothat I succeed. Books, docs, sites, posts anything.

[SpyderTL]

Here are some bullet points that may help get you started. They are essential to understanding how computers (both modern and classic) work:

• The CPU does everything. (Not strictly true, but for the sake of this discussion, it's true enough.)
• The CPU only understands one set of instructions/commands. The most common PC CPU command set is x86, and the most common mobile/embedded CPU command set is ARM.
• The CPU only keeps up with a few registers. Everything else must be stored in system memory, or on the hard drive.
• The CPU executes code one instruction at a time, and it trusts that the code it is executing is bug-free. It has no way of knowing if the code is correct or not.
• When you turn your PC on, the CPU starts executing code at a specific memory address, every time.
• The CPU only has two ways to talk to the rest of the system: 1) Reading/Writing System Memory, and 2) Reading/Writing I/O Ports
• The CPU isn't connected directly to system memory or devices. It is connected to the motherboard chipset, and the chipset redirects these two channels (memory and I/O) to the appropriate device/component. Because of this, the CPU has no way of knowing what it's actually talking to.
• The BIOS is responsible for providing the startup code at the appropriate address above. This is essentially the first program that the CPU runs every time you turn on your computer.
• The BIOS is also responsible for providing "system specific" functions that can be used by the Operating System to do things like read/write floppy disks, read/write hard drives, and even send/receive data on the network or the internet.
• The Operating System provides even more functionality that can be used by various applications to do things like capture events from the keyboard or mouse, drawing things to the screen, loading programs from the hard drive, managing resources like system memory, network connections, etc.
• Programs are made up of code, data, and additional information about the program that the OS can use to load the program properly. The code inside the program is normally specific to the CPU that it is running on, and typically specific to the OS that it is running on. However, there are certain programs that are written for "virtual" machines, like Java and .NET that are written for machines that don't actually exist in the real world, but can be run on almost any machine that has the ability to "emulate" the virtual machine.

Hopefully, that will give you enough information to get you started. You can find a lot of detailed information on this site about all of these items. Just ask us if you have any specific questions about any of these.

[onlyonemac]

• The BIOS is also responsible for providing "system specific" functions that can be used by the Operating System to do things like read/write floppy disks, read/write hard drives, and even send/receive data on the network or the internet.

Actually the BIOS isn't really used that much any more, except during the time when the OS is still loading and initialising. Your post implies that the BIOS is regularly used for the things that you list, which is misleading.

[alexfru]

• The BIOS is also responsible for providing "system specific" functions that can be used by the Operating System to do things like read/write floppy disks, read/write hard drives, and even send/receive data on the network or the internet.

Actually the BIOS isn't really used that much any more, except during the time when the OS is still loading and initialising. Your post implies that the BIOS is regularly used for the things that you list, which is misleading.

But if you consider System Management Mode, you may find out that the BIOS is not only regularly used, but also that its use is growing.

[SpyderTL]

I was trying to get through as much information as quickly as possible, so I didn't want to dig too deep. The BIOS is used a lot by 16-bit operating systems, but it is almost completely useless for 32-bit operating systems. In this case, the OS must provide a lot of the same functionality that the BIOS provides, but it must be done using 32-bit code.

Someone posted this link the other day about exactly what the computer does when you turn it on. Very interesting read.

http://resources.infosecinstitute.com/s ... d-systems/

[onlyonemac]

But if you consider System Management Mode, you may find out that the BIOS is not only regularly used, but also that its use is growing.

System Management Mode is not something that we even have the poewr to concern ourselves with as OS developers, so it's not like one can claim, in a newbie introduction, that the BIOS is still used simply because of System Management Mode.

The BIOS is used a lot by 16-bit operating systems, but it is almost completely useless for 32-bit operating systems.

That's basically the same as my point. Nobody uses 16-bit operating systems anymore, therefore nobody uses (is able to use) the BIOS anymore. So my point is still that "the BIOS isn't really used that much anymore", and that still holds true because almost nobody uses 16-bit operating systems anymore. The two go together.

[loopie69]

• When you turn your PC on, the CPU starts executing code at a specific memory address, every time.
• The BIOS is responsible for providing the startup code at the appropriate address above. This is essentially the first program that the CPU runs every time you turn on your computer.

Kindly look at the red pointed section. Did I point rightly? So when I power on the system, cpu is gonna execute at certain memory location. So the bios program(which is a collection of instructions specific to the processors["move ax, cx;" etc kind of stuff]) must be loaded from that specific memory location? In that case, something must place BIOS program to the system memory on specific location, on startup; right am I? And also about specific startup memory location, is it processor dependent or architecture? I mean both AMD processor and an intel one is x86-64 family, so for all of them this address is same or it varies? Or it it universal?

• The CPU isn't connected directly to system memory or devices. It is connected to the motherboard chipset, and the chipset redirects these two channels (memory and I/O) to the appropriate device/component. Because of this, the CPU has no way of knowing what it's actually talking to.

Interesting! Lets assume I am a motherboard manufacturer. So I must contact/collaborate with AMD/intel for how CPU will interact with my chipsets? Then also I have to do this same job for memory module(system) manufacturer such as twinmos to make those memory works with my motherboard? Same applies to video card manufacturer such as nvidia? Or there is a universal API like thing which standardize things so that I dont have to work with specific company to support them? The connection between motherboard and all other components, is it open to learn? If it is, then is there any docs or sites or books where I can extract those knowledge?

[kzinti]

https://en.wikipedia.org/wiki/Reset_vector

[BASICFreak]

The reset vector for the 80386 and later x86 processors is physical linear address FFFFFFF0h. The value of the selector portion of the CS register at reset is F000h, the value of the base portion of the CS register is FFFF0000h, and the value of the IP register at reset is FFF0h to form the segmented address FFFFF000h:FFF0h in real mode.

I haven't seen that type of segment:offset before - how can one access 4GB - 16Bytes in 16Bit mode?
Am I mistaken that CS can only range from 0000 - FFFF? And if not why are we limited to 1MB + 64KB in real mode?



Sorry to ask a question on another's thread.

[SpyderTL]

So when I power on the system, cpu is gonna execute at certain memory location. So the bios program(which is a collection of instructions specific to the processors["move ax, cx;" etc kind of stuff]) must be loaded from that specific memory location? In that case, something must place BIOS program to the system memory on specific location, on startup; right am I?

Yes, but since the CPU doesn't have "direct" access to system memory, the motherboard can simply "redirect" that memory address (area) to the BIOS ROM instead of system RAM.

I mean both AMD processor and an intel one is x86-64 family, so for all of them this address is same or it varies?

I'm 99% sure that all x86 processors start with the same address, but even if they don't, it doesn't matter, because all of this happens way before the Operating System is loaded up. You can just assume that the system will boot up properly, and your boot loader will "magically" end up being loaded to either 0000:7c00, or 07c0:0000. (Both of these addresses point to the same place, but apparently the BIOS gets to choose which address format to use. The boot loader needs to handle both situations.)

I'm also 99% sure that ARM processors (and other processors) use completely different startup addresses. For instance, I know that the Raspberry Pi actually has a separate processor (other than the CPU) that is responsible for setting up the system, and then "enabling" the main CPU when it's done. But this is another situation where you are not allowed to modify the startup program, so in reality, as far as the OS is concerned, you can ignore all of this "startup" logic.

So I must contact/collaborate with AMD/intel for how CPU will interact with my chipsets? Then also I have to do this same job for memory module(system) manufacturer such as twinmos to make those memory works with my motherboard? Same applies to video card manufacturer such as nvidia? Or there is a universal API like thing which standardize things so that I dont have to work with specific company to support them? The connection between motherboard and all other components, is it open to learn? If it is, then is there any docs or sites or books where I can extract those knowledge?

If you were a motherboard manufacturer, then yes. Although a lot of components are standardized, so it's not quite as bad as you would think.

However, over the years, in order to improve performance, more and more of these motherboard components have actually been moved off of the motherboard and put inside the CPU chip, itself. This does improve performance, but it requires that the motherboard manufacturers and the CPU manufacturers work very closely in order to make these changes work, over time.

Luckily, all of this is fairly transparent to the Operating System, in order to maintain backward compatibility. (You can still run MS-DOS 5.0 on your brand new laptop.) So in most cases, you can ignore these hardware implementation changes as well.

The connection between motherboard and all other components, is it open to learn? If it is, then is there any docs or sites or books where I can extract those knowledge?

That's more of a hardware question than an OSDev question. I'm sure there are sites out there devoted to PC hardware design, though.

[SpyderTL]

I haven't seen that type of segment:offset before - how can one access 4GB - 16Bytes in 16Bit mode?
Am I mistaken that CS can only range from 0000 - FFFF? And if not why are we limited to 1MB + 64KB in real mode?

I'm not sure if this is considered "Real Mode" at this point. I think "Real Mode" is an operation mode of 386+ processors that just so happens to be able to run code for 16-bit 8086+ machines, but it's not exactly the same. In other words, I think the BIOS has to configure the CPU to run in "Real Mode" before the boot loader is executed, but it probably doesn't start out that way when you power on the machine.

Someone probably knows this better than I, though...

EDIT: I decided to check the link that I posted above, and it answers this very question (and much more if you want to check it out...)

1.Start of execution in the CPU (processor) reset vector. In all platforms, the bootstrap processor (BSP) starts execution by fetching the instruction located in an address known as the reset vector. In x86/x64 this address is 4GB minus 16-bytes (FFFF_FFF0h). This address is always located in the BIOS/UEFI flash memory on the motherboard.

2.CPU operating mode initialization. In this stage, the platform firmware switches the CPU to the platform firmware CPU operating mode; it could be real mode, “voodoo” mode, or flat protected mode, depending on the platform firmware. X86/x64 CPU resets in a modified real mode operating mode, i.e., real mode at physical address FFFF_FFF0h. Therefore, if the platform firmware CPU operating mode is flat protected mode, it must switch the CPU into that mode. Present-day platform firmware doesn’t use “voodoo” mode as extensively as in the past. In fact, most present-day platform firmware has abandoned its use altogether. For example, UEFI implementations use flat protected mode.

[BASICFreak]

I'll just assume it does because it does - as there is nothing in the CPU regs that actually allow for accessing this memory that I can see on Bochs.

Funny thing is bochs first command at 0xFFFFFFF0 is jmp 0xF000:E05B

CPU Regs start at, CS = F000, ip = FFF0, cr0 = 60000010 (CD, NW, ET)

then after the jump the CPU Regs are:
CS = F000, ip = E05B, cr0 = 60000010

And after a few more instructions the stack gets setup at 0000:FFFE
A few more and DS gets set to F000


As the CPU doesn't have direct access to system memory, It must just be firmware (SMBus) or something similar that loads that address. Plus if the article is correct, system memory is initialized in step 3, just after this code - which just may make this possible (obviously it's possible as they are doing it).

Anyways interesting article, I have only skimmed it thus far.
I have never worried about how the PC starts up, I just know "my" entry point at liner address 0x7C00 with a size of 0x0200 and you have no clue if BIOS gets you to 0000:7C00 or 07C0:0000 so you have to setup your own offsets.