What generates operating system files/folders, and when?

Discussions on more advanced topics such as monolithic vs micro-kernels, transactional memory models, and paging vs segmentation should go here. Use this forum to expand and improve the wiki!
klange
Member
Member
Posts: 679
Joined: Wed Mar 30, 2011 12:31 am
Libera.chat IRC: klange
Discord: klange

Re: What generates operating system files/folders, and when?

Post by klange »

I would like to provide my own answer to the original question.
This is probably a really simple question, but I'm just beginning to think about hard drive access in my operating system and then realised that I actually don't know what bit of software creates the OS filesystem (for example /bin, /etc, etc.. on linux), and when it's generated.

Does the kernel, when booting up, check that all of these folders/files exist, and if they don't, it creates them? This doesn't seem quite right.
Linux has what is called a virtual file system. It is not alone in this, in fact essentially every modern operating system has a virtual file system. A virtual file system allows many different kinds of file systems to be accessed by the same common interface. Operating systems like Linux, and its precursors and brethren in the Unix family, place all of these filesystems together in one namespace. This allows a directory like /usr to represent the contents of one filesystem while the rest of / is another, which was very common in the early days of Unix. These days you're more likely to see /home as a separate filesystem, rather than /usr, but the idea remains the same. Those directories represent file systems on a real disk, that already existed there before the operating system started, and which the kernel assigned to those virtual paths on startup. But this unified, single-namespace virtual file system also allows Linux to do something else: Have you ever looked in /sys or /proc? Those don't exist on a disk, they're synthesized by the kernel when you access them - truly virtual files within the virtual file system. On newer Linux systems, /dev may also work this but previously it did not: /dev used to be an actual directory on disk, containing special files called device files, which have device identification numbers stored in them instead of data. It used to be that /dev was set up manually by creating these files on disk, but this was later replaced with automated systems like udev, before being replaced with the synthesized version that allows the kernel to do it all automatically.
So let's say I'm booting up a computer from my kernel, which is loaded onto a CD-ROM. Hmm.. As I write this, it's all starting to make sense, so, is this correct?:
Let's start at the bootloader. There are a lot of different architectures where booting works in subtly different ways, but the general idea is the same: The bootloader is a very small program, stored somewhere that the firmware - the computer itself - knows where to find it. In the context of a PC with a BIOS, the BIOS knows how to read hard disks and will check for these small chunks of code at the start of disks. Because the amount of code the firmware can load on its own is often limited (512 bytes in the classic PC BIOS, which is known as a master boot record or MBR), bootloaders would typically be broken up into multiple parts which we commonly call stages. GRUB, for example, has a small initial stage that loads another stage. GRUB is a big bootloader - it's essentially a full operating system in its own right. Its initial stage looks for a stage "1.5" which is stored at a hardcoded place on the disk and contains code to read from the filesystem on that disk. That stage can then look for files within the file system and find the final stage which contains the rest of the GRUB code. All of this data - the MBR, the stage 1.5 code, and the rest of grub - is created by an installer.
In this case, though, how does the bootloader know what is code, and to load into ram, and what is data (such as the /bin, and /etc folders), so to not load those into ram? In other words, when the bootloader (say, Grub), is loading the kernel from the hard drive into ram (by searching for the 0xbadb002 magic number), how does it know when it's got to the end of the kernel program which it wants to load to ram, and into the beginning of data?
As I said, GRUB is essentially an operating system in its own right. It has disk drivers and file system drivers. It knows what files and directories are, and when you give it a pathname it can find those files on disk. Grub also knows what a Linux kernel is and how to read and load it to the right place following Linux's boot protocol. Note that "the 0xbadb002 magic" is part of a specification called multiboot and is not used by Linux, but for a Multiboot-compliant kernel this magic sequence allows Grub to find a header with information on how to load things. Regardless, both Multiboot and Linux's boot protocol define specifications for how the bootloader can pass control to the kernel and give it configuration settings and other data.

Others in this thread talked about initrds - initial ramdisks. Typically in a Linux system, when the kernel first boots, it doesn't necessarily "mount" a physical disk. Alongside the kernel, the bootloader loads a file from disk which contains its own filesystem (called a cpio archive in Linux's most common case, though it supports initial ramdisks in a number of formats). The filesystem stored in that ramdisk can have its own directory structure, including its own /bin or whatever you want to have there. You can run an entire system through this ramdisk on Linux, but typically they just contained programs critical to the start up process - scripts to mount other disks, sometimes a boot splash. But initrds aren't a requirement, and in fact the ones on the Ubuntu installation I'm using right now only contain a directory with microcode the kernel loads for the CPU.

So that's Grub and Linux, but let me go on a tangent and talk about my own OS here:

I provide my OS as a live CD. It comes as a file - an "image" - which contains an ISO 9660 file system, suitable for burning to a real CD, but most people load it in an emulator. I have bootloaders available for BIOS and UEFI, and I do some complicated things with that CD filesystem to make this work well. The ISO 9660 file system contains a file which itself has a FAT32 file system. This is used by UEFI, and some headers in the CD filesystem tell UEFI where it can find that FAT32 filesystem. The rest of the ISO 9660 contains both the BIOS bootloader and specially crafted files which point into the FAT32 filesystem and mirrors its contents. This is known as a "hybrid filesystem", in that two different formats can be used to access the same files.

The FAT32 image contains the EFI bootloaders, the kernel, device driver modules and the ramdisk. The kernel is a multiboot-compliant ELF binary, the device driver modules are ELF object files, and the ramdisk is a tarball. The EFI bootloaders use EFI's APIs to access the FAT32 filesystem, while the bootloader contains an ISO9660 filesystem driver. Both the EFI bootloaders and the BIOS bootloader implement the mutliboot specification and use their respective file system interfaces to load the kernel, modules, and ramdisk into memory. Using the multiboot spec, the bootloader provides information to the kernel on what additional files it loaded (the modules and ramdisk). The kernel then starts, does some early setup, and finishes loading each of the modules (which contain device drivers for file systems, among other things). It then attempts to mount the ramdisk into its virtual filesystem and execute /bin/init. To do that, one of the modules is a file system driver for tarballs, which allows the kernel to perform typical vfs actions (read directories, read files) on the contents of the tarball that is already in memory. /bin/init is a typical userspace ELF binary which can then do whatever it wants. My live CDs use a script-based startup system, so init will run each of the ordered start up scripts, and one of the early ones will replace the tarball filesystem with an in-memory "temporary file system", which can be written and expanded arbitrarily, and then copies the contents of the original tarball into that tmpfs so that the live CD has a "writable root filesystem". Like Linux, my OS also has a synthesized /proc as well as a synthesized /dev.
User avatar
eekee
Member
Member
Posts: 872
Joined: Mon May 22, 2017 5:56 am
Location: Kerbin
Discord: eekee
Contact:

Re: What generates operating system files/folders, and when?

Post by eekee »

bzt wrote:
eekee wrote:if a CF card is plugged in at boot, the CF card will be hda and the internal disk hdb or hdc.
That's the thing, my hardware configuration does not change. I don't plugin-in nor disconnect nothing. I do not change any firmware configuration either.
Yep. What I was thinking was that they failed to fix the underlying problem in my day, (which was serious enough,) and now it's come back in a worse form. Maybe it's not exactly the same problem, but I believe the same attitude is to blame.
bzt wrote:
eekee wrote:varied with module loading order. That might be the problem now, actually.
Nope, both disks are connected to the same sata controller, therefore both should be using exactly the same driver. Also my initrd image does not change, so in theory readdir() should return the modules in the same order for every boot.
Strange. :( Still, I can guess a few reasons. They all come back to randomization of *something* for security reasons. I think the problem can still be stated as "The Linux kernel refuses to tidy up its own messes."
bzt wrote:
eekee wrote:The root of the problem is Torvalds policy of "no policy in the kernel." I don't like heavy policy, but a little policy in the right place can save tons of trouble.
Well said! However I don't recall Torvalds ever said "no policy",
Thanks! "No policy in the kernel" was his reason for rejecting devfs, which was quite a long time ago now. Personally, I was very happy with devfs patches.

I suppose devfs does mean the kernel "checks [...] and if needed makes [...]," but not quite in the way OP meant. :)
bzt wrote:I absolutely agree that without a consistent concept kept no software can be useful to end-users (it could be a marketing-success sold in millions of copies though, that has nothing to do with the software's actual quality unfortunately).
Consistent architectures can be so much easier and more fun! :) I love consistent things from the OS in Atari 800 to the APLX language environment on Windows. Hmm... note to self: try more OS projects from this site, there must be some good ones.
Kaph — a modular OS intended to be easy and fun to administer and code for.
"May wisdom, fun, and the greater good shine forth in all your work." — Leo Brodie
User avatar
Solar
Member
Member
Posts: 7615
Joined: Thu Nov 16, 2006 12:01 pm
Location: Germany
Contact:

Re: What generates operating system files/folders, and when?

Post by Solar »

eekee wrote:
bzt wrote:
eekee wrote:if a CF card is plugged in at boot, the CF card will be hda and the internal disk hdb or hdc.
That's the thing, my hardware configuration does not change. I don't plugin-in nor disconnect nothing. I do not change any firmware configuration either.
Yep. What I was thinking was that they failed to fix the underlying problem in my day, (which was serious enough,) and now it's come back in a worse form. Maybe it's not exactly the same problem, but I believe the same attitude is to blame.
The solution here is to not rely on partition numbering to identify your boot medium, but to use UUID instead (which is what any halfway-modern Linux distro does by default). The sequential numbering is legacy support.
Every good solution is obvious once you've found it.
User avatar
bzt
Member
Member
Posts: 1584
Joined: Thu Oct 13, 2016 4:55 pm
Contact:

Re: What generates operating system files/folders, and when?

Post by bzt »

Solar wrote:The solution here is to not rely on partition numbering to identify your boot medium, but to use UUID instead (which is what any halfway-modern Linux distro does by default). The sequential numbering is legacy support.
Thank you for your advice. The problem is not with the partition numbering, but with device naming: sda sometimes comes up as sdb. Yes, for fixed fses I could use UUID (I already have a fallback grub option for this), but then I can't use any removable media (because I don't know their UUIDs in advance I must use /dev/sdb in fstab, however USB sticks comes up as sdc). The only solution is to keep rebooting the machine (usually 2-3 times) until the first disk comes up as sda and not sdb.

Cheers,
bzt
User avatar
Solar
Member
Member
Posts: 7615
Joined: Thu Nov 16, 2006 12:01 pm
Location: Germany
Contact:

Re: What generates operating system files/folders, and when?

Post by Solar »

You're being fuzzy. What is your X problem, i.e. what are you trying to do? (Not the Y problem that occurred while you're trying to work around the problem you had with X.)

Optional booting? (Boot medium should be selected in BIOS, and the CF card should "know" its own UUID.)

Accessing the medium from a running desktop? (Should be automounted at /media/<username>/<label>/.)
Every good solution is obvious once you've found it.
User avatar
bzt
Member
Member
Posts: 1584
Joined: Thu Oct 13, 2016 4:55 pm
Contact:

Re: What generates operating system files/folders, and when?

Post by bzt »

Solar wrote:What is your X problem, i.e. what are you trying to do?
I'm just using my machine. I have an installed and configured Linux box for years now, it is not a new. I'm booting from sda, and in order to allow non-root mounting of USB sticks, I have a line in fstab for /dev/sdb. This configuration was working for more than 4 years, up until recently. Since a kernel update about a few weeks ago, when I boot, sda sometimes (but not always) comes up as sdb (and consequently the removable USB sticks are then assigned the device sdc as sdb is already in use). That's my problem. The hardware configuration does not change, I do not connect new disks (no USB sticks either during boot, no CF cards, no optional booting etc.), nor disconnect anything, nor do I change the BIOS configuration; which suggests they might have introduced a race condition in the device naming in the disk controller driver.
I'd like to point out that I'm using Linux on several machines with the same configuration, and this problem only appears on one of them and only with the recent kernel. The first disk is always sda on the rest, no matter how many times I reboot them.

Cheers,
bzt
User avatar
Solar
Member
Member
Posts: 7615
Joined: Thu Nov 16, 2006 12:01 pm
Location: Germany
Contact:

Re: What generates operating system files/folders, and when?

Post by Solar »

bzt wrote:I have an installed and configured Linux box for years now, it is not a new. I'm booting from sda, and in order to allow non-root mounting of USB sticks, I have a line in fstab for /dev/sdb.
OK, time has definitely passed you by. Modern Linux distros should not require any tinkering with /etc/fstab. When I plug in any removeable drive in my machine (Linux Mint 19), any partitions on that drive get auto-mounted as /media/<username>/<label>, with an icon appearing on my desktop and a shortcut appearing in the file browser.

Perhaps it's time for you to migrate to a newer distro...
Every good solution is obvious once you've found it.
nullplan
Member
Member
Posts: 1766
Joined: Wed Aug 30, 2017 8:24 am

Re: What generates operating system files/folders, and when?

Post by nullplan »

Also, if you don't like automounters and still use removable media (like flash drives, MMC cards, etc.), there is pmount. Highly recommended for those who want to mount everything themselves. And still no fiddling with /etc/fstab.
Carpe diem!
User avatar
bzt
Member
Member
Posts: 1584
Joined: Thu Oct 13, 2016 4:55 pm
Contact:

Re: What generates operating system files/folders, and when?

Post by bzt »

Solar wrote:OK, time has definitely passed you by. Modern Linux distros should not require any tinkering with /etc/fstab. When I plug in any removeable drive in my machine (Linux Mint 19), any partitions on that drive get auto-mounted
It has not. I deliberately choose to avoid automount because I hate it. I'm not lazy, I can type in "mount" when I need it. Interestingly I have no problems with that on my other machines, and even on this machine it works fine should the first disk assigned to sda during boot (as it supposed to!). Besides, I don't think installing a user-space automounter would solve the device assign issue during boot... Which is the root cause, not that I've added a line to the fstab or that I type "mount" manually.
Solar wrote:Perhaps it's time for you to migrate to a newer distro...
??? How would a new distro solve a _kernel_ issue? I don't think so. I rather keep my eye on the tickets and upgrade the kernel when this gets fixed. (About Linux Mint, I've abadoned release-cycled distros about 15 years ago, and I'm absolutely happy with rolling-release ever since. There's no way I would go back to any release-cycled distro, under no circumstances.) Thanks for the advice though.
nullplan wrote:And still no fiddling with /etc/fstab.
Thanks, but I rather edit a plain text file once after install than to introduce a new dependency on my system. I'm not afraid of fstab, and frankly, I don't understand why are you. Tailoring fstab after install under any UNIX is natural, part of administering and configuring the box, and not something that you should avoid at all costs. I'm very happy with it actually.

Cheers,
bzt
nullplan
Member
Member
Posts: 1766
Joined: Wed Aug 30, 2017 8:24 am

Re: What generates operating system files/folders, and when?

Post by nullplan »

bzt wrote:Thanks, but I rather edit a plain text file once after install than to introduce a new dependency on my system. I'm not afraid of fstab, and frankly, I don't understand why are you. Tailoring fstab after install under any UNIX is natural, part of administering and configuring the box, and not something that you should avoid at all costs. I'm very happy with it actually.
I avoid tinkering with fstab if I have no idea what name the device will have once I actually plug it into the system. A USB stick might be sdb, or sdb1, or sdc, sdd, etc. USB hubs exist, and /dev/sdXY is not a stable namespace. Maybe I boot with a USB stick plugged in and somehow it is now called sda? An SD card might be mmcblk0 or mmcblk0p1, or I might need more than one SD card.

For all of these cases, using the classic route I would first have to create the mountpoint myself, then edit /etc/fstab, before I can finally mount the device. Or, I can just run "pmount sdb1" and be done with it.
Carpe diem!
mmdmine
Member
Member
Posts: 47
Joined: Sat Dec 28, 2019 5:19 am
Location: Iran
Contact:

Re: What generates operating system files/folders, and when?

Post by mmdmine »

nullplan wrote:
bzt wrote:Thanks, but I rather edit a plain text file once after install than to introduce a new dependency on my system. I'm not afraid of fstab, and frankly, I don't understand why are you. Tailoring fstab after install under any UNIX is natural, part of administering and configuring the box, and not something that you should avoid at all costs. I'm very happy with it actually.
I avoid tinkering with fstab if I have no idea what name the device will have once I actually plug it into the system. A USB stick might be sdb, or sdb1, or sdc, sdd, etc. USB hubs exist, and /dev/sdXY is not a stable namespace. Maybe I boot with a USB stick plugged in and somehow it is now called sda? An SD card might be mmcblk0 or mmcblk0p1, or I might need more than one SD card.

For all of these cases, using the classic route I would first have to create the mountpoint myself, then edit /etc/fstab, before I can finally mount the device. Or, I can just run "pmount sdb1" and be done with it.
archlinux's genfstab has a switch to use UUID instead of sdXY. UUID is stable.
I'm using root=/dev/sda1 as my kernel configuration and it fails to boot when I put a usb stick:)
User avatar
eekee
Member
Member
Posts: 872
Joined: Mon May 22, 2017 5:56 am
Location: Kerbin
Discord: eekee
Contact:

Re: What generates operating system files/folders, and when?

Post by eekee »

I'm not sure it's been practical to use Linux without introducing new dependencies for a long time. Consider udev; that's hardly normal for a unix and it's widely hated as hard to configure. I gave up & jumped ship.

Semi-related: I know a supercomputer sysadmin who can't do his job without SystemD's logging library because users submit software which requires it. It's a good thing it can be configured to output to sysklogd because SystemD doesn't (or didn't) have fast enough search tools for its binary logs.
Kaph — a modular OS intended to be easy and fun to administer and code for.
"May wisdom, fun, and the greater good shine forth in all your work." — Leo Brodie
User avatar
Solar
Member
Member
Posts: 7615
Joined: Thu Nov 16, 2006 12:01 pm
Location: Germany
Contact:

Re: What generates operating system files/folders, and when?

Post by Solar »

bzt wrote:Besides, I don't think installing a user-space automounter would solve the device assign issue during boot... Which is the root cause, not that I've added a line to the fstab or that I type "mount" manually.
There's no "device assign issue" because the device enumeration doesn't matter anymore.
bzt wrote:I rather keep my eye on the tickets and upgrade the kernel when this gets fixed.
It is unlikely to get fixed when it ain't broken in the first place. hdX, sdX, no-one cares anymore exactly because no-one wants to fiddle with /etc/fstab and manually assigned mountpoints anymore.
bzt wrote:(About Linux Mint, I've abadoned release-cycled distros about 15 years ago, and I'm absolutely happy with rolling-release ever since.
Good for you. I've actually started with rolling release (Gentoo) because the release-cycle distros of the time sucked. But I eventually came back to release distros where I don't have to tinker that much with configuration myself because I like to focus on using my computer, not fiddling with things that "don't work quite right yet", once release-cycled distros caught up with the rolling-release ones feature-wise.
bzt wrote:I'm not afraid of fstab, and frankly, I don't understand why are you. Tailoring fstab after install under any UNIX is natural, part of administering and configuring the box...
I haven't touched any /etc/fstab for the last... uh... seven to eight years? Because not tailoring stuff in /etc is a natural part of not being bothered with administering the box but instead using it productively.

Anyhow, with the amount of fixed and removable drives I am using every day, I'm pretty quickly at "sdg" or "sdh", and I prefer to access them by their logical label, not their physical one. /media/solar/foo is /media/solar/foo no matter if I plugged it in before or after /media/solar/bar, or indeed no matter if I plugged it into my own box or somebody else's.

Bottom line, you've gotten yourself stuck. I understand that, actually -- I got stuck when I had to switch from AmigaOS to Windows, and boy was that a hard "stuck". (Switching from Windows to Linux wasn't half that bad, but I still miss the Workbench.)

In the end, keeping the good memories and moving on is the only way forward...
Every good solution is obvious once you've found it.
mmdmine
Member
Member
Posts: 47
Joined: Sat Dec 28, 2019 5:19 am
Location: Iran
Contact:

Re: What generates operating system files/folders, and when?

Post by mmdmine »

@Solar
Amiga looks great. I wasn't able to setup an emulator for it (earliest OS I had was Windows XP but I tried everything you guess on emulator, from MS-DOS, Windows 2/3.1/98/ME/NT to Apple System Software 1-6 and A/UX and Mac OS and NeXT Step to QNX and Solaris and more). for Now I missed my PC because I've broken my keyboard and I won't be able to buy one 'till July.
...
No one is able to create a stable /dev/sdXY namespace because it's generated at runtime, instead you can use /dev/disk/by-uuid but it's a big number that's hard to remember.
...
I'm using Archlinux, it's rolling release and I didn't have any problem at all. it's great specially the pacman.
last time I touched my fstab was when I was installing it, too.
User avatar
bzt
Member
Member
Posts: 1584
Joined: Thu Oct 13, 2016 4:55 pm
Contact:

Re: What generates operating system files/folders, and when?

Post by bzt »

Solar wrote:There's no "device assign issue" because the device enumeration doesn't matter anymore.
You can't be serious about this. Can you imagine what would happen if for example the network cards were assigned different device names on each reboot?
Enumeration or not, the first and only disk should be sda, and never sdb, of that I'm certain. This is simply a bug in the kernel (or more precisely in the driver).
mmdmine wrote:last time I touched my fstab was when I was installing it, too.
Same here. I've added one single line to the fstab after install about 4 or 5 years ago, and haven't touched that since. It works great, and I don't need any additional utilities.

Cheers,
bzt
Post Reply