OS Updater

[vfehring]

I am building a custom operating system, and I am trying to figure out how one would implement system updates. In the long run, I want to be able to have a solution similar to Apple's App Store or the Software Center on Linux, where all apps are updated throug a single location, but so would the operating system. I just cannot figure out how this would be implemented.

If the system is using the files, how would I overwrite them without causing serious issues with the system?

Am I supposed to do the update file replacement during the boot process in assembly, or can I run a custom program written in C before the kernel is loaded if there was an update downloaded?

Any help would be majorly appreciated.

[max]

Update the critical files at some point where they are not needed, like in the shutdown/boot process. Everything userspace should be quite easy, kill (or at least ask for quit), replace, run

[vfehring]

Update the critical files at some point where they are not needed, like in the shutdown/boot process.

Can you provide any resources that I can use as a reference to implement the shutdown/boot process overwrite? I am not sure how I am able to determine wether critical files are still in use or not. I have not gotten much out of my assembly language kernel. So, I am trying to figure this all out.

[sortie]

Allow your programs to be updated while in use. See what Linux distributions are doing. Notice how you can delete a file (unlink it) but the backing inode stays alive. This means that you can delete a program, bit it can continue to run even if not fully loaded.

Don't worry that much about this problem. You will not need to solve it for a long time, until then you can just do full reinstalls instead. When you get that far, this problem is easy to reason about because you know your system well.

[alexfru]

Don't worry that much about this problem. You will not need to solve it for a long time, until then you can just do full reinstalls instead. When you get that far, this problem is easy to reason about because you know your system well.

Looks like this is exactly what Microsoft has done (hasn't worried about it for a long time and we've got now what we've got as a result).

[vfehring]

I am worrying about the system updater so early on because I am trying to utilize the custom OS working from the beginning. I want to be able to update it on the beta testers machine without having to do a complete re-install of the system. I don't want to remove any files that are not meant to be removed, such as user data files and usernames. I only want to update the required files with patches.

[Love4Boobies]

You haven't really identified the problem you're facing. The bigger picture is that you're trying to perform a live update, which involves changing your old software's state to the a new software's state. Files are just one aspect of the state. Focusing on this one aspect and postponing the others until the OS is restarted won't make your job any easier. In fact, it might make it much worse, as you'll have a hybrid of the old and the new.

Your first design decision is imposed on you by computability theory. In particular, you do not want both of the following conditions to be simultaneously true, or else the problem becomes undecidable (unsolvable in finite time on a Turing machine):

• Updates are to be performed immediately, meaning that the software can be interrupted while in an arbitrary state. In most cases, this should not be a significant problem, given that bugs have an estimated mean lifetime of 1.8 years and that new features show up whenever they do; postponing an update for a few seconds or minutes can't hurt much.
• Updates are to be transparent, meaning that the only mechanism provided is state transfer. This should only be an issue if it's too late to build in the required intelligence because the software has already been deployed or a change in architecture is too expensive.

There are many subtle points to take into consideration but I'm in a bit of a hurry so I'll edit this post in a few hours. Stay tuned.

[Combuster]

I want to be able to update it on the beta testers machine without having to do a complete re-install of the system.

Have you considered booting from the network? Especially if your OS is in it's primal stages where you lack a stable filesystem, grabbing everything off the network means free updates every reboot.

[AndrewAPrice]

If the system is using the files, how would I overwrite them without causing serious issues with the system?

Once the new OS image has finished downloading put the kernel into an update mode which closes all processes and overwrites critical files, then loads the new kernel image in and reloads it.

[vfehring]

The actual problem I am having is how to implement a software updater. This is not something I have very much knowledge about, I have always used pre-built software to update my programs that I have written. I do not know much about how an updater works other than the fact that it checks a central location for version information, downloads a file if there is an available update, and overwrites the original file with the updated one.

I want to be able to provide major releases and beta releases through my updater as well. After a beta is done, be able to run the major release upgrade. I have considered using Linux as my kernel but I want to write something unique. Hence this question.

[Love4Boobies]

Most update facilities require the process to be closed, in which case the solution becomes trivial. Alas, if you're building this into a package manager, which is intended to cover a wide range of programs, you don't want to do this because some applications demand high availability (e.g., Web servers, firewalls, most OS components). As I've mentioned before, you haven't yet figured out what problem you're trying to solve. That said, apart from security concerns, there is no difference between updating an OS and a program.

PS: I haven't elaborated on my previous post yet because I'm currently on a train, on my phone, and typing everything would be a pain.

[vfehring]

The whole implementation of a software updater is my problem.

[Love4Boobies]

This is precisely why we're discussing its proper architecture, isn't it? Although I wouldn't recommend skipping the prerequisites stage of the development process in order to jump straight to the implementation, if you have a specific issue that hasn't yet been covered in this thread, feel free to ask. Otherwise, it's like asking how to build a farm and then complaining that the answers are explaining what facilities are in order to tend to the needs of your animals instead of explaining how to lay bricks on top of each other so that you can build whatever comes out. Since you're on an OS development forum, we are assuming you already understand how to download patches over a network (this is just an example; it needn't be a network) and what is generally involved in an installation process.

[Candy]

I've designed an OS updater for a commercial product which should fail as few as possible.

It was based on having two copies of the kernel and two copies of the initrd. The algorithms work as follows:

Code: Select all

original kernel is called kernel.1, original ramdisk is initrd.1. 
When downloading an update, download it to kernel.2 and initrd.2.
- Sync filesystem.
Remove kernel.1
- Sync filesystem.
Remove initrd.1. 
link initrd.2 to initrd.1.
- Sync filesystem
Link kernel.2 to kernel.1
- Sync filesystem
Remove kernel.2 and initrd.2.

The boot code then reads kernel.1 and initrd.1 if kernel.1 exists. If it does not, it reads kernel.2 and initrd.2.

No point of failure left, other than regular filesystem corruption. If you have more files, treat them similar to the initrd and it'll be fine. You can also treat the boot logic & data as a single directory and treat the directory as follows. The basic idea is to have a name for the current one, that's guaranteed to be valid until the second one is downloaded fully, and is guaranteed to be valid as soon as the kernel link is back up. The backup copy is then guaranteed to be valid whenever the main one is not present. Also note that all heavy operations - downloading, writing big files - happens before the first sync, and after that you just do minor changes plus syncs. That makes the entire sync-bit take at most a second, which means the chances of it happening is actually pretty small.

Thing is, this update logic isn't tied to your OS much. You need to have a full FS sync for it to work, and you need a bootloader that can load an alternative fileset. That's about it. Those are easy to retrofit though, so don't bother designing it in now.

apart from security concerns, there is no difference between updating an OS and a program

Except that if your program update fails, you retry it. If your OS update fails, you're borked.

[Love4Boobies]

I don't think anyone has a problem with updating something that isn't running, which is basically what you're doing. That's not interesting at all.