Targeting the RPi and then trying to port this over to a new device would really only take care of the ARM part of the equation, and that's assuming that the RPi and your target device have a processor that shares a memory map and initialisation code. If you want a good introduction to bare-metal ARM development the RPi might not be the best place to start. As has been already noted, it's notorious for bad documentation. If you wanted another option, maybe look at some TI or STM32 boards, I've learned a lot working with them.
How would one create something that could boot...
Every device will have a different approach to bootstrapping a kernel. Some, like the STM32 devices that I'm most familiar with will load a kernel executable directly from flash memory. Some might be able to boot directly from an SD card like the RPi models. For all devices, the process is essentially compiling a kernel locally, encoding it in a format the device will recognise, and then copying that to a place that the device's inbuilt
bootloader will load it from.
Typically, evaluation or development boards will be easily connected to your PC via USB. Other more complex devices might need a JTAG or other hardware interface to flash the memory.
...and if possible send data thru the headphone jack at the least. If that's possible with a moderate amount of work, how would one implement more features to this OS?
What you're talking about now is writing device drivers. You'd implement these features the same way that you'd write a VGA driver in the bare-bones tutorials. Each board will have a different set of hardware attached, such as different audio hardware, a different screen, etc. Most importantly for a smartphone, they'll all have different modems to communicate with the world.
What would be the best aproach? (Using android blobs maybe?). And can one create a basic screen driver to complete the barebones on this hypothetical device? Should first start experimenting with the raspberry pi 3 b+ and then port the OS to the smartphone?
If you're trying to learn bare-metal embedded systems development from the ground up, a great place to start is to pick one of the development boards from TI or STM32. A Cortex-M board is a good place to start if you're looking for a good introduction to ARM. Before worrying about an
operating system per-se, there's lots of different aspects of embedded systems to become familiar with before you start worrying about multitasking, ui and other other things that really make an OS an OS.
Once you've picked a board ( also known as an SoC = System on Chip ), you can begin to write drivers that target the embedded hardware. It's easy to find boards, such as the STM32 Discovery boards that come with a screen already attached. This is good because you can start without needing to do any soldering ( even though that's a good topic to learn too ). Once you've got the system booting up, start by writing a display driver for integral display to blit things to the screen.
This is a very broad topic and question, but that's a good way to start.
I'm too busy to really think about that, though.
I'd much rather pay for one of those phones than spend time and effort trying to reverse engineer closed hardware. Mostly, anyway.