To put my conclusion at the top, I suggest targetting the 386. It has a reasonable feature set with not too much to learn, and you get all the x86 PC OS dev resources. There's a document, 386intel.txt which is the entire Intel 386 reference in under 1MB, even when converted to UTF8. (The original is DOS codepage 437 if not 850.) Even that 1MB includes stuff you don't necessarily need to read, such as using segmentation for security.
I've long been looking at ways to simplify what I have to learn. I have serious difficulties taking in complex information. Sometimes, I don't see stuff which is right there in front of me, even to the point of
believing its not there. Despite this, some degree of OS development seems to be possible for me. Planning really exotic new designs is too hard, but I can relatively easily think about many individual components of an OS, such as a scheduler.
The previous posts in this thread cover a lot of what I've found over the years. There are no simple high-performance CPU architectures and good security is horribly difficult, but do you really need those things? It depends what you want to do with your OS. For a web server, I've had ideas of running nginx on OpenBSD as a proxy, handling https so my code doesn't have to. There's more to security than encryption, of course, but I look at it the way banks do: don't spend unnecessary amounts of money (banks) or time and effort (coders) on security, just enough to keep the criminals out. The effort required depends on the site contents and popularity. If you're making a non-POSIX, non-Windows compatible OS, the incompatibility and uniqueness of your system may also work in your favour if your site or network is not too attractive to criminals.
I've had a lot of "desktop" use out of an OS without a modern web browser. For years, I used 9front for plain-text MUD games, text editing, image browsing, all the notes I made on everything, and more, all while using Linux for web browsing and other games. More recently, I've been making my notes using a feature of Forth interpreters left over from the time when Forth was used as a primitive operating system, and now I'm getting parts together to use Forth in exactly that way on a microcontroller.
So, obviously, I'm going for a simpler architecture. I thought about the 8086 PC, but i wanted to do something different and I didn't fancy segmentation so I switched to ARM microcontrollers. Now I have a flat memory space 1/4 the size of one 8086 segment!
(I would have bought microcontrollers with more memory, but with my health problems I get afraid of getting muddled up when spending money.) It'll be fine with Forth which is designed to page out its
block buffers as needed, even without an MMU. A primitive Unix would be another option; it originally ran with only 16KW memory -- 32KB -- and no MMU. It switched tasks every 0.25s, keeping only the currently running task in memory. All the other tasks were swapped to disk.
I have had thoughts of developing an OS for the 6502 -- only 52 instructions! But it's so simple, you have to think differently about a lot of things, and that confused me. Still, people have written Unixes and reasonably-performant GUI OSs for the 6502.
"Encryption: a powerful algorithmic encoding technique employed in the creation of computer manuals"
-- from a very old
joke computer dictionary. The more things change, the more they stay the same.