Hi,
IMHO it is possible to make money with a new OS without investing thousands (or millions) of dollars - you just have to be determined, smart, skilled, extremely lucky and capable of living for extended periods of time without food
Rather than discussing if it's possible, I'd rather discuss how an OS developer can increase the chance of making some money...
First step could be to establish a market niche. If your OS is much better at a specific task than any others then you could attract a small part of the whole market to your OS. For example, if your OS is perfect for games (and comes with several extremely good games) you might get users who use your OS for games only, and other OS's for everything else.
The smart way might be to offer your "market niche" OS for free, and when it is established in that market niche (and some third party developers are writing something for it) use your position to expand. For example, the free games OS gets more features that enable it to be really good for music, video, DVD's, etc. Once you've made progress (gained more market share) in the entertainment field you expand again (3D CAD software?). You'd continually build on your market niche until you've eventually covered everything.
There's a few problems with this. You'd need to expend effort maintaining your market niche, so that people don't lose interest and go elsewhere. This in itself is possible, but the more you expand the OS the more your resources (time) is stretched, until your resources are spread too thin and you can no longer maintain interest in all areas.
Ok, that's enough rambling - let's create some lists...
WAYS TO MAKE MONEY FROM OS DEVELOPMENT
A) Give the OS away for free and ask for donations. I expect the amount of money donated wouldn't cover hardware and other expenses.
B) Take advantage of government grants. Some governements offer grants for specific purposes (with specific restrictions). In Australia there's a grant called "COMET" (COMmercialising Emerging Technologies). It's usually between $5000 and $100000 (in $AU) and is aimed at turning innovative technology into tax paying companies. You need to be willing to lose complete control of your project. A good proto-type is recommended. The project must be innovative and have commercial potential.
C) Make it shareware and/or crippleware. Don't expect large amounts of cash. Do expect difficulty getting market share.
D) Give the OS away for free, but charge for an especially good (non-essential) application for it. This approach may mean that the charging money part doesn't interfere much with the market share of the OS itself.
E) Give it away for free, and make your money as an advisor, developer, consultant, etc. You may be able to talk some companies into paying you to write support for their hardware if your OS is directly involved with the same market niche (e.g. for a games OS a manufacturer of 3D helmets might be interested). If I remember correctly this is the way Linus Torvalds is making lots of money (consultant or developer - not sure)...
F) Ask around for investors (e.g. form a public company). Investors won't invest unless they can expect some return on the investment, and it would be difficult to demonstrate any returns.
G) Use your OS and everything you've learnt to get a job in the OS development field. All your dreams and ideas will be lost, but this may be the most possible solution.
PROBLEMS WITH TRYING TO MAKE MONEY FROM OS DEVELOPMENT
A) If people have to pay for it chances are that they won't bother downloading it. You have trouble getting any market share at all.
B) While developers will donate time to an open source/GPL type of project they won't spend time to make you money. You could promise them a share of the cash (or even a flying pig).
C) Without developers donating time you will have trouble writing all applications yourself, and major hassles writing all device drivers yourself. People won't pay for (or use) an OS that doesn't have applications or device drivers, and people won't write more applications and device drivers for an OS that no-one uses (catch 22).
D) You'd need to provide some form of support. Free projects tend to end up with newsgroups, message boards and mailing lists that are maintained by volunteers.
E) The operating system
must be
extremely easy to install, so that people with minimal technical knowledge can install it (a lesson learned from Linux).
If anyone can think of more ways of making money, or ways around the problems.....
Cheers,
Brendan