Bespoke solution - do you want to team up? - Large idea,

[tsdnz]

Hi guys.

I have come across an idea which I have tested using Visual Studio 2012 in C++.
It works ok-ish, but as it is running on top of windows I have no control over some of the key aspects that I wish to control.
Currently it is running about 470 times faster than windows software, but as above I am tied into the OS. (Linux is the same, it has to be it is an OS)
This is a great performance increase, but when certain conditions happen it slows down for a few minutes, or even stops all programs for a few minutes.
This is not due to my programming but due to the OS
I have talked with a few large companies within my country and a few international companies who are very interested.
I can supply them the C++ version, but I want to provide them with an amazing system, not tied to any OS.
Also, the C++ version, Windows or Linux, etc., will be slower and will require additional PC's and personnel.
I have designed test hardware which is working well.
I have teamed up with a large telecommunications provider.

I am not after an OS, this would defeat the purpose. No hardware abstraction layers wanted.
I am after a bespoke solution.
I know what I want, but my limited knowledge of chip programming is slowing me down.
I know assembly, but setting the chip(s) up and cards is unknown to me.

I want to know if people are interested in assisting with the project.
At this point in time I cannot pay for peoples time.
I have sold one of my companies and about to sell my Silverlight software development framework I designed a few years back.
I have investors that are paying for my time and additional equipment.
I am a contract programmer.
I am not taking on any more contract projects.
I am about to start working fulltime on this project.

A few very large companies have invested millions upon millions in this idea, but they have all used existing technologies built upon existing OS's
They have also approached the project from the wrong perspective.
As such they have put the project into the too-hard-basket.

Due to my limited knowledge in chip programming (setting it/them up) I thought I could quickly learn it.
But I have found that I am spending many hours researching, writing code and testing the results.
I currently have the code written in assembly using Visual Studio 2012 as the IDE.
I have setup F5 to run batch commands, that will build and run it under qemu.
This works nice, I love the macros in nasm.

Basically the system needs (no hardware abstraction layers, just raw bytes):
+ Hard disk access
+ Ethernet control
+ Keyboard
+ Text screen, graphics is fine, it is only to display simple messages.

I have a memory manager that will be used.
I have the hard disk read/write program to use.

When Ethernet interrupt is received we will parse the raw bytes partially in the interrupt and if required for further processing then in another thread.

Each thread will run one program, so no scheduler etc. is required.

The entire system will be designed for one task.
-------------------------------------------------------------------------------------------------------------------------
There will be another bespoke system that needs to be designed.
This system is a bespoke database.
Same as above system but different code.
I have written both systems in C++ up to the point where I am happy that I know what I want and where to head from there.
If I continue with the C++ versions it will not give me the satisfaction that I will get from the asm version.
And also the $$.

Both projects are not large, I think it would take more time to write a basic OS.

I will stop tying here, I was planning to go a little deeper.

-------------------------------------------------------------------------------------------------------------------------
So what am I after??

I would like the system to load the interrupts, gdt, etc.
Map 32 GB
Start all processors (I am testing on a single AMD-8320 8 core)
Start all cores at address xxxx
Set up all Ethernet cards, (I am testing with just one)
Set up all hard disks, (I am testing with just one)
I would also like the source code and if possible written in assembly nasm.
I can provide the nasm files etc to run under Windows Visual Studio 2012
-------------------------------------------------------------------------------------------------------------------------

Any and all feedback is welcome.

Kind Regards, Alistair

[Brendan]

Hi,

I am not after an OS, this would defeat the purpose. No hardware abstraction layers wanted.

So what am I after??

I would like the system to load the interrupts, gdt, etc.
Map 32 GB
Start all processors (I am testing on a single AMD-8320 8 core)
Start all cores at address xxxx
Set up all Ethernet cards, (I am testing with just one)
Set up all hard disks, (I am testing with just one)
I would also like the source code and if possible written in assembly nasm.
I can provide the nasm files etc to run under Windows Visual Studio 2012

You don't want an OS, but you want a (single-tasking) OS with several drivers? I'm thinking you might be looking for something like BareMetal OS.


Cheers,

Brendan

[tsdnz]

Thanks, I will have a look

[VolTeK]

I have teamed up with a large telecommunications provider.

Currently it is running about 470 times faster than windows software

I have sold one of my companies and about to sell my Silverlight software development framework I designed a few years back.
I have investors that are paying for my time and additional equipment.

Pics or it didn't happen, bro.

[gravaera]

Yo:

You should probably work on having an active, working proof-of-concept first; then you can present a working prototype that demonstrates what direction you intend to move in. Additionally your inability to concisely explain the project's niche probably hints at you not really yet understanding the technology well enough to achieve the great things you've projected at your audience

I wish you the best of luck, but I'd also in conclusion like to advise you not to begin talking to all kinds of companies and accepting funding for an inexistent project. It is perfectly possible to write a kernel in isolation at zero cost, and then take developers on board later on when you have something concrete to work on.

--Good luck,
gravaera

[Brendan]

Hi,

Additionally your inability to concisely explain the project's niche probably hints at you not really yet understanding the technology well enough to achieve the great things you've projected at your audience

Or possibly; just not wanting to reveal the idea to the general public to avoid having the idea stolen (because it's cheaper/easier than applying for a patent).


Cheers,

Brendan

[tsdnz]

Brendan, you are on to it. I know you guys can do what I want, I don't want the idea stolen.

My investors are private investors, mum and dads, and companies I know well.
Large investors want a %, which I cannot bring myself to accepting.
They are also very nosey and controlling.
The mum and dad's will get 11 times the amount they invest.
I have capped the investing at $ 2,000 per week, we are not at this level yet, just starting out on this investment idea.
Also only asking between $ 10 and $ 200 per week per person.
The people investing are investing to help me, not to get rich.
I can give them a thank-you if I pull it off, but returning 11 times what they put in.
The time limit is 2 years.
Every person knows that the money could be lost.
Around $2.5 million will be returned is all goes well.
The people know me well and know that this is not certain, nothing is.
I am not after huge $$, just enough to keep myself, my wife, my daughter, our many pets and the bank happy.
And to pay for PCB's, dev costs, etc.

I am planning to sell it once it is ready, the current stupid patent(s) already in place will kill any money I have, but a big company with money resources will easily overcome this issue.

I know of companies investing over $100 million and getting no where.

A few more details are:
The system needs to handle around 200+ million live connections, if I get every customer??
Using either one or many computers, that MUST be linked together.
Using windows and overlapped sockets I can get around 1 million per computer easily.
But when you close these connections (I had a few PC's testing this into one AMD FX-8320 8 core) the system pauses for a long time.
There is also the issue of memory and resource management.
Then there are other issues which slow me down.

If I use C10K as people have talked about
Then 200 million / 10,000 = 20,000 servers.
A big room, lost of cables, etc is required.

Me and my wife sold my wife cows, she was a dairy farmer in New Zealand.
This is the business I invested in, both time and money.
I has been sold, no turning back.

Next sell my Silverlight Framework, called xxxx, won't list it here as it is easy to search business directories and find me.
You will have to trust these words???

By the way, I have downloaded BareMetal.
It looks like what I am after.
Planning to strip it down and set it up the way I require.

Thanks for the link.
And thanks for the interest and comments.
It is great to be able to discuss low-level with people who know what they are talking about.

Cheers guys, Alistair

[gerryg400]

Hi,

You've made some very strange comments and I'm curious about why.

For example.

I am not after an OS, this would defeat the purpose. No hardware abstraction layers wanted.

I'm an engineer and this goes against everything I've learned and experienced in my career. What purpose is defeated by having an OS and abstraction ?

I would also like the source code and if possible written in assembly nasm.

It's unlikely you will achieve better performance than a modern compiler. So why use asm ?

Gerry

[tsdnz]

Hi Gerry, good points.

I'm an engineer and this goes against everything I've learned and experienced in my career. What purpose is defeated by having an OS and abstraction ?

Speed: One example is ethernet.
When the interupt is fired it can use the same or a preset ethernet structure.
Using HAL the data is copied to a buffer for recv and send.

It's unlikely you will achieve better performance than a modern compiler. So why use asm ?

Speed: I will use Visual Studio 2012 C++ to compile certain parts, and I will example and copy/change the asm as required.
Using __forceinline is nice, it makes good code.
But for pure performance you cannot beat assembly if you know exactly what you are trying to accomplish.

I hope this makes sense.

Regards, Alistair

[Kazinsal]

Using __forceinline is nice, it makes good code.

__forceinline doesn't work if the function you're trying to inline contains inline assembly.

VS has some quirks.

[tsdnz]

Hi Guys, downloaded BareMetal and Pure64.

Compiled BareMetal-OS-0.6.0 using
nasm -O0 -f bin -o kernel64.sys kernel64.asm

It compiled fine, but it does not run when added to the Pure64.img (Pure64 v0.5.0 VM for Windows)

Does anyone have any ideas?

[bluemoon]

One of the interesting thing I see is it runs 470 times faster than window software, and it is network related, probably a server.

However, running network intensive application on windows itself is a mistake.
Have you benchmark on BSD or solaris, that run at speed much more reasonable.

For the coping and context switch "introduced by OS" you worried, there was already some discussion on forum you can search for.
Simply put, modern OS do zero-copy and smart scheduling, there is little room for further optimization while you may well spend the development cost on OS for hardware upgrade or distributed solutions.

If I use C10K as people have talked about
Then 200 million / 10,000 = 20,000 servers.
A big room, lost of cables, etc is required.

No, you can use serval card with multiple head. like 6 cards with 4 heads per machine.
On modern OS with well designed application, in my own test, it can support up to 30k connection smoothly (each with 3 packet per second on a gigabit card, that's about the max bandwidth )

So, for 200 million concurrent user you may only need 277 servers.
Note that 200 million concurrent connection, each with 3 packet per seconds is a lot of traffic, you may as well redesign the protocol, turn to connectionless etc to reduce the traffic first.

[tsdnz]

Hi, thanks for your input.

For the coping and context switch "introduced by OS" you worried, there was already some discussion on forum you can search for.

Yes I am concerned with this, it will cost a few cycles.

Have you benchmark on BSD or solaris, that run at speed much more reasonable.

No I have not, never had an experience with these.
I have always developed using Windows.
Can you give me some links to check?
I will also google.

No, you can use serval card with multiple head. like 6 cards with 4 heads per machine.
On modern OS with well designed application, in my own test, it can support up to 30k connection smoothly (each with 3 packet per second on a gigabit card, that's about the max bandwidth )

I can easily get 100k connections using a single 1gb card.
Can reach 1 million, so will need 200 servers.

Note that 200 million concurrent connection, each with 3 packet per seconds is a lot of traffic, you may as well redesign the protocol, turn to connectionless etc.

Yes, connectionless is a possibility.
The devices connected will be using wireless sim cards.
I have the new sim cards that can be inserted directly into the PCB.
The devices will not talk much, at most once every 10 seconds.

Regards, Alistair

[bluemoon]

So, in my opinion, you need a well designed server cluster architecture and protocol, which can handle massive users.
Therefore, server application, but not a OS, which probably just boost 0.01% performance (more naturally a hobby OS get worst performance than mature OS, and bad stability).

The stats I stated is from my own game server, but I guess it's reasonable throughput for any serious project.
I recommend you do some stress test on FreeBSD, you get download it from freebsd.org

I can easily get 100k connections using a single 1gb card.
Can reach 1 million, so will need 200 servers.

With 100k connection on windows it's probably causing great impact on stability.
If the client-side is not talking much, each server should be able to handle 100k connection on BSD or even Linux.

[tsdnz]

Hi bluemoon,

Nice OS, looks great.

So, in my opinion, you need a well designed server cluster architecture and protocol, which can handle massive users.
Therefore, server application, but not a OS, which probably just boost 0.01% performance (more naturally a hobby OS get worst performance than mature OS, and bad stability).

Yes a well designed server cluster is what I want to design.
The system is only going to run one task on multiple cores.
I agree with the stability.
I was hoping to control everything, this way I could squeeze all the performance out that I could.
I love speed.

I recommend you do some stress test on FreeBSD, you get download it from freebsd.org

Thanks, checking this out now.

Also emailed BareMetal asking them for assistance.

Many Thanks, Alistair