OSDev.org

I really think that in order for us to give meaningful advice, we will need a lot more detail about the requirements - not what you are planning to do, but what you are trying to accomplish. Otherwise, we are probably looking at a shoe or bottle problem and can't really solve the actual issue - framing the question in a way that presupposes a set of possible answers becomes a problem in and of itself, especially if none of the presented 'answers' is workable.

• Why would this need to be written from the ground up? Does it require an RTOS, and if so, would an OTS solution such as eCOS or Qnx suffice?
• What kind of equipment is this to be part of? Does it have to be an embedded or single board system, or can it be a stand-alone desktop which communicates with the data source? What hardware communication protocol would it need (USB, 10/100/1000BaseT Ethernet, RS232, some sort of internal bus, etc.)? Does the data need to be transferred as a single unit or stream, or can different parts of the data flow be handled by different data streams (or in other words, does all of the data have to pass through a single 'pipe', or can the data be sent through, say, three or four different USB cables routing to different CPUs with their own memory buses)?
• What is the nature of the data being collected, and will this software need to analyze the data or just record it? How are drop-outs or other lacunae in the data stream to be handled?
• Where do you anticipate bottlenecks (computation, memory bus, cache, communications bus, secondary storage) and have you done any analysis of the dataflow to back these expectations up?
• Is the data to be saved to secondary storage locally, and if not, would it need to persist in some other stable manner (e.g., NVRAM)?
• Why would it need to handle such a volume of data at one time, and how can the data be organized to reduce the immediate memory load (either for the system as a whole, or for the individual cores or memory subsystems)? If you go with a multiprocessor system, can the data load be divided between different processors in such a way as to minimize inter-processor communication overhead?
• What sort of information needs to be presented to the operator, and how?

This is just scratching the surface; I expect (well, hope) you have already worked out the answers to most of these questions, but if you want help from us, we would need to know what you came up with about them, and what you still need to figure out.