OSDev.org

"Stock exchange" was the trigger-word for me in this thread.

During office hours, I am maintenance coder for a database that is keeping market data for a major German bank. That means, we get raw data from Bloomberg, Reuters, and a couple of other data vendors, store it away, do automated quality checks on the data, calculate derived and accumulated data, and provide it to other departments through a server architecture.

We're running a Sun E15k, with 16 CPUs (~800 MHz each), 64 GByte physical RAM, and a couple hundreds GByte of RAID'ed hard drive space in the SAN. Because the RDBMS we're using for static data (time series are kept in plain files) isn't up to the task performance-wise, we're keeping the data (almost 4 GByte) cached. Hard drive I/O is the limiting factor in the system.

And that's by no means all the data that would be available from the vendors - only what is used and required by the other departments.

Just to give a real-live example of size.

Very interesting. If that's how the market data if being munged I wonder how the main transaction database is affected.

I know that Seagate had a 'Barracuda' disk with two heads in the 1990's but the cost was prohibitive and still was stuck on a single interface channel.

Hitachi developed a multi channel storage system, but this was right before IBM figured out the super-GMR heads and boosted capacity into the multi-gigabyte range.

I have a copy of Hitachi's original designs, perhaps I should get the soldering iron out and see what burns. My background is in electrical engineering and the results can't be any worse than my attempts at programming ;D

-Crappy website
-Ridiculous claims that are way ahead of their time
-Small looking PC--yet 1TB memory?

Hoax for sure. I believe that there will be 1TB memory in consumer machines someday. I also believe that someday, everyone will own a supercomputer. But for now, that's just an impossible dream.

BTW, if you think 1TB is hard to fill, look at the rising megapixels in digital cameras, among other things. As technology gets better, and audio and video resolution/quality gets higher, the storage space needed will increase. As will the CPU power required. As those things increase, the I/O bottlenecks will need to be eliminated--which means, of course, that hard drives will have to become obsolete to make room for a new, faster, bigger, better storage solution. Which may just be, some day in the near or far future, 1TB solid-state memory. You just never know.

NotTheCHEAT wrote: As technology gets better, and audio and video resolution/quality gets higher,

You can only make images and sound as good as your ears or eyes will notice. Making images above 6 megapixels (about, 8 is a conservative limit) is pretty pointless since your eyes can't process more resolution. It's only useful for increasing size, but that's better done using analog methods (lenses etc.).

It's pointless storing more than you can use.

Which may just be, some day in the near or far future, 1TB solid-state memory. You just never know.

Yes, but not for the things you specify.

Think more of reducing things you accept as normal. When I'm seeking through a video, it needs some time to start speeding up, then it needs some time to wind down etc. A network stream ideally should be able to just play at 16x speed instantly, going back instantly as well.

Candy wrote: You can only make images and sound as good as your ears or eyes will notice. Making images above 6 megapixels (about, 8 is a conservative limit) is pretty pointless since your eyes can't process more resolution. It's only useful for increasing size, but that's better done using analog methods (lenses etc.).

Professional media editors will disagree. During postprocessing, that extra bit of resolution or sampling rate means you can do more without losing quality.

But basically I agree. Multimedia makes for the majority of net traffic, but in storage terms the real "data bombs" in terms of storage on a single machine are data warehouses.

APC Magazine in Australia (apcmag.com) benchmarked two hard disks this month (December Issue) - I quote -

<I>A great concern was the dramatic heat from both models (of hard disk) tested. Even compared to a high speed 10,000 rpm Western Digital Raptor, both the Seagate and Western Digital drives put out considerable amounts of heat - a large number of these drives in a rack will require a serious cooling solution. Though specific numbers weren't recorded, the Western Digital drives became too hot to touch after a strenuous workout, while Seagate's were slightly cooler. </I>

These hard disks also had an average transfer speed of 29 Mbytes/second compared with SATA's 150Mbytes/Second limit. Plenty of space not being used (or to grow into depending on your optimism).

So whilst hard disks are getting bigger to take all that data, there might also be a problem of reliability. CPU's are hot enough, add couple of boiling hard disks and all those pixels might start running out.

@Solar, does that storage for the database server require any special cooling requirements? Like 'fan racks' between 'storage racks'? I'm wondering how much of the cost of maintaining a large storage system (cooling, environment) might contribute to the overall cost of the entire system.
The information stored on any system is of course worth a lot more than the system itself.



On pixels - Imagine taking a 6MP picture and printing it on a massive advertisement billboard. That's a lot of interpolation and a lot of wierd pixelation/colour loss. Digital SLR's now come standard with 9 megaixels (Fujitsu comes to mind) and are definitely going to increase. I don't think that camera manufacturers will never say 'Oh 20 MPixels, that's enough quality for everyone, We'll just stop here.'

On network streaming - all of those streams have to come from somewhere. HD video + 7.1 surround sound takes up a lot of space even with compression.

On solid state memory - still fails my previously mentioned point. (This includes the much taunted holographic memory).
All are serial devices incapable of providing multiple pieces of information simultaneously without buying wads of duplicate hardware or putting up with an I/O bottleneck.

Xardfir wrote:@Solar, does that storage for the database server require any special cooling requirements? Like 'fan racks' between 'storage racks'? I'm wondering how much of the cost of maintaining a large storage system (cooling, environment) might contribute to the overall cost of the entire system.

No idea. It's in the SAN somewhere, i.e. not even in the same physical location as the server. But as far as I know, storage is handled very conservatively here - i.e. loads of 9 GB disks in vast RAID arrays. I'd guess that doesn't require exceptional cooling.

On pixels - Imagine taking a 6MP picture and printing it on a massive advertisement billboard. That's a lot of interpolation and a lot of wierd pixelation/colour loss. Digital SLR's now come standard with 9 megaixels (Fujitsu comes to mind) and are definitely going to increase. I don't think that camera manufacturers will never say 'Oh 20 MPixels, that's enough quality for everyone, We'll just stop here.'

This I would have to disagree with. At the distance advertisement billboards, or even small posters, for that matter, are seen from, a 6MP source picture looks just fine. Look at some posters, and look how big the individual pixels on them sometimes are.

Hiya,
Ok I get that. What about from a reverse perspective.
Your new 6Mpixel camera does only 72DPI based on a 1600x1200 CCD matrix. Your printer can do 600DPI or quite a lot more. The printer has to do a lot of interpolating which is a lot of extra 'non existant' information.

This is the advantage film still has over digital. The film can be 'blown up' to any size without introducing artifacts. Sure signal processing can improve digital data (I dabble in satellite image processing) but the result is useless unless you have the resolution in the first place and of course the space to store it
(a few terabytes would be welcome). The image you view on paper (forget 'video displays' as they are worthless for detailed images due to the low 'DPI' viewing factor) will always have unoriginal additions due to the low DPI of the digital camera.

This is also why a lot of magazines recommend you buy a digital camera with the highest 'optical resolution' possible. Digital zoom is rotten for any kind of good imaging as it isn't based on the physical CCD capabilities.

Xardfir wrote: This is the advantage film still has over digital. The film can be 'blown up' to any size without introducing artifacts. Sure signal processing can improve digital data (I dabble in satellite image processing) but the result is useless unless you have the resolution in the first place and of course the space to store it
(a few terabytes would be welcome). The image you view on paper (forget 'video displays' as they are worthless for detailed images due to the low 'DPI' viewing factor) will always have unoriginal additions due to the low DPI of the digital camera.

My point in question was, you can only see 6MP with your eyes - provided that you look at the full image. When you're looking at somebodies toes in a picture of a football stadium, you're going to need quite an expensive camera for that to give a good result. On the other hand, nearly all pictures are only for a few views, and at a size in which you don't focus on one part of it. In all those pictures, >6MP is a pure waste.

This is also why a lot of magazines recommend you buy a digital camera with the highest 'optical resolution' possible. Digital zoom is rotten for any kind of good imaging as it isn't based on the physical CCD capabilities.

Digital zoom is removing the edge (3/4...8/9th) part of the image and scaling the rest. Pure crap, you could say. But, with 54MP camera's, who's (referring to non-photo-pro's here) going to notice?

PS: satellite image processing? Where do you work?

Hey. I understand what you mean in general. Your facts are a bit incorrect, though. First (and this is the one I'm not sure in), wouldn't a digital camera with a 1600x1200 CCD matrix be marketed as being 1,92 MP? Second, a digital camera, or the pictures it has made can't be classified by their DPI, unless you have set a rule for yourself that every one of your photos must be printed out with constant measurements. A digital photo can be printed out at any size, shown at any size, and besides, if someone has changed the DPI of the image file (and stripped the metadata coming with it) you can never be sure what it was originally shot with, since for all you know, it could be cut or resized. I suppose you could calculate the DPI value for the CCD, but it would be pointless, due to reasons you might figure out yourself. Third, film isn't exactly an analog data carrier. Film can be classified by it's DPI, and it is what ISO/ASA ratings essentially are. So you can't exactly "blow up" film to any size.

I'd be interested to know where you work, too.

Eero R?nik wrote: I'd be interested to know where you work, too.

At the moment at Astron (www.astron.nl), going to Topic (www.topic.nl) in February, so I wanted to know where he worked since I'm in satellite image processing (earlier in the chain however, still at the signal processing bit) section too.

Hiya,
@Eero:
To my understanding Megapixel = the image size of the shot taken, so at 1600x1200x32bpp that would equal a 7.5 Megapxel camera. I have an old one that does a 640x480 but says it is only a 1.3 megapixel camera having only 8 bit Analog to Digital Converters. So the 'megapixel' size varies as a function of the Analog to Digital Conversion.
Sure there is a physical limit to 'blowing up' images, but my reasoning is that it is better to have as much information as possible in the 'capture' stage, whether it is for personal photography or something larger. Film grain sizes go down further than CCD's can currently handle. I'm sure in the future they'll catch up and then I'll change my tune.
@Candy:
The quality is totally dependant on the size of the CCD.
Take a CCD of 100x100 (A really bad camera!). Using a 2x digital zoom all of the image you see is based on light coming in on a 50x50 square. 4x = 25x25 square. As a result, there is a huge amount of interpolation. Going to a 7 megapixel camera as mentioned above, at 52x digital zoom, your photo is going to be based on a 32x23 matrix. Yechh! I'd assume that when using a 52x digital zoom, your camera will extend itself to it's full optical zoom before even attempting to take a photo. So the quality still comes down to the optical zoom.

Where me work? Anywhere that will pay me .
Seriously. I started a business (Australian Central Processing Unit Pty Ltd) in the mid-90's when I was studying at technical college to earn those needed bits of paper . Some archival work with the State Library's geography department sparked an interest in how it's all done. That let to some work using the CSIRO's microBRIAN imaging system (image representation, geometry, statistics, transformation's etc).

I went back to college last year to finish an eletrical degree and have only just got the company up and running again fully in the last 3 months, partially due to the posts on this thread. Remember a few posts ago I made the comment that I had Hitachi's original designs around somewhere for a multi-channel hard disk and that I should get my soldering iron out? I contacted Hitachi and am currently set to patent some novel additions to their storage device with the intention of licensing the design out to whoever will pay me. (My soldering is a LOT better than my programming!) Of course it'll take a couple of years to actually sell anything (non-disclosure, non-commercial use etc.etc.)

Xardfir wrote: To my understanding Megapixel = the image size of the shot taken, so at 1600x1200x32bpp that would equal a 7.5 Megapxel camera.

Color depth of a pixel doesn't count against the number of pixels. 1600x1200 is about 2 megapixels. My not-too-new Powershot A95 is a 5 megapixel cam (2592 x 1944).

I have an old one that does a 640x480 but says it is only a 1.3 megapixel camera...

0.3, more likely...

...having only 8 bit Analog to Digital Converters. So the 'megapixel' size varies as a function of the Analog to Digital Conversion.

I doubt that...

Sure there is a physical limit to 'blowing up' images, but my reasoning is that it is better to have as much information as possible in the 'capture' stage, whether it is for personal photography or something larger.

Correct. Same for audio processing - 16/44 samples are fine for making into a CD, but as long as you're still applying filters and such, you want much more fine-grained input information.

Hiya,
I was digging through the shed last night and found the manual for the old camera - apparently it has a resolution of 1280x1024 which accounts for the 1.3 megapixels.
This led me to look up a few sites to see exactly what was going on and I found some interesting info. It turns out that the Analogue to Digital conversion is camera dependant and that is what impacts on quality. Obviously whilst my old camera has good pixel resolution, the conversion is so bad as ro render the fine resloution unusable (it looks to be about 256 colours most of them brown or green).

Your Powershot, Solar, appears to have a proper 32-bit RGBG conversion (as you'd expect to get from a 5 megapixel camera).

I guess it comes down to 'you get what you pay for', and in the future I'll be asking for the 'conversion encoding fomat' not just resolution (as if the salespeople don't get nagged enough already ).

So getting back to my previous rambling and the point raised by Eero about image classification, I believe that you CAN classify the pictures you have taken by their DPI, dependant on the intended application. A good example of this principle already in wide use would be flatbed scanners (same technology, different application, about 10 years more mature though.) Now that the price and resolution (compared to the early versions) of scanners has gone down, no-one would really care what resolution a scan is done in, but what the intended output will be. A 75 dpi scan for OCR work, 360 dpi for line graphics, 600 dpi for photos, 1200 dpi for professional production.
When was the last time anyone asked their scanning software for an image exactly X by X pixels in size? The scanning software just sucks it in and selects an arbitrary image size after the scan.
The same I believe will happen (and is happening) with digital cameras. Who cares about the resolution, what is the picture to be used for?
75 dpi for the web, 300 dpi basic printing, 600-1200 dpi for photo printing, 1200+dpi professional photography/print material/ posterboards. So long as the file is stored in the correct format (the old JPEG vs. TIFF format question) everything will turn out as expected.
Any thoughts on this.

@Solar: I was wondering if I may quote your description of the storage system described earlier as a 'case study' on the website being set up for the storage device my company is trying to 'get off the ground'. I'll add a hyperlink to this forum as the original source of information, perhaps it will generate a few interesting visitations in the future if anyone dares to click it.