Optane vs. StoreMI vs *sane* tiered caching approaches

[Schol-R-LEA]

I know it's just one data point from a single (and somewhat questionable) source, but this is not a good look for either Intel or AMD, if their hardware solutions are getting thrashed by a generic software one.

I am not so sure if Mr. Sebastian's enthusiasm for the 'PrimoCache' software is all that justified, though.

In any case, all of these are dealing with the inherently broken mess that is Windows drive management (not that Linux is any better, of course), which means this comparison is like a car race in which each car has one wheel that has been locked with a wheel clamp.

[frabert]

To be honest, from the perspective of someone who has not done any research on how Optane and StoreMI are implemented, I would be tempted to say that to me, all of the three solutions are actually software-based, with the difference being that the AMD and Intel ones are vendor-locked with some kind of motherboard-model check...

[Schol-R-LEA]

To be honest, from the perspective of someone who has not done any research on how Optane and StoreMI are implemented, I would be tempted to say that to me, all of the three solutions are actually software-based, with the difference being that the AMD and Intel ones are vendor-locked with some kind of motherboard-model check...

Well, firmware, at any rate, that's true.

With Optane, part of the difference is that it only works with an Intel Optane memory; IIUC, it doesn't work with anything else, including any of Intel's own NAND-flash SSDs. I'm not even sure if it works with their 3D XPoint SSDs which aren't specifically meant for that purpose.

StoreMI, OTOH, will work with any SSD (or maybe it is just any m.2 NVMe SSD; I'm not sure if it works with SATA SSDs but I would think that doing that would somewhat defeat the point given that SATA latencies are something like an order of magnitude higher than NVMe latencies). There are a number of limitations, such as the 256GiB maximum size for the fast tier drive, and from what this video and some others I've seen say, the algorithms used appear to be far inferior to those used by Optane's firmware and support software.

On the gripping hand, the main selling point of this PrimoCache system seems to be that it does everything without any specific hardware or motherboard firmware. I am guessing that it is meant more for small-enterprise systems rather than gaming rigs, as it sounds as if it is a scaled down version of the sort of tiering software used in server farms, but without the third or higher tiers or the ability to talk to the load-balancing systems for the server farm as a whole.

(IIUC, third tier is usually on-site NAS drives, while fourth and so on would be various types of offsite storage - which might or might not be described as 'cloud storage' ugh by people with more marketing knowledge than computing knowledge - and can involve nearly any sort of high-capacity and/or high-fidelity machine-readable storage, up to and including mainframe-style tape archive libraries, which are still used by some major firms for their considerable stability compared to most other media. There a number of different configurations possible, though.)

[Brendan]

Hi,

To be honest, from the perspective of someone who has not done any research on how Optane and StoreMI are implemented, I would be tempted to say that to me, all of the three solutions are actually software-based, with the difference being that the AMD and Intel ones are vendor-locked with some kind of motherboard-model check...

Well, firmware, at any rate, that's true.

I haven't looked into it either; but (at least for server grade equipment) Intel has a history of doing "data transfer engines" in hardware (e.g. the "Intel Corporation 5520/5500/X58 Chipset QuickData Technology Devices" in my aging workstation) where DMA/bus mastering is used for shifting data around between devices (and between devices and RAM). I wouldn't be surprised if AMD did something similar (and wouldn't be surprised if this is why Intel's and AMD's solutions require specific/recent chipsets); and that PrimoCache achieves similar transfer speeds by using more CPU load than either of Intel's or AMD's solutions.

I also think that the approach used by PrimoCache is "very different" to the others (and not directly comparable for that reason). For example, if you had a 512 GiB SSD and a 512 GiB "rotating disk" hard disk; I'd expect Intel's and AMD's solutions would let you store 1 TiB of data (because it's not a cache), and I'd expect that PrimoCache would only let you store 512 GiB of data because it is a pure (write-through) cache. Based on that assumption I'd also expect a major difference in write performance (e.g. writing to SSD only vs. writing to both and being limited by the slowest).

For what it's worth; my solution (for my OS design) is to do it at a higher layer (virtual file system layer) and not at a lower layer (storage device layer); such that all native file systems are mounted on top of each other at the root of the (distributed) virtual file system, and the VFS can "intelligently (in theory)" manage space vs. performance vs. redundancy trade-offs with a very varied mixture of underlying storage devices at different locations (remote vs. local). Unfortunately my solution "requires" a versioning file system to avoid major synchronisation and fault tolerance issues - it's not something that can be retro-fitted on top of typical operating systems. It'd also be a lot slower for "cold cache directory look-up" (VFS has to ask all online file systems for directory info and merge the results, and can't just ask one file system) which is something I'm planning to mitigate with a "distributed FS sync" stage during boot (which would tend to pre-heat the local VFS cache).


Cheers,

Brendan

[Schol-R-LEA]

AFAICT, Optane accelerators are dedicated cache; you can't use them as stand-alone drives at all.

[Brendan]

Hi,

AFAICT, Optane accelerators are dedicated cache; you can't use them as stand-alone drives at all.

As far as I can tell, for optane hardware there's 2 different categories:

• "optane SSD", which just plugs into a standard M.2 slot or PCIe slot, uses NVMe, doesn't need special motherboard support or special drivers, and (other than performance characteristics) are no different to any other SSD that uses NVMe. This is what was being used in the video (the "M.2 slot" version aimed at desktop, not the PCIe version aimed at enterprise).
• "optane non-volatile RAM", which hasn't been released yet, but would plug into DDR4 slots and would probably show up as "non-volatile RAM" in the physical memory map the firmware gives the OS. This would use the same "driver" as normal (volatile) RAM - CPU would just read and write to it directly

How either of these are used by the OS depends on what software you've got and how you feel like configuring it.

Also note that (as far as I know) for the "optane M.2 SSDs" that are targeted at desktop users; currently Intel only has 2 sizes (16 GiB and 32 GiB). I wouldn't be surprised if this is the only reason Intel's marketing department decided to pretend that they're "accelerators" (to speed up larger drives) - otherwise almost nobody would buy them because they're too small for most people to use as stand-alone drives by themselves.


Cheers,

Brendan

[Schol-R-LEA]

AFAICT, Optane accelerators are dedicated cache; you can't use them as stand-alone drives at all.

"optane SSD", which just plugs into a standard M.2 slot or PCIe slot, uses NVMe, doesn't need special motherboard support or special drivers, and (other than performance characteristics) are no different to any other SSD that uses NVMe. This is what was being used in the video (the "M.2 slot" version aimed at desktop, not the PCIe version aimed at enterprise).

Well... maybe, but that isn't what I've understood reports about it to be saying. They are SSDs, yes, but my impression is that they can only be used as accelerators - they won't show up as drives and you can't configure them for use as stand-alone drives, or at least, that was what I thought was the case. I am pretty sure Intel said that it wouldn't when it was rolled out, but I imagine that even if they did lock it out like that, someone found a way to get around that lock-out.

This stands in contrast to SmartMI, which just uses standard NVMe NAND Flash SSDs, period.

The test in the video shows that the Optane accelerator does show as an SSD on an AMD motherboard, but I had the impression that a) it doesn't show up as one on Intel boards, and b) it isn't supposed to work with AMD motherboards at all; but Intel didn't lock it down as far as they had meant to.

Why would they do something so mindless, you ask? Well...

Also note that (as far as I know) for the "optane M.2 SSDs" that are targeted at desktop users; currently Intel only has 2 sizes (16 GiB and 32 GiB). I wouldn't be surprised if this is the only reason Intel's marketing department decided to pretend that they're "accelerators" (to speed up larger drives) - otherwise almost nobody would buy them because they're too small for most people to use as stand-alone drives by themselves.

My impression is that the accelerator exists primarily as a) a stopgap to cover up the delays in getting the 3D XPoint NVRAM - the actual product for which Optane (a name which only refers to the accelerator system, AFAIK, not the NVRAM or the SSDs themselves) is the preview - out the door; and b) to get people to pay for the privilege of beta-testing a new technology which is having more teething pains than they want to admit. The general consensus seems to be that they never intended to make SSDs based on 3D XPoint; it was solely intended as an NVRAM implementation, as a drop in replacement for (again, IIUC) the entire DRAM subsystem (they expected it to be fast enough for that by now, I guess).

The fact that they put out the 'accelerators' at all, and the way they are now hedging the issue of 'all-NV memory' by saying that it was always meant to be paired with faster-but-volatile DRAM, is being interpreted as a sign that Crosspoint isn't quite working out according to plan, or that the leaks about it a few years back caught them off-guard and they hoped to have more time to get it working before announcing it, or more likely still, that another manufacturer's claims that their MRAM (another NVRAM implementation) would reach the market in 2016 (which wasn't true, it still isn't even close to ready and the company got a lot of flak for indicating that it would be) caused them to jump the gun.

[Brendan]

AFAICT, Optane accelerators are dedicated cache; you can't use them as stand-alone drives at all.

"optane SSD", which just plugs into a standard M.2 slot or PCIe slot, uses NVMe, doesn't need special motherboard support or special drivers, and (other than performance characteristics) are no different to any other SSD that uses NVMe. This is what was being used in the video (the "M.2 slot" version aimed at desktop, not the PCIe version aimed at enterprise).

Well... maybe, but that isn't what I've understood reports about it to be saying. They are SSDs, yes, but my impression is that they can only be used as accelerators - they won't show up as drives and you can't configure them for use as stand-alone drives, or at least, that was what I thought was the case. I am pretty sure Intel said that it wouldn't when it was rolled out, but I imagine that even if they did lock it out like that, someone found a way to get around that lock-out.

This stands in contrast to SmartMI, which just uses standard NVMe NAND Flash SSDs, period.

Here's one review using Intel's "M.2 Optane SSD" on an Intel motherboard with Intel's software (that was all provided by Intel themselves); that says (at the top of the 2nd page):

"Since Optane Memory uses NVMe as its underlying protocol, an Optane Memory module will first appear as a small SSD in Windows' Disk Management utility."


Cheers,

Brendan

[Schol-R-LEA]

Ah, I must have misunderstood, then. Thank you for the correction.

However, it is still my understanding that they really aren't looking at this as a replacement for NAND Flash; the real goal is to replace DRAM, or at least, to create a non-volatile tier of Random-Access Memory, rather than a have faster type of drive. The SSDs are less a product and more a testbed that they've tricked consumers into buying.

[Brendan]

Hi,

However, it is still my understanding that they really aren't looking at this as a replacement for NAND Flash; the real goal is to replace DRAM, or at least, to create a non-volatile tier of Random-Access Memory, rather than a have faster type of drive. The SSDs are less a product and more a testbed that they've tricked consumers into buying.

There are some people (e.g. HP's "The Machine" project) that want to replace all volatile RAM with non-volatile RAM and implement a "persistent object store" OS. I don't think Intel is one of these people - the reality is that "non-volatile" is useless for most of the things RAM is used for (and becomes more of a security risk than a benefit), and nobody has any non-volatile RAM that matches the speed or price of DRAM either.

Instead, I think Intel was aiming for something more practical - augmenting DRAM with Octane so that the physical address space contains both (partly to allow more creative uses than just storage, and partly to get much higher bandwidth). Of course they've been working on 3D Xpoint for about 6 years now and still haven't released any memory modules, so it might be a case of "aim high, fall short, change plans and sell it as something else".


Cheers,

Brendan