The Skinny On NVM-Express Flash And IBM i

November 4, 2019 Timothy Prickett Morgan

A couple of weeks ago, buried in the October Power Systems announcements, we told you about the new NVM-Express flash cards that Big Blue brought to the Power iron and that are specifically enabled in the IBM i 7.4 release with Technology Refresh 1, or TR1. We didn’t have a lot of detail back then about the new drives other than some capacities and pricing, but we know more now and we are going to tell you what we know.

We will circle back with Big Blue and have a discussion about what the addition of NVM-Express flash means – and does not mean – for a database-centric system like the IBM i platform.

To recap so everything is in one place, in announcement letter 119-059, in addition to an 11-core Power9 processor feature and some new disk drives, IBM rolled out six NVM-Express flash adapter cards. These are not units that pit into an SSD slot, but rather that plug into PCI-Express 3.0 x8 slots in the server. There are NVM-Express ports that make use of an SSD style slot in some servers, but these ain’t it. NVM-Express, if you are not familiar with it, is a direct to flash protocol that knows how flash is addressed in its on-board controllers and does not try to pretend that it is a block device like a disk drive. Because it is speaking native flash memory addressing, NVM-Express delivers lower latency and higher throughput than SATA or SAS flash SSDs. These new PCI-Express flash cards are only available on Power9 systems, as such:

The 1.6 TB NVM-Express flash card for IBM i platforms is feature #EC6V, costing $3,099, for Power S914, Power S924, and Power H924 servers, while #EC6U, costing $4,059 is for the Power E980.
Feature #EC6X, costing $6,099, for the low-end machines and feature #EC6W, costing $7,989, for the Power E980 doubles up the capacity to 3.2 TB for IBM i platforms.
Feature #EC6Z, costing $12,099, for the low-end machines and feature #EC6Y, costing $15,848, for the Power E980 doubles up the capacity again to 6.4 TB for IBM i platforms.

As we pointed out before, the 3.2 TB and 6.4 TB flash adapter cards are not allowed in Power S914 systems with four cores or fewer. You can only get the 1.6 TB flash card.

Now, here’s some additional information about these flash drives that comes to us courtesy of IBM i offering manager Alison Butterill. Here are the things to consider as you ponder adding NVM-Express flash to your Power9 machines:

NVM-Express uses its own definition of namespaces, and in this case, a namespace is a collection of logical blocks whose logical block addresses range from 0 to the size of the namespace – 1. A namespace ID (NSID) is an identifier used by a controller to provide access to a namespace.
NVM-Express is able to provide higher performance then SSDs (as we said above)
NVM-Express will provide additional virtualization capabilities on Power9 machinery Every device is a PCI-Express endpoint that can be dedicated to an LPAR
NVM-Express devices are to be initially ordered in pairs, but advanced configurations will allow for an odd number of NVM-Express devices
The P05 platform is limited to two 1.6 TB devices only (again as we said above)
IBM i support for virtualized NVM-Express is through the Virtual I/O Server (VIOS)
NVM-Express devices are included as supported direct attached devices for Db2 Mirror

It is not clear who IBM’s NVM-Express flash adapter card supplier is, but our guess is that it is not Intel and it is probably either Samsung or Micron Technology – or a mix of the two so Big Blue has multiple suppliers. Given the performance specs shown below, which are different in some respects, it could be that the 1.6 TB adapter comes from one source and the 3.2 TB and 6.4 TB adapters come from another source.

The I/O operations per second (IOPS) are pretty good for these devices on both random reads and random writes, and are many times faster than what you can get on an SSD that is attaching through a SATA or SAS controller and has all of that disk drive fakery overhead. The data throughput is also a lot higher, which is what matters for a lot of applications. These new NVM-Express adapters can be used as load source storage, by the way. Linux systems have software-based RAID – with RAID 0, 1, 5, and 6 levels of coverage – that can be used to protect data on storage devices. Linux has had software-based mirroring for more than a decade as far as I know, and the overhead used to be pretty substantial but not crazy. Obviously, with only two devices, you are limited to RAID 0 striping (which makes them both look like one logical storage device) or RAID 1 mirroring (which keeps two copies of the same storage on the two devices). RAID 5 requires at least four devices as far as I know (three for data striping and one for parity data from the RAID 5 algorithm, but I usually see four devices plus one for parity) and RAID 6 does RAID 5 with a hot spare to restore a device automagically in the event of failure for a total of five or six devices.

IBM would love to sell you six NVM-Express flash drives to create a RAID group, I am sure, but remember, these things don’t hang off of storage controllers that have RAID algorithms built into their electronics, and neither AIX nor IBM i have software-based mirroring for disk or flash storage. It could be added, to be sure, but IBM’s position at the moment is to use high availability system mirroring software – what it refers to as OS mirroring in the chart above – to protect data stored on flash adapter cards. No one wants to buy two machines jam-packed with flash adapters, but backing up to a Power S922 sitting in the Microsoft, Skytap, IBM, or Google public clouds could do the trick. Especially if these are equipped with fast flash devices like IBM is offering here for customers.

One last thing: while we are very excited for NVM-Express over Fabrics connectivity, which allows for storage spread across a bunch of distinct machines to all look local to all of the servers in a cluster, it is only available on Power9 machines running Linux. At some point, when IBM gets tired of seeing its DS series of Power-based SANs decline in sales, it will create storage clusters based on Power-Linux iron using the NVMe-oF protocol and allow for flash to be disaggregated completely from servers and used like a modern SAN. That’s what a slew of startups are doing, and that is how the hyperscalers and cloud builders do it, too. There is no reason to keep flash tied to a specific server, but offering NVM-Express flash in any form is an improvement over not having it at all.

This Issue Sponsored By

Table of Contents

Content archive

Recent Posts

Subscribe

Pages

Search