IBM Gooses Power Systems Storage and Networking

October 24, 2011 Timothy Prickett Morgan

As we discussed in last week’s issue, on October 12 IBM rolled out some new entry and enterprise Power7-based systems with double the maximum memory capacity and with support for PCI-Express 2.0 peripheral cards, doubling up the bandwidth per slow. But there were a bunch of other enhancements to storage and networking adapters, too, and these are useful for all Power Systems shops on reasonably recent machinery.

The new SAS disk and flash drive controllers are the most interesting new devices that IBM put out in the October announcements. With solid state drives now coming into vogue to boost the I/O processing capacity of systems, you have two options. One, you can put a relatively dumb disk controller in the system and let the central processor handle most of the heavy I/O coming through flash drives–I have seen this in a flash-based supercomputer called Gordon that is being built at the San Diego Supercomputer Center. (So I ain’t just making this up.) Or two, you can put more cache memory on a fatter and more capable disk controller to handle the increasing amount of I/O that an SSD can push compared to a similar number of disks. Given the relative cost of processors, I can see why you wouldn’t want to do a lot of I/O processing on a Power7 machine and would want to offload as much as possible to a smart peripheral card. (And one most likely based on a PowerPC processor, in this case, if I were IBM.)

The new feature 5913 large-cache SAS controller plugs into the PCI-Express 2.0 x8 slots that are available on the new Power 710, 720, 730, 740, 770, and 780 machines that were announced on October 12 and that started shipping on October 21. It will also be available for older Power7 and Power6+ and Power6 servers through EXP12S or EXP24S external I/O drawers. This feature 5319 controller is a full-height, single-wide controller that allows customers to mix and match hard disks and SSDs on the same controller and offers three times the SSDs per slot and two times the disk drives per slot as prior SAS disk controllers; it also packs 1.8 GB of write cache, which helps boost system performance. The 5319 controller offers 6 Gb/sec SAS links to disks and controllers, which can handle data transfer rates of up to 4GB/sec; a pair of them can handle 72 disk drives, but you have to get the right enclosures to max it out. Some enclosures top out at 42 or 50 drives.

Here’s how the new large-cache SAS adapter for PCI-Express 2.0 slots stacks up to the two other key SAS adapters, which plug into PCI-Express 1.0 slots (and are therefore compatible with the Gen 2 slots as well):

Interestingly, the feature 5319 controller also has flash memory as its cache, so it does not need to be backed up by memory as the prior generations of cached disk controllers in the Power Systems lineup required. So now, you don’t need to go mucking about changing the batteries on the controllers when they drop dead, and if a system goes haywire, the flash memory on the controller will hold the data until the power comes back on. As has been the case for many years, IBM recommends that disk controllers be paired on mission-critical back-end systems where data loss cannot be tolerated.

The feature 5319 controllers cost a little more than three times that of the feature 5805 and 5903 controllers they replace, so having that I/O expansion does not come cheap. But at many companies, PCI slots are scare and storage capacity and I/O bandwidth demands are growing a lot faster than CPU capacity requirements, which is why IBM had to make this beefier controller and move to PCI-Express 2.0 peripheral slots. Moreover, IBM warns customers that the feature 5805/5903 controllers were really designed for disk drives and will get bogged down once you have three to five busy SSDs hanging off them; if the SSDs are not particularly busy, you can probably hang eight or nine of them off one of these older PCI Gen 1 controllers.

You may be using the old feature 5904, 5906, and 5908 PCI-X large-cache adapters in your Power Systems machines, which are very respectable as it turns out. But the new controller is still better and cost less money, as you can see:

And, as with the older PCI Gen 1 SAS controllers, this PCI-X controller was not designed to take the heavy I/O load of SSDs. It is also a double-wide card and you cannot mix and match disks and SSDs on the old PCI-X controllers. The other thing to consider is that read cache is basically useless because SSDs can just pipeline data straight through the controller into the system bus at full tilt boogie. You don’t have to compensate for the slowness of a disk drive. And IBM says that for customers using disks with this fat memory PCI Gen 2 controller, don’t worry about the lack of read cache on the controller unless you are getting more than 35 to 40 percent hit rates.

In addition to the new large cache SAS adapter, IBM put out a slew of new PCI-Express 2.0 adapter cards to take advantage of that extra I/O. Feature 5729 is a four-port 8 Gb/sec Fibre Channel adapter card for linking out to SAN storage; this card has twice as many ports as the previous PCI Gen 1 adapter and costs 14 percent less, too. On the entry Power Systems machines (by which I mean the Power 710, 720, 730, and 740 machines), this new card costs $6,000; on the new enterprise models (that’s the new Power 770 and 780 boxes) it costs $7,859.

There is also a new Quad Data Rate (QDR) InfiniBand networking card, which runs at 40 Gb/sec; that’s feature 5285 and it is a two-porter that costs $1,000 on the new entry machines and $1,310 on the enterprise boxes.

IBM also kicked out a bunch of new Ethernet adapters, which again double up the port count over their PCI Gen 1 predecessors. Feature 5287 and 5288 are two-port Ethernet adapters running at 10 Gigabit Ethernet speeds; the feature 5287 card is more expensive because it uses SFP+ cables. On the entry machines, the feature 5287 costs $3,000 and on the enterprise boxes, you’re in for $3,930. Feature 5288 uses cheaper SFP cabling and costs $1,600 on the entry machines and $2,096 on the enterprise boxes.

IBM has also kicked out new low-profile features that put two 10 Gigabit Ethernet and two Gigabit Ethernet ports on a single card; this device is only supported with the Linux operating system at the moment and plug into the new entry Power Systems machines. Feature 5279 comes with SFP+ cables on the 10 GE ports (it costs $2,499), while feature 5280 has two optical ports (and costs $3,499). Feature 5287 is a two-port, full height 10 GE card for PCI Gen 2 slots that has SR cables (it costs $3,930) while feature 5288 has two 10 GE ports using SFP+ cables (it costs $2,096).

And last but not least, IBM is doubling up the capacity on the 2.5-inch SAS disks spinning at 15K RPM for IBM i, AIX, and Linux. Feature 1948 is a 300 GB disk formatted down to 283 GB for the IBM operating system that fits into first-generation 3 Gb/sec SAS bays, while feature 1879 is the same capacity but plugs into 6 Gb/sec SAS slots. Either way, the disk costs $950 in entry machines and $1,244 in bigger boxes. If you want them formatted for AIX or Linux, that’s feature 1953 or 1880 (for the different small drive form factor slots) and they cost depending on the machine size.