Goosing Big Iron Power Systems With Power9 Migrations

December 3, 2018 Timothy Prickett Morgan

The Power9-based servers from IBM’s Cognitive Systems division have been rolling out over the course of the past year, and the big iron has been in the field only since the late summer but has perhaps had the largest impact on the revenue and profit stream for the Power Systems line, excepting maybe the installation of the “Summit” and “Sierra” supercomputers for the U.S. Department of Energy’s Oak Ridge National Laboratory and Lawrence Livermore Laboratory.

As has been the case since the AS/400 line debuted in 1988 and even with the combination of the System/36 (low-end and midrange) and System/38 (high-end) minicomputers that predate the AS/400, generally speaking the little iron gives IBM the volumes it needs to have a diverse and large customer base while the big iron is what gives Big Blue bigger chunks of money for more sophisticated and scalable systems.

The Power E980 system, which is a scale-up NUMA machine that supports up to 16 Power9 processors in a single system image, replaces the Power E870 and Power E880 machines that have a similar architecture. We discussed the “Fleetwood” Power E980 system way back in October 2017 based on some internal information we got our hands on. We covered their launch in August of this year, and then subsequently drilled down into the Power E980 a lot deeper. Now, we turn to the upgrade paths that companies have to move from older Power Systems big iron based on Power7, Power7+, and Power8 processors to the “Cumulus” Power9 chips. There are a lot of different things to consider, and I talked about many of these two months ago in a feature called Riding The Upgrade Cycle, and had to constantly remind myself that the Power Systems base, and especially the portion of it running IBM i as the main platform, really does migrations – shifting out one box and putting in a new one – and does not do upgrades. I said back then that I was trying to get my hands on upgrade/migration comparisons, and IBM has provided me with some data for big iron and is working on some additional data on little iron for next week’s issue.

The point about the big Power Systems iron is at least there are upgrade paths with serial number preservation – which is important for allowing the machine to be depreciated on its original schedule – unlike with the low-end and midrange Power Systems machines. This wasn’t always the case in the past, when even small AS/400 machines could be upgraded with a processor and memory swap. But upgrades are a hassle and are only warranted when the machines are very expensive and the components removed hold such substantial value that it cannot be thrown away. With big Power9 machines costing many millions of dollars, companies have to be able to preserve their investments and not wreck their balance sheets and budgets, which is why IBM still offers upgrades for the big iron.

Before we get into the migration paths, it is probably a good idea to talk about what IBM i versions and releases are supported on what Power processors. Having new hardware that can’t run older operating systems does not do some customers any good at all because either they can’t or won’t (for economic or technical reasons) update and recertify their applications on newer releases. All of the shiny iron and blinding bang for the buck in the world can’t dislodge them from their older iron, unless IBM had emulation modes for the older i5/OS and IBM i operating systems within the newer ones. This could be done, and if IBM could make a lot of money supporting it, you can bet it would do it. It may be technically difficult and economically impractical, and Big Blue is still, after three decades, a little gun-shy after System/36 and System/38 customers were annoyed by the initial poor performance of the emulation environments on the initial AS/400s. (This is something worth reconsidering, I think.)

In any event, here is the table of Power processors and their systems and the IBM i operating systems that are supported on them:

As you can see, if you want to move to a Power9 machine, regardless of make or model, you need to move to either IBM i 7.2 or IBM i 7.3.

For those who want to upgrade their machines – meaning preserve their serial numbers and swap out Power8 system components to Power9 parts as opposed to doing a new Power9 machine install and then find a new home for the old Power8 iron – IBM is offering such upgrades, as it promised to do way back in July 2017. Here are the upgrade paths:

Since the Power5 big iron machines back in 2005, the enterprise-class machines have been built using up to four nodes of NUMA servers that are lashed together with high speed interconnects. The Power 595 based on the Power5 from 2005 and the Power 795 based on the Power7 from 2010 were the two last big NUMA machines that used book packaging midplanes that were inspired by IBM System z mainframes, and now IBM does not have to push up to 32 sockets using these mainframe-like NUMA machines. The NUMA chipsets that were external to the processor with the Power5 chips were pulled onto the die with the Power7 chips, and have been subsequently enhanced. The X-Bus on the processors is used to link Power chips inside of one of the enterprise-class enclosures to each other, and the A-Bus links are used to hook multiple enclosures to each other.

The enterprise-class machines based on the four node architecture can now, thanks to the performance increases with the Power8 and Power9 chips, do a lot more work than even these big bad boxes with twice as many sockets back in the days of the Power 595 and Power 795. The on-chip NUMA interconnects are particularly good with the Fleetwood Power E980 machines based on the Cumulus variants of the Power9 processors, as we discussed back in August. The intranode X-Bus NUMA ports on each of the Cumulus Power9 chips runs at 16 Gb/sec, which is twice the bandwidth of the Power8 chips, and the A-Bus links on these big Power9 processors run at 25 Gb/sec (the same as Bluelink and NVLink ports on the Power9 chips because they are the same underlying electronics), which is four times the bandwidth of the A-Bus links on the Power8. Any of the processors within an enclosure are only one hop away on the X-Bus, and any chip can talk to any other chip in a Power E980 with two, three, or four enclosures in two hops. We think the latency is a little lower on the A-Bus links, too, but have seen no data on this, and that should help smooth out the performance of NUMA clustering for machines with more than one enclosure.

Here is how the performance shakes out for the Power E980 on the Commercial Performance Workload (CPW) transaction processing test for different SKUs of the system:

Depending on the frequency and core counts on the Power9 chips, a single node of the Power E980 system delivers anywhere from 508,900 CPWs to 687,500 CPWs, and fully loaded four-node Power E980 machines have north of 2 million to 2.7 million CPWs. The CPW benchmark may not stress systems as real-world applications do, mind you, but the scalability across the nodes is pretty close to linear and within the nodes it is as well. (I think the L4 cache memory embedded within those “Centaur” memory buffer chips used with the Cumulus Power9 processors is helping quite a bit, too. And it is a wonder that more companies don’t use buffered memory. But with the DDR5 generation of memory, buffers are going to be moving onto the memory sticks for the whole industry and maybe L4 caches will at some point, too.)

There are a bunch of different ways to qualify the performance of the Power E980 compared to its predecessors. You can look at per-core performance over time, which is important for batch and other single-threaded applications that are still important to most IBM i shops, and you can – I would say must – look at the overall throughput of the systems with their mix of core generations and clock speeds. Here is how the Power7, Power8, and Power9 cores and single nodes with 32 cores each stack up:

The Power7 core running at 4 GHz delivers 6,384 CPWs in the Power 795 system, and the Power7+ core running at 4.4 GHz delivered 7,719 CPWs, which is a 10 percent increase in clock speed but a 21 percent increase in performance thanks to architectural changes between the Power7 and Power7+ processors. The Power8 processor raised the per-core CPW to 10,902 running at 4.19 GHz in the Power E870 (two-node NUMA system) and the clocks were cranked a little faster to make up for the NUMA scale of the Power E880C (a four-node NUMA system) to 4.35 GHz to 11,287 CPWs. That’s nearly twice the oomph of the Power7 chip used in the Power 795. With the Power9 chip running at just a hair under 4 GHz, the per-core CPW rises to 15,903 in the Power E980. That is 41 percent more oomph per core from the top bin Power8 chip to the top bin Power9 chip.

The upshot of this, explains Steve Sibley, vice president and offering manager of Cognitive Systems at IBM, is that a 24-core Power E980 delivers the same or more performance as the fully loaded Power 795 and Power 770 and Power 780 systems that replaced them with the Power7+ chips and a 46-core Power9 system can replace the Power E870C and Power E880C. Here’s a chart that outlines this fact, lining up 64-core machines of all of the generations of Power7, Power7+, Power8, and Power9 systems:

The interesting thing about that 64-core line is that most customers running even big Power Systems iron have 64 cores or less. What we wanted to know is if there are customers who push the big iron to the limits.

“We have customers that certainly leverage our biggest systems with the most nodes, but they are likely running multiple partitions on those systems as opposed to one single 192-core IBM i partition,” says Sibley. “We do some very detailed performance optimization for customers of that size. We do have AIX customers that do that. It’s not a lot, probably a couple handfuls of customers that have a partition of that size. Most of these systems are running with multiple partitions on a single system. But I visited a customer the other day that was running IBM i right alongside SAP HANA, and they are migrating their Business Warehouse initially to HANA and they’ll be looking at S/4 in the future, and they can run those on the same system side by side. Is it the best resource utilization in their environment, which you can’t do on any other platform with SAP HANA.”

The interesting thing about the big NUMA iron in the Power Systems line is that processing scalability has actually been less impressive, in some respects, than IBM might have been able to engineer if it had boosted the top-end systems in the Power7+, Power8, or Power9 machines to 32 or even 64 sockets. Instead, IBM went with making the cores more powerful within each socket and reducing the socket count while boosting main memory and the efficiency of the NUMA connections across the sockets. This, we think, is actually a wise choice. We suspect if IBM needed to hook eight nodes together to build a 32-socket Power E980, it could do it, but it would add latency to the interconnect network (perhaps three hops between any two processors in the complex instead of one or two).

In any event, the performance scale over the past eight years has been pretty linear at the top of the Power Systems range, as you can see:

While the performance increases that come from moving to Power9 systems are important, what really matters to customers is how much the new systems costs compared to maintaining the old one. Sibley gave us two examples of how the math stacks up. The first one pits a 64-core Power 770 based on Power7 processors at a customer that owns the machine and has bought the perpetual licenses for IBM i for it, but is paying hardware and software maintenance on the machine. Three years of hardware and software maintenance runs to over $2 million:

The Power E980 with 24 cores activated and 1 TB of memory, the same memory as the Power 770 it replaces, has, with 381,000 CPWs, 30 percent more throughput and each core has more than 3X the performance per core (great for batch work). The new machine costs $358,000 at list price, and transferring the IBM i licenses from the 64 cores of the Power 770 to the activated 24 cores (out of a total of 32 cores) on the Power E980 costs $141,000, and adding hardware maintenance on the machine for three years going forward costs $31,000 and Software Maintenance (SWMA) costs another $475,000, for just a tad bit over $1 million. So the payoff for the upgrade happens in about 18 months, which is not bad for a machine that could be in the field going forward for four or five years. This comparison does not include reduced energy consumption for power and cooling, and leaves another eight cores of latent capacity in the machines for future performance expansion.

Shops running on Power 795 iron will see a similar payback time, as the following chart illustrates:

The Power 795 machine has 96 cores and all of them are activated for 655,200 CPWs, plus 2 TB of main memory. Again, it is assumed that the customer owns the hardware and has licensed IBM i for the box Power 770 system. Paying for three years of hardware and software maintenance runs to $3.63 million. Moving to a 64-core Power E980 with 43 cores activated yields 679,938 CPWs of performance, and that machine configured with 2 TB of main memory has a list price of $670,000. Transferring IBM i licenses costs $253,000, and with three years of hardware and software maintenance going forward, the total cost comes to $1.83 million. The machine size reduces from a giant rack and down to a two-node server, and there will against be significant savings in power and cooling by moving to the Power E980. The payback, even without including anything other than the hardware, software, and support, comes after about 18 months, the same as with the other example. The Power E980 machine has about the same aggregate CPWs of performance, but there is over 2X the performance per core, which should mean faster performance indeed on single-threaded jobs. The software is again in a much lower software tier, and that cuts the cost dramatically.

Next up, we will take a look at the migration strategies and economics for smaller scale machines that are more typical in the IBM i base.

This Issue Sponsored By

Table of Contents

Content archive

Recent Posts

Subscribe

Pages

Search