Advice For The Power Systems Shop That Has To Buy Now

October 23, 2017 Timothy Prickett Morgan

Having to buy a new computer as one generation is ending and the new one is not quite yet beginning puts users in a tough spot. Without knowing the feeds and speeds of the system, you can’t compare what you might be giving up if you buy now. And without knowing the price of the new machine, and the current price of its predecessor that you might buy, you can’t tell how hard you have to negotiate if you buy the current model instead of the one that is coming down the pike in a few months.

IT budgets tend to run in annual cycles, and for most companies, the calendar quarters are the fiscal quarters, so the current budget runs out in December. And that means there is tremendous pressure on IT shops to spend the budget they have on hand now rather than push it into the next year because any money that doesn’t get spent now is harder to argue for at some future point.

Like many people in the IBM i community, we had been expecting for IBM to launch at least some Power9-based systems before the end of 2018. But back in May, IBM clarified that the (unofficial) plan had changed and there would be no IBM i or AIX machines launched in 2017 and that such commercial Power Systems iron based on Power9 processors would not come out until early next year. That clarity is good, inasmuch as it eliminates some uncertainty, and Big Blue is to be commended for telling customers this. But, if you had budgeted an upgrade for this year and if you are using a machine that is three, four, or five years old that is looking very long in the tooth, then you still have a problem. Do you buy a Power7+ or Power8 machine now, or do you wait until the Power9 machines are shipping early next year and order then?

This buyer’s quandary is made somewhat more complicated by the fact that memory prices have skyrocketed in the past year, and flash prices followed suit because DRAM and flash are made in the same factories. With heavy demand for both in the mobile computing sector (mostly in smartphones where the profit margins are very high, even higher than X86 server memory on a per-bit basis), the memory makers are enjoying boom times, and they are not exactly able – or inclined – to crank up capacity and drop prices. Memory fabs cost billions of dollars, and even if they did want to crank up the volumes, this will take time. The downside is that with DRAM and flash becoming more important in modern systems, high memory prices adversely affect demand for systems, particularly with companies that had a fixed budget for systems. At this point, main memory in a system can be more expensive than processing, and that is new. No one expects for relief in memory and flash pricing until next year, possibly later in the year. Server makers have eaten some of the cost as the spike initially hit, but they can’t hold out forever and are starting to pass the costs on to customers.

The good news, if there is any, is that the low end, scale out Power9 machines will be using standard DDR4 memory sticks instead of IBM’s homegrown sticks that are put onto cards with its “Centaur” buffer chips on them, which double the capacity per socket and double the bandwidth per socket as well as giving an L4 cache layer to the system. IBM’s CDIMM memory is considerably more expensive than standard DDR4 DIMMs. IBM cut prices for DDR4 memory sticks for Power8 scale out machines last October, as we reported at the time, by 10 percent over the DDR3 sticks they replaced with the same capacity. The 16 GB feature cards on entry machines cost $69 per GB, and 32 GB features cost $53 per GB, 64 GB features cost $54 per GB, and 128 GB features cost $62 per GB. If you price up a Dell PowerEdge R540 2U server with two sockets based on Intel’s “Skylake” Xeon SP processors and use 2.67 GHz DDR4 memory, 16 GB sticks cost $22 per GB and 32 GB sticks cost the same $22 per GB. There are not 64 GB or 128 GB sticks available from Dell for these machines.

Now, if IBM charges similar prices for standard memory on the future Power9 machines and only takes the margins that Dell is doing here, then this will be a huge price cut for memory for IBM i and AIX shops; Linux shops using Linux-only Power Systems LC machines are already getting a price cut on CPU and memory capacity. It is my guess that IBM will hold memory prices constant for the standard DIMMs, meaning they will be a lot more expensive than similar X86 server memory, and this could engender third party competition in the memory market for the first time in a long, long time. Or, IBM could split the difference, maybe somewhere around $35 to $45 per GB and that might not be enough for competitors to jump in.

So, knowing this, what do you do? Well, as Francis Bacon correctly observed at the beginning of the use of the scientific method, Knowledge is power.

First, if you can find a reseller or an IBM rep who knows what the plan is for the Power9 rollout, sing a non-disclosure agreement and get them to tell you what they know – now. We know that the “Nimbus” Power9 scale out processors, which we detailed back in August 2016, will offer somewhere between 1.5X to 2.25X the performance of the Power8 chip on a range of workloads on a socket-for-socket basis at a constant clock frequency (which we presume is 4 GHz as it has been with the Power7 to Power8 comparisons). That is a big jump per socket, which is being done mainly by moving from 12 cores with Power8 to 24 cores with Power9. But that also suggests that the cores will be about the same performance. Most IBM i shops only need a few cores, and with the highest clock speeds as possible, which is why we have suggested overclocking the Power9 CPUs for IBM i and even the more radical move of porting IBM i to the System z14 processors used in the latest IBM mainframes.

So we might be in a scenario where processing capacity per core is roughly the same, and pricing for memory could be flat or a little down. Flash prices won’t go down much, and networking cost per bit will go down, but the bandwidth will rise so moving from 10 GB/sec adapters to 25 Gb/sec adapters might be cost neutral; ditto for those companies that need 100 Gb/sec ports instead of 40 Gb/sec ports for real high bandwidth.

But here is another mitigating factor: The systems using the Nimbus Power9 chips will see their memory bandwidth per socket cut bigtime to 120 GB/sec. That’s way down from the 192 GB/sec per socket in a Power S822 or Power S824 machine, but still better than the 78 GB/sec available in the Power7+ systems like the Power 730+ and Power 740+ machines. If you want higher memory bandwidth, you need the Centaur buffer chips, and they will not be available in the scale out machines as far as I know. You will need a Power S950 or higher to get the Centaur buffers, and these will use the “Cumulus” scale up Power9 variants, which have the fatter cores with the full SMT8 multithreading and that only offer 12 cores per socket. (So memory bandwidth per core will be four times higher in the Cumulus systems than in the Nimbus ones. That is something to think about right there.)

In terms of I/O bandwidth, the Power7+ chips delivered 20 GB/sec per socket in the entry machines, while the Power8 chips delivered more than four times that at 96 GB/sec per socket. The jump with the Power8 chips will be similarly large. The 48 lanes of PCI-Express Gen 4 and CAPI 2.0 ports deliver 192 GB/sec of bandwidth per socket, and then the additional 48 lanes of “BlueLink” ports running at 25 Gb/sec, which can be expressed as OpenCAPI or NVLink 2.0 ports – or used as NUMA links in a single motherboard – have an aggregate of 300 GB/sec of bandwidth. That’s 492 GB/sec of aggregate bandwidth per socket, 5.3X times as much I/O bandwidth as the Power8 socket and more than 10X that of the Power7+ socket.

Chew on that for a moment. Given this, I wonder why IBM doesn’t plunk the Centaur chip down on the motherboard just for its L4 cache capability, forget the memory buffering. And maybe, in certain cases, that is precisely what Big Blue will do.

What all of this means is that you have to qualify and quantify the value of having a machine that has a much longer technical lifetime but which might have only a bit more performance per core, the same or higher memory capacity (depending on the pricing, not on the capability of the architecture), almost certainly a lot lower memory bandwidth per core, but also higher I/O bandwidth per core. The cores could have the same or slightly more performance; my guess is the cores plus a slightly higher clock speed compared to Power8 chips could boost performance by about 30 percent per core in the scale out class of machines using the Nimbus chips; because of improvements in the NUMA interconnect, I think the big Power9 boxes will see an additional 30 percent performance boost per socket (and hence per core because these big NUMA machines will be set up for 12 cores per chip, just like the Power8s).

No matter what, the Power8 architecture will be four years old when the Power9 systems come out, and that is something to think about. A Power9 system will, by definition, have four more years of technical life.

The second thing to consider is that you have to be careful who you ask for advice. IBM is ramping Power9 in the HPC arena now, and will be installing the “Summit” and “Sierra” supercomputers for the US Department of Energy over the next one or two quarters. So this will pump up Power Systems considerably, and IBM may even be constrained when it comes to the Power9 processors it can get from fab partner Globalfoundries. So IBM will have a vested interest, for its own financial results over the next year, to try to convince you to wait.

Conversely, there are those who work on commission and they are going to tell you to buy whatever is on the truck. Any reseller of new equipment is going to tell you to buy Power8 now rather than wait for Power9. Any dealer of used equipment is going to tell you to buy Power7 or Power7+ now rather than spend the extra dough on Power8 – unless they have used Power8 gear in the barn – and will tell you that Power9 will be overkill. This may be true, but Power7+ and Power8 were overkill for a lot of workloads, too. The issue is to make the machine do more work and run the legs off it if for maybe five years.

At a certain price, buying a Power7, Power7+, or Power8 machine might make sense. It might not. If these machines cost $X now, they will cost some fraction of $X in a few months when the Power9 systems start coming out. It is hard to guess what that fraction is, but we are going to case the new and used markets – such as they are – to try to sort it out.

For the moment, in the absence of such data, here is some advice. If you can wait, and you can pass your system upgrade budget into 2018 and make sure it is locked down, then that is precisely what you should do. If you are on older iron and older IBM i releases, and you are committed to get current and stay there, this is the best way to do it. The Power9 machines will have incredible I/O bandwidth, with direct links between the processing complex over NVM-Express protocols to flash, and will sport OpenCAPI and NVLink ports to boot. These may not be supported today in the IBM i environment, but they very likely will be and if you wait and buy such a system, you will be positioned for the future and, importantly, you will have a machine that will last a long time. If you have a system with very high memory bandwidth now, and your applications require it, take a very hard look at the Power8 iron and the added cost of CDIMM memory. If you need that memory bandwidth and won’t be shortchanged on the I/O bandwidth by sticking with Power8, then argue for deep discounts now before other people start doing this math and it props up the values of the Power8 iron even after the Power9 machines come out.

It is hard to argue for Power7 or Power7+ unless it is a very cheap system indeed, and maybe only then if you have to hang back on IBM i releases and are coming from a Power5, Power6, or Power6 + system. Even then, I would counsel to get current and stay there so you can enjoy all of the new technologies that IBM is bringing to bear.

This Issue Sponsored By

Table of Contents

Content archive

Recent Posts

Subscribe

Pages

Search