IBM Tweaks Power10 Hardware: Fatter Memory, Other Stuff

August 19, 2024 Timothy Prickett Morgan

We haven’t seen a lot of hardware announcements this year from Big Blue for the Power Systems platform, and you really should not expect much in the way of new hardware when Power11 systems are expected to be launched in 2025 and the Power10 machines have been in the field since late 2021 for the Power E1080 and since July 2022 for the rest of the line.

To be fair, we did have the entry “Bonnell” Power S1012 machine come out in May this year. So it has not been a total drought. But hardware launches are not on an annual cadence like they were in the days of the AS/400 three decades ago.

On July 9, IBM switched the main memory used in the Power10 machines from DDR4 technology to DDR5 technology, which is now more commonplace, less expensive, and more widely available when the Power10 machines first came out. IBM has created its own OpenCAPI Memory Interface, which is based on its PowerAXON SerDes I/O that is used to implement other kinds of I/O in the Power10 chip. With OMI memory, certain SerDes lanes are designated as memory, and the memory DIMM itself has a controller that translates OMI calls to DDR4 or DDR5 protocols. So IBM can change memory chip technologies on its memory cards at will, unlike every other server on the planet, which has controllers on the CPU complex that are explicitly for memory and that explicitly speak DDR4 or DDR5 protocols.

The Power10 chip was designed from the get-go to support DDR4 and DDR5 memory, and if we had to guess, we would say that the DDR5 memory that IBM is selling now for Power10 machines will be compatible with Power11 machines that come out next year and possibly into 2026, depending on how IBM stages the Power11 rollout. These DIMMs are technically called differential DIMMs, or DDIMMs, and we talked about them here when the “Denali” Power E1080 was launched in November 2021.

The new DDR5 memory cards for the Power10 line, outlined in announcement letter AD24-0474 dated July 9, became available on August 13. IBM used a mix of 2.93 GHz and 3.2 GHz DDR4 memory in the original memory cards for the Power10 line. The 3.2 GHz memory yielded a maximum of 409 GB/sec of memory bandwidth per socket (which was only available on 32 GB and 64 GB memory cards), while the fatter cards (with 128 GB and 256 GB capacities) ran at 2.93 GHz and delivered 375 GB/sec of bandwidth per Power10 socket. As far as we can tell from the announcement letter, the new DDR5 DDIMMs are twice as dense as their DDR4 predecessors, and come in capacities of 64 GB, 128 GB, 256 GB, and 512 GB for all of the machines, and the Power E1080 has an even fatter 1 TB memory card. As far as we can tell, all of these DDR5 memory cards run at 3.2 GHz, so for entry machines using fatter memory cards, there will be a memory bandwidth boost to the full 409 GB/sec bandwidth per socket.

DDR5 memory in servers runs anywhere from 4.8 GHz to 8 GHz these days, but this memory is not cheap and is not necessarily the best choice for the very wide OMI memory interfaces of the Power10 chips. That said, we expected to see a bump up to 4.8 GHz at least with the DDR5 upgrade. Almost certainly faster DDR5 memory, with a minimum of 4.8 GHz clocks, will be supported on Power11 chips. The “Sierra Forest” Xeon 6 processors from Intel, for instance, have DDR5 memory running at 5.6 GHz and 6.4 GHz when they have one DIMM per channel on their DDR5 controllers. (If you do two DIMMs per channel, they have to run a little slower.)

IBM was not precise about the performance or price differences between its DDR4 and DDR5 DDIMMs, but said that the new DDR5 memory features “provide improved bandwidth, energy efficiency and load latency compared to equivalent DDR4 memory with the same high-performance, enterprise-class memory currently offered with DDR4 technology.” We will try to get more specific data.

IBM says further that the older DDR4 DDIMMs used on Power10 machines will not be available for orders after November 12 for new machine orders, but will still be available for an indeterminate time for memory upgrade orders. When IBM clears its inventory of DDR4 DDIMMs, the axe will fall on these, too.

One last thing: You cannot mix DDR4 OMI memory and DDR5 OMI memory in the same machine, even if they run at the same speed.

In the same announcement letter, IBM said that it was delivering new “enterprise” class NVM-Express flash drives, with capacities of 1.6 TB, 3.2 TB, and 6.4 TB, for its Power10 machines, as well as a new “mainstream” NVM-Express flash drive at 15.3 TB of capacity. Big Blue also added a new PCI-Express 1.0 four-port 1 Gb/sec Ethernet adapter for the Power10 line – how retro – as well as a 25 Gb/sec RoCE Ethernet adapter that works across the Power10 line and that plugs into PCI-Express 4.0 slots. Additionally, IBM says that it can run the KVM hypervisor inside a PowerVM hypervisor now, which is neat, and that PowerVM now supports Live Partition Mobility on the Bonnell Power S1012 server.

On August 13, IBM made a few other small hardware announcements in the Power10 line. In announcement letter AD24-0605, IBM says that the Power E1050 and Power E1080 servers can now attached a fourth PCI-Express 4.0 Expansion Drawer for storage and I/O to an x8 slot in each node in these servers. Before now, you were tapped out at three I/O expansion drawers per node.

RELATED STORIES

Power10 Boosts NVM-Express Flash Performance

Direct Attached Storage Gets Massive NVM-Express Expansion

IBM Tweaks Some Power Systems Prices Down, Others Up

Some Power Systems Hardware Tweaks

Power9 Servers Get Updated Flash Drives

How Much Does NVM-Express Flash Really Boost IBM i Performance?

Tweaks To Power System Iron Complement TR Updates

IBM Revamps Entry Power Servers With Expanded I/O, Utility Pricing

IBM Doubles Up Memory And I/O On Power Iron To Bend The Downturn

The Skinny On NVM-Express Flash And IBM i

Power Systems Refreshes Flash Drives, Promises NVM-Express For IBM i