The Herculean Task Of Applying Spectre/Meltdown Patches

October 1, 2018 Timothy Prickett Morgan

The Spectre and Meltdown speculative execution vulnerabilities are, as our resident chief technology officer and author of the weekly IBM i PTF Guide, Doug Bidwell, is fond of saying, the gift that just keeps on giving.

We had the shock of finding out in January that there were vulnerabilities in all processor architectures that use speculative execution in their instruction chewing engines – that means all existing processors, by the way. There are none that do not use this very useful architectural feature. And then we had the wait to see what the industry would do to patch these vulnerabilities, and then the longer wait to try to assess what impact the patches to Spectre and Meltdown would have on performance. The answer is that IBM is only patching Power8 and Power8+ (those for supercomputers that have NVLink included) processors as well as the new Power9 chips; the performance impact on based on the hit that machines took on the generic Commercial Performance Workload is around 5.2 percent of the aggregate throughput. Which ain’t too bad, all things considered. But there are other impacts, and a big one is the time it takes to get an IBM i system all patched up. And the second issue is convincing companies to take down their systems to actually apply the patches in the first place.

“There is no real urgency about Spectre and Meltdown because no one knows of any specific exploits,” explains Bidwell. “But the thing is, in many of the key industries where the IBM i platform plays, these companies do not want to be the story that everyone all hears about because they got hacked because they didn’t apply the patches, or worse yet, didn’t do them right. The other thing is that the longer you wait to apply the patches, the harder this us getting, although I can’t imagine it getting much harder than it is.”

The table below, pulled from the IBM i PTF Guide tells you the patches you need to pull and apply, but it does not express what this means. So we are here to enlighten – or in this case, enheavy – you.

That’s a lot of work to do, just gathering up all the patches. But it is even more work than it looks like. According to IBM’s best practices you have to do a GO SAVE: Option 21 full backup of the system before you even get started. The system firmware has to be updated, either before the patching begins or after, and the order of operations apparently does not matter. As a matter of course, we think you should work down from the lowest levels of the hardware with patching and then go up the stack, so we would do the firmware patches first. Then you need to do the Cumulative PTFs and Group PTFs for the specific release of the operating system that has been patched, and thus far these patches are only available on IBM i 7.1, 7.2, and 7.3. And when you are all done, you have to do another – and different – system backup.

That means if you are not willing to take the Spectre and Meltdown risk and you are on older iron – say Power6, Power6+, Power7, or Power7+ machines – then you have to upgrade to at least IBM i 7.1. And once you go through all of that grief, maybe it is time to upgrade to a Power8 on the cheap or a Power9 and have a machine that has another five, six, or seven years of technical life and maybe more. While you are at it, you might as well bite the bullet and upgrade from Java 6, which is dead, to either Java 7 or Java 8, and it might as well be Java 8.

It could be worse. On the Intel Xeon server platform, the chip is vulnerable to a related set of speculative execution vulnerabilities called L1TF, also known as Foreshadow, that have similar security exposures, but the bad new is that on Xeon chips, the only way to fully secure the chip with a root of trust with server virtualization is to turn off simultaneous multithreading, which is called HyperThreading by Intel and which presents two virtual instruction pipelines, or threads, to the operating system for each core. HyperThreading allows for more VMs to be pinned to a given processor (twice as many, if you do one VM per virtual thread) and also boosts overall instruction throughput by somewhere between 20 percent and 30 percent, depending on the operating system and workloads. The AMD Epyc chips are not susceptible to the L1TF/Foreshadow vulnerabilities because of the way that security and threading and virtualization are implemented. As you might imagine, AMD is trying to make a lot of hay out of the fact that companies have to choose between security or threads. It is important to not do pride before the fall here. The IBM Power architecture, the AMD Epyc architecture, or any number of Arm architectures could fall prey to a different kind of speculative execution vulnerability.

Believe it or not, Bidwell’s company, DLB Associates, has over 300 IBM i customers, many of them with multiple machines, and he is still finding customers who have no idea about the Spectre and Meltdown threat. So far, Bidwell has patched about 100 of the customers, and expects that it will take between now and the end of Christmas break to get all 300 up to snuff. Bidwell does not want any of his customers to be vulnerable should someone create a hack that goes wild on the Internet that exploits Spectre and Meltdown.

The full process of applying the Spectre and Meltdown patches to a given machine, including the before and after backups, is taking approximately 20 hours, and that is a lot of downtime for any company to take all at once. So Bidwell advises to take it in smaller bites, doing each step methodically and carefully over a couple of weekends. This might increase the number of backups, depending how much risk you want to absorb, of course. Those with HA clusters could, we think, update one machine and then do a rollover, but that presents all kinds of risks, too. Three-way replication is the way, but it is hard to keep everything in synch and way beyond the budget of most IBM i shops.

No matter what, these patches have to be applied and the performance hit has to be taken, just in case someone does write an exploit for Spectre and Meltdown. There is no getting out of it, unless you want to play ostrich until you get hacked and then fired.

This Issue Sponsored By

Table of Contents

Content archive

Recent Posts

Subscribe

Pages

Search