Update On The Spectre And Meltdown Patches For Power
January 15, 2018 Timothy Prickett Morgan
When it comes to the Spectre and Meltdown speculative execution security vulnerabilities that hit as the new year was getting going, the important word to ponder is “mitigated.” Everyone is talking about mitigating the issue, but no one is using the word “fixed.” As we discussed last week, one of the two types of Spectre vulnerabilities – the Variant 2 known as branch target injection – is particularly tricky to hack and to fix, so IT vendors are choosing their words very carefully.
The odds were that unintended consequences for such a low-level fix will occur, so you can understand the caution, particularly for enterprise-grade platforms like IBM i and AIX, which have customers who categorically cannot take downtime in their systems and who are understandably hesitant to apply a patch that has not been thoroughly tested by IBM and vetted by some customers.
Microsoft, for instance, stopped distributing patches for older AMD X86 processors (meaning ones not introduced last year) after some customers had their machines lock up after applying the security updates. Canonical’s Ubuntu Linux also had some issues, according to users on its forum. And, disturbingly, Intel’s own microcode patches for the “Haswell” and “Broadwell” generations of Core desktop and Xeon server processors also had issues, causing an abnormally high number of system reboots after the microcode was applied – something that Intel shared with system makers and hyperscalers, but not with the community at large even after it promised with its new Security-First Pledge to specifically not withhold such information. (Intel and other chip makers have known about the Spectre and Meltdown speculative execution vulnerabilities since last June, and the IT community had to weigh trying to fix the problem with someone possibly finding the exploit on their own and wreaking havoc. So far, there is no malware in the wild that exploits these speculative execution holes to gain access to information on machines that should not be visible, such as decrypted passwords.) The only way the world knows about this rebooting issue with Intel systems after the Spectre and Meltdown patches are applied is because the Wall Street Journal found out about it from some hyperscalers and wrote about it. In any event, Intel seems to be passing the responsibility buck downstream to PC and server makers, saying they should get patches from their vendors. For all we know, the vendors asked Intel to butt out because they have tweaks to their firmware that Intel should not be messing with.
At least with Power Systems and IBM i there is one vendor, but that may be more of an illusion with the IBM i team in Rochester being very distinct from the Power Systems team in Austin.
As promised, the initial patches that help mitigate against Spectre and Meltdown for the Power Systems firmware became available through IBM’s Fix Central patching site last week. If you drill down through the Fix Central system, you will find the firmware patches for Power8-based systems at this link, right at the top of the page identified as SV860_138_056 / FW860.42, issued on January 9. (There are a bunch of other HIPER PTFs there, too, that have nothing to do with Spectre or Meltdown.) The similar firmware updates for machines based on the Power7+ processors is at this link and is identified as AL770_120_032 / FW770.91, also issued on January 9.
As far as we can tell, these patches use a different means of being applied to systems that is distinct from the normal PTF patching methods used on IBM i platforms. With this PTF system, IBM denotes firmware updates with MH and a following set of numbers, licensed internal code (meaning the operating system kernel and adjacent layers in the IBM i operating system) with MF plus some numbers, and then other patches have a designation of SI or SF plus some numbers. So, here’s the interesting bit. As of Friday morning, when we are going to press with The Four Hundred, our PTF Guide Doug Bidwell could find no MH series PTF patches for IBM i that correspond to the updates shown above through Fix Central for Power7+ and Power8 machines. There should be, we reckon.
Now, here’s the funny bit. The MF patches for the licensed internal code – implying the kernel patches that are expected to be delivered on February 12 – for IBM i are available for order, and Bidwell ordered them for all three releases – IBM i 7.1, 7.2, and 7.3 – just to make sure they were real. The cover letter describing the patches for IBM i 7.3, which are under number MF64551, LIC Mitigate Spectre and Meltdown Vulnerability, is at this link, and it is clearly a kernel update. It does not say which speculative execution exploits its stops, but it does say this: “This PTF can be loaded and applied independently from Power FW fixes that have been created to address the Spectre and Meltdown vulnerabilities, but both IBM i PTFs and available Power FW fixes are required to mitigate vulnerabilities.”
The FW in there is firmware, and it implies the patches we outlined above for Power7+ and Power8 iron that are available through Fix Central. It is not clear when IBM will make the firmware patches available as MH patches through the normal IBM i PTF channels, but now would be a good time.
After we went to press on Friday, Bidwell received a security bulletin from IBM concerning the Spectre and Meltdown vulnerabilities. In that, there is a link for each release For IBM i, here are the patch groups and links to each:
Release 7.1 – MF64553
Release 7.2 – MF64552
Release 7.3 – MF64551
There is also yet another separate link to the Power Systems firmware patches, which you can find here. This document is useful in that it actually provides links to the firmware updates and shows which ones covered by processor type (Power7+ or Power8, which also includes the few Power8+ machines that IBM created), by IBM model number, and by system name.
Every announcement letter from Big Blue should always provide such detailed descriptions for every product.
The one thing that we have not seen from IBM, and we still want to see, is clarification as to whether or not any Power Systems are affected by the Meltdown (Variant 3), or rogue data cache load, vulnerability, and if the kernels in IBM i, AIX, and Linux are each and individually required to have the Kernel Page Table Isolation (KPTI) patches or other techniques to plug this hole. To review, here are the details on the three security holes with links to their Common Vulnerabilities and Exposures (CVE) description:
- Variant 1, CVE-2017-5753: Bounds check bypass. This vulnerability affects specific sequences within compiled applications, which must be addressed on a per-binary basis.
- Variant 2, CVE-2017-5715: Branch target injection. This variant may either be fixed by a CPU microcode update from the CPU vendor, or by applying a software mitigation technique called Retpoline to binaries where concern about information leakage is present. This mitigation may be applied to the operating system kernel, system programs and libraries, and individual software programs, as needed.
- Variant 3, CVE-2017-5754: Rogue data cache load. This may require patching the system’s operating system. For Linux there is a patchset called KPTI (Kernel Page Table Isolation) that helps mitigate Variant 3. Other operating systems may implement similar protections – check with your vendor for specifics.
Variant 1 and Variant 2 are collectively called Spectre, and Variant 3 is known as Meltdown. IBM has also not said what the level of mitigation is accomplished for either Spectre or Meltdown if only the firmware patches are applied, and what the additional mitigation level is with the kernel patches. It would be nice to know the extent of the vulnerabilities for all three variants of the Spectre and Meltdown holes, and precisely what IBM is doing to fix them.
Moreover, if for some reason Power is less susceptible in any way to the Spectre or Meltdown exploits, IBM should triple check that and then do it one more time and then start bragging a bit. But we would wait until the patches are all out and known to work before starting to brag. Moreover, look for Bidwell’s advice for the Spectre and Meltdown patches in the IBM i PTF Guide within Wednesday’s edition of The Four Hundred. For now, Bidwell says sit tight and let the dust settle for a few days, and let some other people go first and see that these patches all work. With no known exploits, it is riskier applying the patches and taking downtime to apply them and possibly more if they don’t work right than being hacked right now.
We will be keeping an eye out for the performance implications of the patches, too. Stay tuned.
