Getting A Firm Handle On Power Systems And Storage Firmware
June 15, 2022 Richard Warren
Back in the old days of the AS/400 and the iSeries, most customers had a single box or maybe two, one for production applications and databases and one for development of high availability. And everything that box needed was inside of itself.
And at most, you applied two kinds of PTFs – those for the operating system and those for the microcode – to the machine, and you did that maybe once or twice a year and every once in a while you might add some group PTFs to update security or other important features.
But the world has changed since that time, and in a lot of ways that make keeping systems up to date a pain in the neck. Starting in the late 1990s, machines got logical partitions. And with the convergence of the pSeries and iSeries machines, we not only got the PowerVM hypervisor but also the stripped-down AIX that is known as the Virtual I/O Server. Then a lot of systems have Hardware Management Consoles, or HMCs, which have their own baby operating system and their own set of firmware that need to be kept up to date. And there are also now Fibre Channel switches and SAN storage arrays that have their own firmware and software. And further complicating things – thankfully taken care of in the PTF process – is the patching of any open source development and systems software that is now more common on Power Systems machinery and integrated with the IBM i platform.
It is a lot to keep track of and keep current, but the thing is this: Most IBM i shops are not doing it. They might have a good handle on keeping the IBM i operating system, database, and open source software they are running on the machine relatively current. But when it comes to the firmware and software on the Power Systems machine and auxiliary hardware such as Fibre Channel switches or Storwize or DS series SAN arrays, when this hardware is installed, that’s it. Ditto for the service processor inside of the Power Systems machines, which is not integrated with the PTF patching process at all. Oh, and by the way, even the storage drives inside of the SAN arrays have their own microcode and firmware patches. . . .
All of this firmware – let’s just lump it all into one word – is a serious challenge in a number of ways. First, this is very low level software and if you don’t know what you are doing, you can “brick” your hardware. Meaning, it no longer does anything but act like a brick. If you don’t keep firmware current, you are open to all sorts of interoperability issues because things have to be patched on both sides of a wire to work properly and securely together. And then, there is the security risk imposed by not keeping firmware patched, because vulnerabilities are found in all software, even firmware, and it is only a matter of time before someone, somewhere writes a hack to exploit it.
Firmware is tricky, and we say this because we offer a service to keep it updated for IBM i shops running a variety of hardware components – Power Systems servers, Fibre Channel switches, SAN storage, HMCs, and so on. You don’t have to do firmware patching all that often, and it is a very specialized skillset, so it is very likely that your IT organization does not do this. One of the tricky bits is that if you get too far behind, you can’t just jump to the latest release, you have to do double jumps instead of one, and you need to know when, where, and how to do that on a system that is running concurrently without crashing the system. You don’t want to do a double jump attempt the first time you patch firmware – any more than you would try to do a double jump the first time you were in the rink with ice skates on. . . .
A managed service is a natural thing for people to share when it has very specialized skillsets and that does not provide competitive advantage above and beyond keeping a machine safe and secure and stable. That’s why Chilli IT has formalized a firmware management service, which we have been offering on an ad hoc basis for the past five years, so that all IBM i customers, regardless of where they are in the world, can keep their firmware patched.
The firmware management service has a fixed price to deliver firmware patching on the components in the system once a year, with critical patches coming as needed for security or stability reasons. Like, for instance, looking for Log4j exposures in the hardware and firmware when that security vulnerability was revealed and staying on top of that each and every day for our customers.
The price of the firmware management service varies depending on the nature of the setup, of course, but we do an estimate after reviewing your inventory of gear and then set a fixed price so you can budget. If it takes us more time to keep things current than we expected, we eat that time because you get a fixed cost from us. The price of the service ranges from a low of around $5,000 for a single machine to as $10,000 per year for customers with multiple machines, often with high availability software running, which we know plenty about as well. (And, if you want, we can manage your HA cluster for you too, now that I think about it. . . . )
A lot of this firmware can be patched while the system is running, which is called hot patching or hot fixing, which is a great thing. It wasn’t always this way, but be forewarned: If your firmware falls too far behind, then there is a chance that it will require a reboot to be applied after it is patched.
One last thing. We are experts at this, and if a patch is bad or something goes bad, we know how to roll it back to the previous state. We provide configuration backups as part of the service, which means you don’t have to rebuild SANs or VIOS from what you think you remembered about your settings if something goes very badly. This is a rare occurrence in the Power Systems world, but it does happen and for those who are not experts at patching, it could cause a coronary event. You want to avoid that, and put the strain on our hearts, not yours.
Richard Warren is co-founder and managing director of Chilli IT.
This content was sponsored by Chilli IT.
RELATED STORIES
Immutable Copies Are Only As Good As Your Validation
