Immutable Copies Are Only As Good As Your Validation
May 23, 2022 Stan Wilkins
A system can always be replaced, but the files and objects that comprise the application and the data that makes it useful can fall victim to all sorts of decay, neglect, or abuse in a modern system. And that is why we did backups to tape subsystems, or even tape libraries and then virtual tape libraries based on disk drives for so many years. And for those who cannot afford to have downtime or lost data, the IBM i base has been fortunate to have some of the best high availability clustering ever invented.
With ransomware and malware attacks on the rise, it is more important than ever to create immutable copies of data – a snapshot of the information in the machine that cannot be tampered with. This capability is built into IBM’s FlashSystem arrays, and is increasingly used by IBM i customers to create snapshots of their data that can be used in the event of an attack or some other kind of data corrupting event.
We have talked to a lot of IBM i customers who are interested in or are making immutable copies using their FlashSystem arrays, and they think it is fine to do an immutable copy of the system every hour and stack them up. And if they get hacked and someone, for instance, tries to encrypt their archived data – a common attack method these days – then the practice is to keep going back into the archive of immutable copies by hand until one of them works, until one of the copies is not infected.
We don’t like that approach. And that is why we have come up with a safe guarded copy methodology that we call CopyAssure. With CopyAssure, we are perfectly happy that customers make lots of immutable copies of their key data. But we believe that as you make these immutable copies, you have to perform the extra set of steps and make sure that this immutable copy is valid and can be put back onto a recovered system in the event of a disaster or an attack.
This means every immutable that gets taken is validated at the time it was taken, automatically made available, added to a partition and the IBM i OS booted up and then the integrity of the database and application can be verified. Now you know you have a point time of recovery that’s been tested and validated and more importantly quickly available if needed.
(An attack is just a kind of unnatural disaster, when you think about it. Not capricious like the weather, but causing just as much damage in the end.)
These days, in fact, you might have a higher chance of the unnatural disaster striking you than a natural one, although with climate change, the incidence of natural disasters is also on the rise. . . .
Best to be covered against both, then. And there is no time like the present to get started.
The IBM i Can Be Hacked, Directly And Indirectly
The first thing that customers have to realize is that despite all of the wonderful security measures in the IBM i platform, hackers can gain access to data even if they can’t plant malware easily as a runtime on the IBM i platform itself with the proper security settings and practices in place.
Take a look at this set of screens:
You may find this hard to believe, but most User IDs and Passwords used in 5250 green screen sessions – and there are a lot of companies still using 5250 terminal sessions in their applications – are unencrypted. And using a common tool such as the Wireshark network protocol analyzer, which is available for free out there on the Internet, you can put the sniffer on the box and see those unencrypted User IDs and Passwords.
It is absolutely possible to encrypt these 5250 sessions, but historically, people don’t do it – and largely because there is this misplaced notion that the IBM i platform can’t be hacked. All platforms can be hacked, and usually the job is done from the inside, not the outside. And that is the real danger.
That is an example of a direct and fairly simple way to get access to the IBM i system from a user desktop. But there are indirect ways, such as gaining access to IBM data and applications through another platform, typically Windows Servers at many IBM i shops share connections with each other. Windows Server is still a popular adjunct platform at companies that run their mission-critical applications on the IBM i platform, either running application servers that are fed by the Db2 for i database on the IBM i platform or running SQL Server databases or other kinds of middleware (such as Active Directory SharePoint) or other applications that work in concert with those IBM i applications and databases. These Windows Servers have access to the IBM i platform, and should someone gain access to Windows Server on the corporate network, then they can potentially get access to IBM i.
As a side note from IBM i, be aware, calling for example your Windows backup server the “xxxxxx Backup Server” isn’t the smartest naming convention, and every hacker knows the family jewels are out there and they can encrypt those and lock the company out of them and send their ransom note. . . . When was the last time you tested a backup/restore? For all you know, someone has already encrypted your files!!!
This really happens.
So having immutable copies of IBM i data that can restore a system to working order is therefore important.
But as we have said, that is only the first step. You have to do more.
Making Sure The Immutable Copy Is Usable
We have taken those next steps. The experts at Chilli IT have created a managed service called CopyAssure, a fully automated process that makes sure that immutable copy is, in fact, not only immutable but usable.
So with CopyAssure, we take the immutable copy and use our custom scripting to map it to a new logical partition and automatically IPL the system to the IBM i command line, run a series of IBM i OS tests and then startup up applications and run integrity scripts that are custom for each IBM i customer which actually validates that the immutable copy made on the FlashSystem is valid and usable. Then, you can exhale, at least for an hour or so. And then do it again every hour on the hour if you feel like it. And if something is out of normal parameters, we will be able to see that it is not right in the data like a set of values in a key file is incorrect, so that even though the data is in fact immutable – meaning it has not been tampered with since it has been copied off the system – it is in fact not valid – meaning it is garbage.
We look for simple things in these integrity checks. For instance, to create a succession of immutable copies, you don’t make whole snapshots of the data, but only the data that changes between snapshots. As a rule of thumb, over a 12-hour period with a lot of transactions, we might see about 20 percent of the data changes in the system. If we see a number that is higher than 20 percent, then we know something peculiar is happening and it throws a red flag.
Security information and event management, or SIEM, is a field within the field of computer security where software products and services combine security information management and security event management. They provide real-time analysis of security alerts generated by applications and network hardware – products like QRadar are a key component of this ransomware strategy but confirming the immutable is usable is what surely matters.
The other important thing to our approach of validating immutable copies is that you don’t have to keep stacking up the immutable copies – burning up huge amounts of storage – just in case a prolonged malware has been running or hack has been going on inside your system for months, or even longer. Because we validate the immutable copies and know they are valid, we can start overwriting old immutable copies, much as video surveillance equipment has done in retail locations for decades starting with VCR tapes and then disk and then flash storage.
Moreover, the immutable copies we make are local to your IBM i machine and its FlashSystem storage. If you are doing backup to the cloud, and you have to restore from a remote copy of your applications and data, it could take a day or more to get that data off the cloud and back into a clean system on your site. You can still do high availability and disaster recovery replication to the cloud, which is a good idea of course. But that covers different kinds of disasters. The idea with CopyAssure is to get your IBM i system back online in the shortest time possible.
No one has the time to do such validation by hand, which is why we have automated the process. And that, in fact, is what automation is for.
If you want to find out more about CopyAssure, watch this video explanation, and then feel free to reach out to us for more information. We want to help protect your . . . assets.
One more thing: We will be presenting CopyAssure and our approach to immutable copies at the International i-Power 2022 conference, which is being held on June 7 and 8 in Northampton in the United Kingdom and which you can find out more about at this link.
Stan Wilkins is co-founder and technical director at Chilli IT.
This content was sponsored by Chilli IT.
