One IBM i Shop’s Close Call With Ransomware

July 28, 2021 Alex Woodie

Think the ransomware epidemic won’t affect you, that it’s somebody else’s problem? After reading this story about one IBM i shop’s recent experience with cybercriminals, it may have you thinking twice about your approach to security.

Greg is the IT manager at a midsize distribution company located in the South. IT Jungle is abiding by his request to keep his last name and the name of his company out of this article. But Greg was determined to share his story with the wider IBM i community, in the hopes that it will spur them to take the ransomware threat seriously, and to improve their approach to security.

The ransomware attack started on May 15, 2021, a Saturday. When employees arrived to work on Monday, May 17, they discovered they were locked out of the Windows-based PCs and servers. Greg started poking around the network, and noticed some files with odd extensions. He also stumbled across a “readme” file in a strange location.

“That’s where we found the ransom note,” Greg told IT Jungle. However, nobody at the company actually looked at the ransom note, nor read its demands. “We didn’t download the browser they said to download. We didn’t look at the ransom. We weren’t paying it, period.”

Like many midsize IBM i shops, the distributor relies on a mix of different systems, most of which ran in-house before the attack. In April, the company had just taken delivery of a new IBM Power9 server, which runs its IBM i-based ERP applications. It also ran its own Microsoft Exchange server, an Active Directory server, two additional Windows servers for shipping software, and a separate Windows server for serving AS2 transactions.

The company subscribed to several backup and security services, including endpoint protection from Webroot, firewalls from WatchGuard Technologies, online backup for the IBM i server from Carbonite, and backup for the Windows systems using an overseas company.

All of the company’s servers and PCs were compromised in the attack, except for the IBM i server and the AS2 server, which was offline due to a malfunctioning fan.

According to a forensic investigator hired by the company, the attackers had exploited a security flaw in Exchange Server to send a malicious email attachment from a legitimate email account, Greg said. When an employee opened the malicious attachment, which looked like legitimate company business, it enabled the cybercriminals to enter the company’s network. The ransomware strain that was used is called Conti, Greg said.

This was not an automated ransomware attack. Once the inside the network, the attackers created an Exchange administrator account on the Exchange server, according to Greg. “Then they hopped over to Active Directory,” he said. “They got on that server. They deleted the server backup files we had on the file server.” Someone, or something, put the ransomware onto a group policy in Active Directory, which allowed the encryption routine to roll out to all endpoints.

An image of a different ransom note left by cybercriminals using the Conti ransomware in a different attack. (Image courtesy Sophos)

The Webroot endpoint security software blocked some of the malware from spreading, but not all of it. It turned out, there had been an update, and as part of the update, the distributor was not informed of the new “portal” required to activate the software. With the help of the forensic investigator and Webroot software, which Greg said was “very, very good,” the company began its recovery that Tuesday, just one day after discovering the attack.

The first step in the recovery was tracking down the offending DLLs and EXEs, and blocking all the IP address that the cybercriminals were using for communication. The company initiated a recovery, which took time because the DR company it uses for Windows was overseas, and all communication took place via email. “You get what you pay for,” Greg said. (The company is in the process of consolidating its online backup for Windows and IBM i with Carbonite.)

Through all of this, the IBM i server was untouched, even though it was on the network and the cybercriminals had full access to it. “I guess they just didn’t see it, or they saw it and just didn’t know what it was,” Greg said.

The company was very lucky, because the company had relatively poor security on the IBM i server. “For the most part our object authorities are wide open,” Greg said. “Public has *USE or *ALL, because the legacy application we’re running does not use adopted authority or anything like that. We were at password level 0.”

Since the core of the company’s business runs on IBM i, it was able to get back up and running and shipping product by Thursday. “I was told by numerous people who are familiar with these types of attacks, who said we recovered far more quickly than most,” Greg said. “We were lucky.”

However, the company’s luck was soon to be tested. On Thursday night, as Greg and another employee in the IT department stayed late to monitor the systems, they suffered another attack. Greg had just left to get dinner, which he and the other employee ate at the site before returning to work. When they left their consoles to eat, the cybercriminals returned, and once again encrypted the systems.

“When that happened, we went to the computers and were pulling cables out of servers, network cables, Internet connections. We even dropped the Wi-Fi in the building,” Greg said. “Forensically we found out after the fact, the term they used is ‘bricked our server.'”

They packaged up the server and sent it out for analysis, and they could not do anything to recover it, Greg said. “They couldn’t see anything after that second attack,” he said. “So at that point, we air-gapped the IBM i and basically wiped every server, wiped every workstation, and reinstalled.”

While the company was able to resume shipping product thanks to the “security through obscurity” provided by the IBM i server, it would be another six days before they got the Windows systems up and running. But IT would never look the same.

“We completely re-evaluated, completely revamped all of our security,” Greg said.

The company adopted a new security device called the Coretex XDR from Palo Alto Networks, which adds behavioral analytics to the security mix. It outsourced management of the firewalls to a dedicated company. It turned off Exchange Server and moved to Office 365. It reduced all users’ Windows credentials to the bare minimum. It’s also in the process of implementing multi-factor authentication.

“I was under the assumption that our Windows environment was relatively secure,” he said, “and I was wrong.”

But the overhaul doesn’t stop at Windows. Greg is also re-evaluating the security of the IBM i server. He took advantage of a free security assessments, and the results were not horrible, but there are a few things to clean up. For starters, Greg moved from password level 0 to password level 3, which is required to use complex 12-digit passwords. The company adopted TLS encryption for ACS and RDi sessions, even for developer and admins running the software in-house.

The company is also in the process of adopting exit-point software from one of the well-known IBM i security vendors. While the menu-based navigation system of its legacy ERP system minimizes the chance for harm, Greg said, there is still the possibility that criminals or ne’er-do-wells could wreak some havoc. The ability to access or delete just about anything on the system via Run SQL Scripts pushed Greg to tighten up security. With an exit point for SQL and ODBC in place, that should eliminate that threat, he said.

In April, as part of the Power9 upgrade, the company also started using a new LTO-7 tape drive, which the company did not use before. With weekly backups hitting the tape drive on Sunday nights, Greg can rest easy knowing that at least his company’s data is safe.

“I’ve been assured by our IBM business partner where we backup our data that there’s no way they could delete any of our backups, because that’s not an ability within the software,” Greg said. “In other words, you couldn’t get on the IBM i and delete a backup set, or anything like that. You would actually have to be in the data center.”

All told, this has been a big learning experience for Greg and his team. Ironically, Greg was already taking the initiative to tighten up the IBM i security before the events, but it turns out that he needed to make investments across the board.

After recovering from not one ransomware attacks, but two, within the course of a week, the experience has made a lasting impression. He reported the attack to the FBI, which took the report. “The field office that handles this is being overrun, so we were talking to a different field office,” Greg said. “It’s bigger than what the news lets on.”

While the company recovered its systems with only a small amount of data loss, it has made a lasting impact. “From a personal standpoint, I lost a lot of sleep,” Greg said. “I had nightmares over it. It feels like someone broke into your house, tore everything up, and you’re not sure if they’re still there.”

Greg credits the company’s owner with being very proactive about security. He talked with all the employees about identifying ransomware, and scheduled time for the IT department to train on important tasks, such as re-imaging Windows machines, re-installing ACS, and adding people to the domain.

With the backing of the owner, Greg has vowed to share his story in the hopes that it will spur others to take action.

“People need to know understand the seriousness here,” he said. “The advice I would give is take it seriously and take action now, not after it occurs. I would strongly recommend looking at your overall security policy, having a conversation with your end-point protection, reviewing your firewall, reviewing your admin accounts, implementing multi-factor authentication.”

“If we can help anybody because of what we went through to avoid this – that’s what we want to do.”

RELATED STORIES

IBM Thwarts Ransomware in FlashSystem with New Data Safeguard

Ransomware Attacks Hit Closer To Home

Ransomware Epidemic Hits Epic Proportions, And IBM i Shops Take Notice

Christmas for Ransomware: How COVID-19 Is Fueling Cybercrime

Assessing The Ransomware Threat On IBM i