IT Horror Stories

October 30, 2017 Alex Woodie

Let’s face it: IT can be downright scary. From unpatched servers and failed ERP implementations to clueless CIOs and that clown in the server room, information technology is a business fraught with danger. Here are a few IT tales that could keep you up at night.

Are hackers lurking in your system? It’s more likely if you haven’t kept up on your patches. That’s the horrific lesson learned by Equifax, which recently disclosed that it lost sensitive data for 143 million consumers after bad guys worked their way into the system through a vulnerability in some open source middleware that was left unpatched for months.

Equifax’s terrible, horrible summer of 2017 is having major repercussions, especially in light of the coming activation of the GDPR law next May, which will bring stiff penalties for any firms mishandling data of European citizens. And the breach was so bad that it even cost Equifax CEO Richard Smith his job – although that $90 million parachute will soften his landing just a bit.

We don’t know what type of server was at the center of the Equifax breach, but it’s clear it could happen to any type of server – even the fortress-like IBM i, which has been outfitted with all manners of open source software. “These servers are breached on a regular basis,” security expert Patrick Townsend says of the IBM i.

The peculiarities of the IBM i platform – including text-based interfaces, function key-driven navigation, and cryptic file names – can sometimes trip up unsuspecting passers-by. For the next tale of IT gone bad, we look to the recent past, when a change management software vendor called Aldon (now owned by Rocket Software) held a contest to see who suffered the most.

What’s lurking in your server room?

The gold medal was handed to a representative of a bedding manufacturer that employed a programmer who probably wished he’d stayed in bed that day. Apparently, the coder was working on a project that that involved a shipping and re-order application.

“That developer created a logical file but did so right on top of the production physical,” we told you back in 2008. “This created a bug that turned product order quantities from just two or three towel bundles into two or three million!” Because of the lack of a quality check, the systems nearly melted down trying to calculate and distribute the huge orders. Lots of overtime (and hilarity, we’re sure) ensued before business as usual was restored.

Outsourcing is often viewed as a win-win. It’s great for the people who get new jobs, and it’s great for the company that’s going to save a big bag o’ cash and pass huge savings on to you, the dear consumer. But sometimes, just occasionally, big outsourcing projects go terribly, horribly wrong.

IBM was involved in one such project back in 2007, when the Australian territory of Queensland awarded a contract to Big Blue to take over the payroll system for its health department. IBM won with a low initial bid of about AUS$6 million.

However, IBM’s bill for the work quickly quadrupled to AUS$25.7 million, according to the official 2013 Queensland’s government report on the matter. “Just as the price increased sharply, so too did the time within which the Project was to be delivered,” the report stated.

The replacement of the legacy Lattice payroll system eventually went live in 2010 – two-and-a-half years late and way over budget. All told, the project was to cost the government over a billion Australian dollars, but it never fully worked. Some staffers didn’t receive paychecks while others were overpaid or underpaid.

In its scathing report, Queensland admitted that it could have been clearer in defining the scope of the project. But the group didn’t buy IBM’s excuse that it was simply following directions. “IBM. . . . had an obligation to be more than merely a passive recipient of the customer’s instructions,” the report concluded. The episode disturbed the Aussies so much that, to this day, the Queensland state government has barred IBM from doing business with its entities.

The United States Air Force has its own billion-dollar horror story too. Starting in 2005, the organization took its budget past the 10-digit mark to consolidate hundreds of disparate legacy apps and centralize them under a brand-spanking-new ERP system from Oracle dubbed the Expeditionary Combat Support System (ECSS).

The network guy looks a bit off today.

The ECSS project was supposed to be ready to meet the military’s 2017 Financial Improvement and Audit Readiness (FIAR) guideline. However, after seven years of development, Computer Science Corp, the system integrator on the project (which recently merged with Hewlett Packard Enterprise Services to form DXC Technology) didn’t have much to show.

In 2012, the USAF said it would take another $1.1 billion to complete just a quarter of the original scope of the initial project – and that wouldn’t be ready until 2020, according to a Computerworld story on the failure.

It’s no surprise that the biggest and most glaring horror stories originate from our public institutions, since they are spending taxpayer dollars and answer to the people. But you can bet that large private firms – and a fair number of large public companies, too – have experienced their share of horrific IT failures.

If you’ve worked in the IBM i midrange business for very long, you’ve undoubtedly heard about failed attempts to get off the platform. There was the heavy equipment manufacturer that tried and tried to migrate off the AS/400-iSeries-System i, only to move back to its original system after spending truckloads of cash.

And then there was the big software maker that tried for years and years to get off the AS/400-iSeries-System i platform, only to give up and outsource the whole shebang so they could claim to be “clean” – or at least that’s what we’re told through unofficial back channels. These companies never fess up to their mistakes, because they don’t have to.

So what IT horror stories keep you up at night? Share them with the IT Jungle team – if you dare!

This Issue Sponsored By

Table of Contents

Content archive

Recent Posts

Subscribe

Pages

Search