PSPS: The Latest Threat To Business Continuity
November 6, 2019 Alex Woodie
Businesses in Northern California learned about a new threat to business continuity in October: the Public Safety Power Shutoff, or PSPS. With weather conditions ripe for the rapid spread of wildfire, Pacific Gas and Electric de-energized thousands of miles of power lines, sending thousands of homes and businesses into disaster recovery mode.
In late September, as weather conditions turned dry, hot, and windy, PG&E warned that it would have to cut power across large swaths of its Northern California service territory, potentially shutting off power to 2.4 million people. In the wake of power line-sparked wildfires that destroyed tens of thousands of buildings and killed over a 100 people in the last two years, PG&E decided extreme prudence was preferable to delivering electricity over its aging grid.
By the end of October, after a series of severe weather events, the PSPSs were not as bad as the utility company had initially warned. During the last “red flag” windstorm, which generated 90 mph gusts to kindle-dry mountaintops the week before Halloween, the bankrupt utility “only” had to cut power to areas serving 940,000 across 36 counties at a given time. It was a small victory, especially considering that Santa Rosa was evacuated due to the Kincade Fire, which was suspected of being sparked by PG&E’s equipment.
A wind-driven wildfire is one thing, but the PSPSs had their own impacts. Hundreds of thousands of schools, businesses, and homes were without electricity, some for almost a week at a time. Food spoiled without refrigeration and cell towers went idle as battery backups ran out, as work and play came to a grinding halt in the most prosperous region on the planet.
Without backup generators, residents were forced to charge portable electronics in their cars. But getting gasoline proved a trick, as some gas stations couldn’t pump fuel without electricity. And if you happened to drive an electric car and didn’t have an off-grid source of power, well, you were just plain out of luck.
Sales of portable generators spiked during the first PSPS in early October, as residents and businesses scampered to power refrigerators, lights, and electronics. According to one report, there were no portable generators available for sale in the entire Bay Area. Stocks of companies that manufacture generators surged by 10 percent. Generac, the country’s largest maker of generators, estimated a $100 million increase in sales from California through 2023 thanks to PG&E’s declaration that PSPSs would continue for the next 10 years.
The overall impact of the PSPSs was likely in the tens of billions of dollars. Companies and individuals will have a hard time recovering that from the San Francisco-based utility, which filed for bankruptcy protection in the wake of the Camp Fire that leveled the foothill town of Paradise one year ago.
For businesses operating in California, PG&E’s massive power outages should be a wakeup call. Disruptions due to wildfires, earthquakes, and floods are de rigueur in the Golden State. But the PSPS adds a disturbing new wrinkle to the disaster preparedness worksheet.
Having an alternative source of electricity is a good first step to avoiding the impact of a power outage. Companies operating their own gear can usually get the electricity they need from backup power systems, such as a permanently installed diesel or propane generator. However, as fuel supplies run low and supply chains are interrupted, you might not be guaranteed of getting a truck in to re-fuel.
Depending on the industry, getting an IBM i server up and running could be pointless if all the other machines are not running, or if your workers can’t get to work. The larger the operation, the larger the backup generator has to be.
A modern supermarket store would require an industrial-sized generator to keep millions of dollars’ worth of food and pharmaceuticals from spoiling. Consider that a 40,000-square-foot grocery store could require on the order of a 500 kilowatt generator, which would cost on the order of $200,000 to install. A large factory could need one or more 2,000 kW generators, costing upward of $1 million apiece, to keep machines running when the grid is not available.
Hospitals are required by law to have backup electrical sources, since people’s lives depend on a steady flow of electricity. Companies in the hospitality and healthcare businesses often have lower electrical needs, and can often justify installing backup diesel generators. It’s worth noting how many of the recently constructed casinos owned by Native American tribes — many of which rely on IBM i servers — were up and running across California during the power outages last month. (With many citizens unable to go to work or do much of anything during the blackouts, the idea that it’s business-as-usual at your local casino could be a great marketing tool.)
Besides keeping their core servers, equipment, refrigeration, and HVAC units running, businesses should look at other ways they can keep online during outages, including how they design their mobile apps.
One company that was able to keep running during the PSPS last month was Pied Piper Exterminators, which is based in California’s Scotts Valley. With mobile connections down, the company’s field agents had no connection to its IBM i server running VAI enterprise software.
However, because the field workers were equipped with mobile apps that were resilient to a disconnected network, they were able to place and process orders, email service completion notifications, and post invoices and payments to accounts receivable.
“When the mobile app was developed, the VAI mobile team recommended that we include this offline feature, and I’m sure glad we did,” Pied Piper Exterminator President Byron Finney said in a press release. “In today’s world where we rely heavily on mobile technology, particularly the iPad and our mobile service app, we were able to continue our business operations despite having limited power.”
If a power outage takes a primary server offline, a high availability solution can keep business applications up and running from a secondary site, according to Becky Hjellming, senior director of product marketing at Syncsort.
“Ideally, enterprises locate HA systems sufficiently far from the production data center to ensure that the secondary copy is safe from regional events,” she tells IT Jungle. “In the case of an outage, HA solutions can switch operations to the second system proactively or in reaction to an outage, without losing data or experiencing downtime. This means customers, partners and staff can continue to access systems undisrupted.”
Power outages have always been on the minds of DR professionals. However, with PSPSs becoming the norm across California, it’s now the threat of an emergency, not just the emergency itself, that is triggering the loss of power. IBM i shops in affected areas should plan accordingly.
RELATED STORIES
How The Latest TRs Bolster HA/DR And Security
