Superstorm Sandy Puts DR Plans To The Ultimate Test

November 5, 2012 Alex Woodie

When Superstorm Sandy roared ashore a week ago, residents of the Northeast knew they would be in for a wild ride. Sandy didn’t disappoint, as she brought a record storm surge to the New York City area and devastated the Jersey Shore. The storm also poked hurricane-sized holes in the disaster recovery (DR) plans of some businesses, like those that placed backup generators in flood-prone areas. For some IBM i shops and managed service providers (MSPs), DR plans were strained but not broken, and gained the Sandy stamp of approval for good DR housekeeping.

Before Sandy made landfall as a class one hurricane last Monday night, most companies in the New York City and New Jersey areas knew that little business would be done in the immediate aftermath of the storm. After all, the New York Stock Exchange pre-announced that it would be closed until Wednesday, and many businesses in the area–not just financial services firms, but retailers, manufacturers, and distributors–followed suit by closing. It was to be a four-day weekend.

Representatives with the Florida MSP Premise Inc say most of its IBM i customers in the area with high availability (HA) contracts chose to shut their servers down instead of performing a failover to IBM i resources managed by Premise. “Most customers felt the interruption would be minimal, meaning they could be up and running between 12 and 24 hours,” says Premise engineer Ron Venzin. “Taking the power down option was preferable to many of these customers, even though they had a product like MIMIX.”

But when the lights failed to come back on in New Jersey, Manhattan, and Long Island, companies counting on a four-day weekend suddenly had to enact their DR plan if they didn’t want it to turn into a five-, six-, or seven-day weekend–or worse. When the cleanup began in earnest last Wednesday, Lower Manhattan businesses without an effective business resilience strategy–that is, those without redundant backup generators, redundant servers, and redundant network connections–simply weren’t able to do business.

Hurricane Failovers

The call center has been very busy at Vision Solutions, the biggest provider of HA software in the IBM i ecosystem, and owner of the MIMIX, iTera, and ORION HA and DR products. Vision has 1,100 customers that were in the path of the storm, and had been involved with about 70 planned and unplanned failovers as of Friday, according to Pete Robie, senior vice president of customer care for Vision. Additional customers performed failovers without any assistance from Vision, he says.

Affected Vision customers were all over the map–from the casinos in Atlantic City, New Jersey, north to Burlington, Massachusetts, south to Charlotte, North Carolina, and west to Ohio. Many of the customers chose to execute role swaps prior to the storm, while others switched only after losing power, being flooded, or both. Some customers were still running on backup systems as of last Friday (November 2), while others had switched back to their production systems. Some customers lost both their primary and secondary sites, and were working with Vision to recover their systems.

Robie expects a second wave of failovers to occur for customers running primary systems on backup generators, and who are having a difficult time obtaining fuel to keep the generators going. “We still expect to hear from some of the customers. That’s why we’re going to continue to be on high alert,” he says.

Maxava reported one customer performing a failover. The client–a New Jersey insurance company–failed over to its secondary machine, and was still running on it as of last Friday. Maxava had several other failovers with clients who run the Maxava high availability software via MSPs.

MSPs Called to Action

MSPs who run for HA or DR as a service for IBM i shops were quite busy last week. Velocity Technology Solutions was put on alert by several clients as Sandy approached, and the company actually executed failovers for three clients. Velocity spun up 35 Windows servers, three IBM i servers, and the necessary network and VPN lines to support 1,500 workers at the companies, which are distributors and manufacturers located in New Jersey, Massachusetts, and Pennsylvania.

“A hurricane is a good news-bad news situation. It gives you advance warning, so we started getting ready as early as Friday,” says Richard Dolewski, a vice president of infrastructure and business continuity services at Velocity. “A lot of these hurricane threats are overstated panic-ridden news media event. This thing was real. I think we all realized that pretty early in the game.”

One of the customers Velocity helped last week still relies on tape. The company was lucky enough to successfully deliver a fresh backup to Velocity before the storm hit. However, the experience highlighted the benefit of electronic vaulting. “The roads, airports, trains–everything is shut down on the east coast,” he says. “That’s why we built this vaulting product for iSeries, which is based on MIMIX.”

mindSHIFT Technologies weathered the storm exceptionally well. The MSP, which is based in the middle of Long Island, New York, performed proactive failovers of several Windows applications, but didn’t perform any failovers of IBM i applications.

Bob Lamendola, vice president of operations for mindSHIFT, says the company’s IBM i clients were confident enough in the high availability software to allow it to automatically failover if power or network connectivity is lost. “The nice thing about the iSeries platform is the [high availability] software is highly automated,” Lamendola says. “There’s really no reason to proactively fail over unless it’s such a dire state that a failure is inevitable.” mindSHIFT has worked with all types of replication software, and is currently focusing on Maxava’s HA software, he says.

mindSHIFT kept its 80,000 square foot data center running for five days despite losing power from the grid thanks to a pair of diesel generators located on trailers, and contracts with local fuel delivery companies that re-supplied the generators every 12 hours. While one data center operator was forced to use a bucket brigade to keep its diesel generators fueled, and an IBM BCRS in New Jersey suffered a power outage for several hours, mindSHIFT’s generators kept the juice flowing for 300 or so racks of IT equipment, which serve upward of 700 customers.

Full-Service Recovery

Sandy turned mindSHIFT into an impromptu recovery center, a la the SunGard or IBM BCRS business model. “We had several people walk up with cars full of servers, asking us if we can rent space,” Lamendola says. “We’re not a DR center from that perspective. Being a business recovery center for people is not our core business model, but we kind of did it ad hoc this time.”

A spokesman for SunGard Availability Services reports that the company has had 115 disaster declarations as of Friday, with more coming in every day. That is nearly the same number of declarations the company had following 9/11. “The enormity of the scale of the storm was unprecedented,” SunGard spokesman Chris Drago says.

SunGard has about 1,500 workspaces (desks, PCs, phones) in use at its Business Continuity Center in Carlstadt, New Jersey, which is serving more than 6,000 displaced workers. The company is at capacity in New Jersey, but still has seats available in Boston and Philadelphia. It is also in the process of opening two mobile data centers in New York City in the coming days.

SunGard’s Carlstadt facility was one of the only functional buildings in a particularly hard-hit area–the only dry area with electric light–and it became a make-shift mobile command center for emergency responders and residents during the storm, Drago said. The company sent about one-third of its workforce, or about 1,000 employees, to the Northeast in preparation for its customer declarations.

Representatives for IBM BCRS did not respond to requests for comment by IT Jungle before this issue of the The Four Hundred went to press. An IBM spokesperson said Big Blue would not comment on the company’s DR efforts regarding Sandy.

Representatives of three other MSPs with IBM i HA and DR hosting businesses contacted by IT Jungle for this story–including Symmetry, Datanational, and Logicalis–reported no affected customers.

DR Preppers

Sandy has turned up stories of both good and poor DR planning. There are perhaps no DR stories worse than what happened at New York University Langone Medical Center and Bellevue Hospital Center. More than 1,000 patients had to be evacuated from the hospitals in the middle of the storm due to power failures, even though both hospitals had a pair of backup generators. While the hospitals’ secondary generators were located above ground level and theoretically should have continued to work, the pair of secondary generators had critical components, like fuel pumps and fuel tanks, located in basements, which were flooded.

But there was general agreement among HA providers and MSPs that the businesses they were in contact with seemed well-prepared for Sandy. “In this particular case, we have had more customers do active role swaps than any other storm or event that I can remember,” Vision’s Robie says. “The big hurricanes that happened in 2005 down in New Orleans–those were more traumatic. Maybe they weren’t as well prepared. With this storm, it seemed there were a lot more customers that were more prepared.”

The memory of Hurricane Irene’s impact on the Northeast in October 2011 was still fresh in many people’s minds. One of Velocity’s HA clients in Massachusetts who got hit by Irene was wary of Sandy, and declared a disaster and performed a role swap before landfall. Sandy turned west and didn’t hit Massachusetts hard, but the company would have been prepared if it had.

Lamendola, who was helping customer set up DR plans on the fly last week at mindSHIFT, also recognizes a change in attitude. “After two years and two hurricanes a year apart, customers are starting to understand that they need to take more ownership of their DR capabilities, and take it more seriously,” he says. “People just assume that they can live without business for a certain period of time, until it actually happens, and then they realize it’s really not possible.”

Editor’s Note: As most of you know, I live at the northern tip of Manhattan island with my family, but what you may not know is that I live high up on a hill, facing due east, right below a cellular equipment tower. The building, which was built in 1939, faces out into a vast open valley carved by the Harlem River, several hundred feet into the air. I thought the cell tower might break loose a time or two, or the tired old windows might give way in my pre-war apartment building and blast in on us, but my family and my neighbors were all spared. Unlike so many others who were not so lucky and lost so much. There is much to be sad about, but this is just one aspect of what is becoming the new normal. Our hearts go out to those who lost family and friends from the storm. Do what you can to help, and I will, too. And get your plan together for when this kind of event comes your way. Being prepared really makes a difference, taking evacuation orders seriously does, too.–TPM