Business Continuity Planning: Are OS/400 Shops Ready for Disaster?

None of us, even those who live a good distance from Hurricane Alley, will soon forget the devastation caused by Katrina and Rita in the summer of 2005. These names, now infamous, have joined and even surpassed the likes of Andrew and Charley and Hugo in our collective memory. But, having presumably learned from experience–or at least been awakened from our slumber–are we ready for Alberto, Isaac, Leslie, and Oscar? The odds are a few of these names will make headlines in the 2006 hurricane season.

Last week, AccuWeather released its 2006 hurricane forecast, and it isn’t pretty. “An active hurricane season appears imminent, which could have major repercussions for the U.S. economy and the one in six Americans who live on the Eastern Seaboard or along the western Gulf of Mexico,” says chief forecaster Joe Bastardi. “There are few areas of the U.S. East Coast and Gulf of Mexico that will not be in the bull’s eye at some point this season,” said Ken Reeves, AccuWeather’s director of forecast operations. Ironically, though, the region that was hammered the hardest last year–the central and eastern Gulf Coast–has one of the lower probabilities of receiving another major hurricane strike in 2006.” Geographies with a “high” probability of suffering a direct hit are the east coast of Texas, south Florida, and New England. Those targeted with a “very high” probability of a direct hit are the Carolinas, New Jersey, New York (Long Island), and Connecticut.

For those who enjoy jokes about forecasting the weather, chuckle away. The rest of us who are sitting ducks in the paths of the storms had better be making plans.

Lynn Aucoin, IT manager for Drago Supply in Port Arthur, Texas, located on the coast of the Gulf of Mexico about 250 miles west of New Orleans, is taking the forecast very seriously. Drago is a wholesale distribution company, stocking more than 30,000 items of industrial supplies such as power tools, contractor supplies, and janitorial supplies, serving such industries as refining, petrochemical, construction, and manufacturing . The company, which is celebrating its 75th anniversary this year and has eight branch locations in Texas, Louisiana, and Arkansas, runs its business with an iSeries 810 with V5R2 and four additional Wintel servers for such functions as e-mail, fax, and Web content.

Last year, the company was hit, reports Aucoin, when one of its branch offices lost power for about two weeks. “We were able to get a backup generator and get back up and running pretty fast,” he says, but the absence of a real business continuity plan taught them some lessons that they are taking to heart. To rectify that, a committee was formed made up of managers, the company owners, and other personnel. Together, they reassessed their preparedness from the points of view of their customers, owners, employees, and vendors. All staff was named either “first responders” or “others.”

As a result of this effort, Drago Supply now has a disaster preparedness plan that covers a lot of issues they had never thought about. “We now make sure that people will take their cell phones with them,” says Aucoin, “and that they have printed contact lists from any contact management software like Outlook.” And, while the company’s phone system hasn’t changed since last year, Aucoin has made sure that the staff will communicate with the company’s customers in advance of a storm and provide them with emergency numbers.

Port Arthur does fall in an area that might be subject to evacuation. In fact, during Hurricane Rita, some of the Drago Supply staff did move to the Houston facility and operated from there. “Our biggest shortfall was in communications,” recalls Aucoin. “In the event of a disaster, everybody spreads out and it’s hard to know where everyone is. Cell phone batteries run down, so we’ve made sure we now have backup cell phones with mobile chargers as well as AC chargers. There are just some of the little things you don’t think about.” All employees now have checklists as well as emergency numbers so that they can call to report where they are and what their status is.

Another lesson learned is that many people have forgotten (or never knew) how to do some things manually. “For example,” Aucoin says, “we have a warehouse management system that barcodes everything, and we have handheld scanners that tell people where to go to pull the merchandise. When the system was down and we didn’t have power, our people had to go pull the merchandise by hand, but they didn’t know how to do it or where to go because they system had always automatically provided them with the bin locations.” Furthermore, many of the people who were able to return to work the fastest were the management staff, who not only did not know how to pull the merchandise but who weren’t current on the day-to-day operating procedures. These issues have now been addressed through printed documentation and additional staff training.

Drago also now has a better disaster recovery plan in place and has made contingency plans to move the iSeries box to another location in the event of a disaster. “If we know a storm is coming,” Aucoin says, “we will pick up and move the iSeries to our furthest branch location, probably in Little Rock. The cost will be minimal since we have our own fleet of trucks. We’ll just load it up and take it to our other branch and tie it into our communications link over there. If possible, we would try to do this the weekend before the hurricane hit, and could probably move it and have it up and running again within 24 to 48 hours. This would get us out of the direct path of the storm.”

This plan should work for hurricanes, which tend to announce themselves in advance. For other disaster situations, one might not be lucky enough to have advance warning. So Aucoin is looking at the possibility of implementing a mirrored system. At this point, however, the company has not made a final decision on that.

With some luck, Drago Supply will never have to use its disaster preparedness plan. “But if we do have to implement it,” Aucoin says, “we’ll be much more prepared. It’s still a work in progress, but we are much further ahead than we were.”

Gulf Data Systems offers a perspective on business continuity planning from two different standpoints, both as a company located in a potentially high-impact disaster zone and as a software vendor and service provider. With headquarters in Mobile, Alabama, Gulf Data Systems develops and operates software and offers services to credit unions, including PC installation and support, local and wide area networking, network security, accounting and payroll services, staffing, consulting, and auditing. And, while the company does have customers nationally, Gulf Data Systems’ primary customer base is in the south, primarily in the hurricane-prone regions. The credit unions in turn have customers who may at any time, but especially during a disaster, need to access funds at a moment’s notice. And there are laws requiring that banks and credit unions make funds available.

Gulf Data Systems runs its business on an iSeries 520 with three logical partitions running Windows and Linux; it also has Windows servers. The company currently has a hot site backup in Montgomery, Alabama, and another in Grand Rapids, Michigan.

Leo Vaulin, the company’s chief executive officer, reports that Gulf Data Systems learned a lot from the experience with hurricane Katrina last year. “You think about your primary server and you know you have that backed up, but you don’t always think about the secondary systems that are now necessary to allow the organization to function, such as telephones and secondary servers for audio response or home banking.”

Prior to the storm, Gulf Data Systems had implemented a business continuity plan for its own business and felt pretty good about its own level of preparedness. It had sufficient redundancy in its power systems and in Internet connectivity, and the company had recently installed an underground, fiber-optic connection right into a SONET, a high-speed Internet backbone ring. Gulf Data Systems also has an onsite generator with portable generators to back that up, as well as multiple air conditioning systems.

“We were in great shape,” Vaulin says, and although Gulf Data Systems’ office was not directly hit by the storms last year, many of its customers were in real need. He points out that when Katrina swept through coastal Mississippi, many offices were completely under water and unusable for many months. “In normal disaster situations, you assume that the surrounding region has sufficient infrastructure to be able to support you. All you have to do is physically move to some other place. You move your server there and go back into business. Well, you can’t do that when the entire region has suffered the way that Louisiana and Mississippi did after Katrina. For example, you couldn’t order brand new telephone lines for several months in the state of Mississippi. All of the utilities folks were tied up just repairing existing lines, and that was the correct call on the part of the local telephone companies. So what do you do?”

Because Alabama was not as affected, Vaulin says, and because Gulf Data Systems had extra lines coming in through its own telephone provider, the company was able to dedicate a number of those lines to supporting an extra call center that it made available to credit unions, regardless of whether or not they were current customers. “We had credit unions coming in and using our data center as a temporary call center. We were also able to crank up online banking solutions for credit unions that hadn’t even been using our software or services. We ended up deploying solutions in a few hours that let their clients transfer money and use ATM cards, something that’s very important when people are displaced.”

The phone communications are critical, Vaulin explains, pointing out how much we rely on debit cards, even for things like getting gas. Most of those card authorizations are done over dial-up phone lines. “When those are down, people don’t have cash. Those are the kinds of things we need to think about when disasters hit.”

But although the Gulf Data Systems story from last year can be viewed as a success, there were still experiences that the company is building on this year, says Vaulin. “The storms did open our eyes to the fact that our customers–the credit unions–were nowhere near prepared enough. The lesson learned was that I need to reach out ahead of time to make sure that everyone is prepared and knew how to reach us if anything happened. I would hear stories about credit unions that were really in bad shape and I couldn’t do anything; I had no way to reach them. Here we were sitting here with air conditioning, food, water, phones, and the ability to serve them, and I couldn’t reach them.”

The company has also done more to instruct and plan for communications with its own staff, Vaulin says. This plan includes a designation of primary, secondary, and tertiary roles for everyone in the company, with a periodic review of those designations to ensure that no one is assigned to a primary role if he or she can’t take on the responsibility. “We can run basic operations with about 25 percent of the staff,” he says. “At any point in time we can find enough people who are ready to relocate in advance of a storm, and we encourage these people to take their families with them and we pay for their hotel bills. The company will also make the public areas of the buildings available to anyone who wants a more secure location.”

Vaulin believes that last year’s experiences were a big wake-up call on disaster preparedness for companies throughout the South. “I don’t think there will be a single IT shop that will still have its head in the sand this year.”

Hurricanes are, of course, only one type of disaster that companies need to plan for. But last year’s experiences were certainly the catalyst that got a lot of people thinking and planning for the worst. For those who are just embarking on a business continuity plan, Vaulin offers this advice: “Talk to people who have been through it. Get them to tell you what it’s really like. You are going to think about the things that are really obvious, but you’re going to forget something. Let others roll play your plan and poke holes in it. Midrange users are a close-knit community and chances are that you’ll be able to reach out to someone who’s been through it. Create a network of people around you and share.”

If something good can be said about hurricanes, it’s that we know they are coming and have some opportunity to prepare (though they may fool us by changing paths at the last moment). On the other hand, their paths of destruction can be very broad, covering wide geographic regions. But many other disasters–both natural and manmade–have different characteristics. Earthquakes, tornadoes, and terrorist attacks, for example, give little or no warning and, in general, are more limited in their geographic impact. Still others have (or may have) different characteristics: the avian flu, biological or chemical disasters, and blackouts or brownouts. Next week, I’ll take a look at the experiences and thoughts of some companies in earthquake-prone regions to see how they plan for disasters that offer no warning in advance.