As I See It: Just Bag IT
August 6, 2018 Victor Rozek
Only two days in, my stay in Lassen Volcanic National Park was cut short by vanishing stars. Evidently the “new normal” knows no boundaries and has no respect for vacationers. It seems that every year now, there are fires, terrible fires, consuming the West. In Redding, California, 50 miles west of the park, 966 homes and six people were lost. More are missing. Over 90,000 acres of drought-stressed brush remain ablaze. As of this writing, 38,000 residents may have to be evacuated as swirling winds are spreading flames in all directions.
But in the distant mountains I knew nothing of this, except that smoke was quickly engulfing the park in toxic fog. Suddenly the stars disappeared and the moon glowed dirty red. Sleep became as elusive as a breath of clean air. If things didn’t improve by morning, it would be time to leave.
They didn’t improve. If I’d had a vacation disaster recovery plan, I would have implemented it. As it was I settled for a very long drive home through choking smoke.
Last December, I wrote a piece called “Disruption.” It was a predictive essay on what IT professionals could expect in the coming year. My premise was that global warming is in the process of creating hellish conditions that will increasingly disrupt our lives and business operations. OK, so I’m not Nostradamus, but last year in California alone there were about 9,000 wildfires that destroyed nearly 11,000 structures and killed 46 people.
This year looks much the same. Just 100 miles north of Redding, another mega-fire threatens 10,000 homes. Fires are no longer seasonal events. Firefighters now speak of fire years. For most of us, disasters used to be something that happened to other people; we are all “other people” now.
At one time IT disaster recovery plans were voluminous documents written by managers and promptly stored in some anonymous filing cabinet where they would not be a bother to anyone. Once written they were seldom updated, and rarely – if ever – were they actually tested.
With the advent of cloud computing and the emergence of companies providing HA services, formal disaster recovery plans became less important. Hiring a third party to protect against the vicissitudes of nature and the destructiveness of digital criminals became the industry’s de facto disaster recovery plan.
Although HA services, or an in-house rendering of data redundancy, are wise and essential precautions, nonetheless many smaller IT installations are ill prepared for dealing with disruptions. For those with budget and personnel constraints, HA, DR, cloud-based computing, even offsite backup storage are too often either luxuries or afterthoughts.
Just coping with day-to-day IT services can be a challenge for small businesses. Safeguarding data security, monitoring for threats, offering technical support to employees for their collection of desktop and hand-held devices, upgrading those devices as needed, maintaining servers, installing software, training employees to use it, liaising with vendors and fielding the endless flow of requests and complaints – all of this leaves little time to consider contingencies.
Regardless, a catastrophe can destroy a building, a community, or a life’s work in a few horrendous moments, so however little time there is, it’s far better to make contingency plans, even imperfect ones, before an actual event delimits choices.
According to IBM, here are the six major goals of a disaster recovery plan.
- To minimize interruptions to normal operations.
- To limit the extent of disruption and damage.
- To minimize the economic impact of the interruption.
- To establish alternative means of operation in advance.
- To train personnel with emergency procedures.
- To provide for smooth and rapid restoration of service.
The first three and the last one are self-evident and will be achieved to the degree that goals 4 and 5 are successfully implemented.
The key is to have an alternate means of operation, ideally housed in the cloud or on a system provided by a managed IT vendor. Service providers not only replicate your hardware and software configurations, but offer expertise which, during an emergency, may not otherwise be available.
Less desirable but workable is having a backup server located a goodly distance from the main facility. If a disaster is of sufficient scale, having a substitute server nearby will only double your losses.
Assuming an alternate site has been identified, it will require people to staff it. Identifying and training two skeleton-staff teams provides the best chance of minimizing service interruption. Depending on the nature of the disaster, employees may have their own homes and families to tend to. Relying on one or two key people may be unrealistic. Redundant training provides flexibility and improves the odds of quickly rebooting the business.
An emergency move to an alternate site will be aided by having an IT version of a go-bag on hand. For those unfamiliar with the concept, a go-bag is recommended for people living in high-risk environments vulnerable to immediate unplanned evacuations. Normally it would contain such things as valuables, important documents, a flashlight, portable radio, batteries, a first-aid kit, clothing, and enough food and water to last a few days. An IT version would contain everything needed to restore computer operations offsite.
That may include an organization chart with addresses and contact numbers for all employees. A hard copy of the system configuration. Software installation media. Software licenses and security keys. The latest full and partial backups. A flash drive containing photos of your facility and equipment for insurance purposes. And any pertinent documents (customer contact info, receivables, check books). In short, everything you would need to start over. Some of this information may be redundant since it should reside on backups, but during volatile and unpredictable conditions, redundancy is your friend, particularly if the DR plan hasn’t been thoroughly tested.
It is also helpful to create an evacuation to-do list. Note the items you want to add to your go-bag, and anything that needs to be done to safeguard first responders and the facility such as turning off utilities, closing interior doors, and locking exterior doors. Of course, if you have enough time and help to load your server into somebody’s pickup truck, that works too.
Finally, make sure you leave enough time to get the hell out.
There is an arbitrary cruelty to life that allows one man to vacation in the mountains, merely inconvenienced by smoke, while others see their homes and businesses destroyed, or worse yet, perish in flame. And while even the best planning does not guarantee we will be spared the worst consequences of a warming planet, a lack of preparation is an invitation to regret.
