ICT Management. > Making Policies & Best Practice
Security > Disaster Preparation & Recovery

Could your IT cope if your office burned down?

By Lasa Information Systems Team

What would you do if all your computers were stolen, broke down – or were destroyed in a fire? On one fateful day in August 2001, one voluntary organisation in London faced just that test when a blaze broke out in their building.

The principle that “if something can go wrong, it will” was vindicated all too clearly when a fire struck Thames Reach Housing Association at their office near London Bridge. The fire – a small blaze in an office which may have started, ironically enough, with an electrical fault in a computer – should have been easy to contain. But the office was on the far side of the building, and the road giving access to it was too narrow for the fire engine. So the fire spread and much of the building was destroyed, including Thames Reach’s network of 35 computers and three servers. Worse was to come.

Thames Reach had developed procedures to back up their data – tapes were made daily, weekly and monthly and should have been taken off site. But staff shortages meant that the tapes remained in the building. The good news was that the tapes were kept in a fire-proof safe – the bad news was that the safe was buried somewhere in the burnt-out remains of the office, so the tapes weren’t much use.

The first thing to say is that Thames Reach were extremely unlucky – they couldn’t have foreseen that the fire would be so disastrous, and they had thought through a sound procedure only to have it fail. Most important, as a result of regular fire drills, they were able to evacuate the building promptly and get all their staff out in safety.

Back It Up

What lessons can other organisations learn from this story? Computers are a vital part of any advice agency today, yet organisations frequently don’t have any plans for how they would cope if a disaster wiped out their IT system. Though they may not happen every day, such disasters are far from unusual: a key computer system may simply stop working, computers may be lost in a burglary, or an office may suffer a fire or flood.

Disasters can seem unlikely, and the news that you should prepare for the worst is something nobody wants to hear. But ask yourself “How long could our organisation survive without computers? If we lost the machines and all the data they contain, could we survive at all?”

The main thing you must do is repeat the following mantra – I will make backups regularly, and I will take them off-site - and act on it!

“Regularly” means that you should make backups every day, so make sure you have hardware and software which allows you to do this. For all but the smallest agency this means a tape drive and at least a dozen tapes, which will cost some hundreds of pounds. There are alternatives – such as rewriteable CDs, CDRs, DVDRs or Zip disks, all of which are cheaper. But the capacity of all those media is a lot less than a tape – Zip disks hold 100MB or 250MB, while CDs can hold up to 700MB and DVDRs up to 5GB.

If you have more data than this, backing everything up would mean you had to put in one disk, wait for it to fill, put in another, and so on. In practice, this is the sort of system which quickly breaks down. A tape should have the capacity to back up everything on your server – you just need to change the tape once a day. Get advice from the company that supports your system about which hardware is right for you.

Develop a backup schedule

This means that you should still have files from months ago, without needing to buy a tape for every day of the year – which is what you need when someone tries to open a financial plan written three months ago and finds that the file is corrupted. A schedule might work as follows: you have four tapes for days of the week, let’s say Monday, Tuesday, Wednesday and Thursday. Every Monday you record over last Monday’s data. For Fridays, you have a further four tapes for the second Friday of the month, the third Friday, and so forth. Finally, you have six more tapes, each for the first Friday of month – these get overwritten every six months. In this way, you have a good chance of finding any file from the last six months, without having a tape for each of 130 working days.

When you’ve made the tapes, disks or whatever, get them out of your building. Arrange for a member of staff to keep them securely at home. Check that this system will continue to work through staff holidays or – as in Thames Reach’s case – if posts are vacant. Check also that you can restore files from the backup tape.

Finally, ask your system support company to check the tape drive periodically – tape heads can get misaligned, so that tapes created on a drive will restore from that particular one but not from others.

Not Just Backups

These precautions are the bedrock of surviving a disaster, but others are important too. For example, make sure that someone keeps contact details for all staff members securely out of your building – and updates them.

Ensure that you also keep off-site a full inventory of all the hardware and software you possess – it will be essential for the insurance claim, and for getting the equipment replaced. Bear in mind that the more standardised your hardware and software, the easier it will be to replace.

Thames Reach had some luck in this respect – they had recently employed consultants, who developed an IT specification for them, so they were able to base their inventory on this. Also, they were able to reconstruct some computer data from files that staff kept on their own computers when working from home.  However, as their insurance depended on them having off-site backups, the insurance company did not pay the costs for them to reconstruct their data – for example, the £25 an hour that the bank charged them to supply them with the cheques they had written.

Of course, you should make it less likely you’ll suffer a disaster in the first place by managing your IT properly – for example, make sure your virus protection software is updated. If you have a server, ensure that a power cut won’t cause it major damage by buying a UPS (Uninterruptible Power Supply) – a sort of large battery that will tide the computer over small interruptions in electricity, and shut it down properly if required.

Lessons Learned

Chris Smith-Gillespie, Finance Director at Thames Reach at the time, had overall responsibility for IT.

How did the organisation cop with the reduction of a 35-machine network with three servers to five standalones in a temporary office?

“The first thing you must do is to prioritise – our first priorities were to secure our income and to pay our staff. We’d run the payroll on the morning of the fire, so people had been paid for that month – but we have 180 staff, and had four weeks to reconstruct our data.”

Despite the tough deadline, everyone was paid. Chris emphasised that those priorities have to take into account the stress that staff will be under. In some cases that will include trauma from the disaster itself. Thames Reach staff didn’t have that to cope with, but every detail of their working practices has been thrown up into the air. This is not the time to demand too much of people – Chris explained that:

“I’ve very much taken the attitude that the glass is half-full – this is a major inconvenience for us, but not a disaster.”

If she had her time over again, would Chris have done more to prepare for a disaster? Thames Reach had discussed some issues with their treasurer and insurance company, so they had an informal strategy. However, they didn’t have a formal disaster recovery strategy of the kind that commercial firms often do, with detailed assessment of the risks they faced.

Chris feels that every organisation should think about what issues they might have to face, and what procedures they would put in place to respond. On the other hand, she feels that while disaster recovery is an important part of business planning, voluntary agencies with limited resources need to be cautious about investing a lot of money in developing formal recovery procedures.

“Written documents can give you false comfort – these procedures are meaningless unless they are checked and updated on a very regular basis.”

No-one likes to think about the worst that could happen to their organisation, but for many agencies an IT failure could threaten their whole existence. It’s worth spending a little time thinking through how you would cope if a disaster happened to you.

Resources

Techsoup's comprehensive disaster planning and recovery toolkit.