This is a report of an incident we can all learn from. The names have been changed to protect the ignorant. Elizabeth is the receptionist and office manager at Acme Services, she also does most of the bookkeeping. Yesterday she got an error while entering work orders. Elizabeth is fairly computer literate so she started some basic troubleshooting. She discovered that a directory of accounting files was apparently missing. The system and the files are absolutely essential for accounts payable and accounts receivable. Acme does not employee a full-time system administration staff. The owners nephew, Frank, is a computer wizard and helps them with computers when needed. Fortunately, Frank had installed a backup system that backups up all of the data files to tape every night. Elizabeth changes the tapes first thing every morning.
The end of the month is only a few days away and Elizabeth has a number of invoices to send out as well as quite a few payments to vendors and some commission checks. She can't do any of this without the system and the missing files. She was somewhat familiar with the backup system so she was not in full panic yet. Elizabeth got the previous nights backup tape, which she had just pulled out of the tape drive an hour earlier, and placed it back into the tape drive. She attempted to restore the files but got nervous when a message warned about overwriting some files. She called Frank and explained what was going on.
Frank was puzzled about what happened to the files in the first place. He was legitimately concerned that the hard drive may be corrupted or failing. Frank decided it would be prudent to backup the data on the HD before doing anything else. He asked Elizabeth if she had changed the backup tape this morning and she replied affirmatively. He used a remote desktop tool to access the computer remotely and performed a backup of all important data on that computer. Frank also suspected that there was a distinct possibility that there was nothing wrong with the computer or the hard drive and the files were just accidentally deleted. He decided to restore the deleted files and see if the problems occur again before doing anything drastic like replacing the HD.
Frank asked Elizabeth to insert last night's backup tape so he could run the restore. Elizabeth replied that the tape was already in the drive. Apparently there was some miscommunication about when the tape was changed and which tape was in the drive. Bottom-line, Frank had just backed up the hard drive and overwritten the most recent backup of the missing files. WHOOPS!
Forget about pointing fingers or placing blame. This kind of mistake is common with backup media. The fact is the data is gone, and the current backup is overwritten. Frank finally restored the missing files from a backup that was 3 days old. It initially took about four hours of Frank's time to work on this problem and restore those files. Elizabeth is still making phone calls and re-entering data that was lost. When it's all over, Elizabeth will probably spend about 20-30 hours recovering from this situation, and there will be at least a few customers and vendors who will be upset about payment and billing problems over the next few weeks.
The hard cost of this incident is far in excess of the cost of using an online backup service. An online backup service would not only have avoided the overwritten backup tape, but also saved enough money on backup hardware and media to pay for the backup service for several years. In addition if an online backup service were used, Acme would have their critical data off-site and protected from a fire or other disaster.
No comments:
Post a Comment