7. Backups

Contents:
Make Backups!
Sample Backup Strategies
Backing Up System Files
Software for Backups

Those who forget the past are condemned to fulfill it.

George Santayana, Life of Reason

Those who do not archive the past are condemned to retype it!

Garfinkel and Spafford, Practical UNIX Security (first edition)

When we were working on the first edition of Practical UNIX Security , we tried in vain to locate a source of the quotation "those who forget the past are condemned to repeat it." But no matter where we searched, we couldn't find the source. What we found instead was the reference above in George Santayana's book, Life of Reason . So we printed it and added our own variation.

What's the point? Few people take the time to verify their facts, be they the words of Santayana's oft-misquoted statement, or the contents of an unlabeled backup tape. In the years since Practical UNIX Security was first published, we have heard of countless instances in which people or whole organizations have had their files lost to computer failure or vandalism. Often, the victims are unable to restore their systems from full and compete backups. Instead, restoration is often a piecemeal and lengthy project - a few files from this tape, a few files from that one, and a few from the original CD-ROM distribution.

Even if backup tapes exist, there are still problems. In one case, a researcher at Digital Equipment Corporation lost a decade's worth of personal email because of a bad block at the beginning of a 2GB DAT tape. The contents of the tape had never been verified.

In another case, we know of a project group that had to manually recreate a system from printouts (that is, they had to retype the entire system) because the locally written backup program was faulty. Although the staff had tested the program, they had only tested the program with small, random files - and the program's bug was that it only backed up the first 1024 bytes of each real file. Unfortunately, this fault was not discovered until after the program had been in use several months.

Making backups and verifying them may be the most important things that you can do to protect your data - other than reading this book, of course!

7.1 Make Backups!

Bugs, accidents, natural disasters, and attacks on your system cannot be predicted. Often, despite your best efforts, they can't be prevented. But if you have backups, you can compare your current system and your backed-up system, and you can restore your system to a stable state. Even if you lose your entire computer - to fire, for instance - with a good set of backups you can restore the information after you have purchased or borrowed a replacement machine. Insurance can cover the cost of a new CPU and disk drive, but your data is something that in many cases can never be replaced.

Before we go on with this chapter, take some time for the following quick test:

When was the last time your computer was backed up?

A) Yesterday or today

B) Within the last week

C) Within the last month

D) My computer has never been backed up

E) My computer is against a wall and cannot be backed up any further

If you answered C or D, stop reading this book right now and back up your data. If you answered E, you should move it out from the wall to allow for proper cooling ventilation, and then retake the test.

Russell Brand writes:[1] "To me, the user data is of paramount importance. Anything else is generally replaceable. You can buy more disk drives, more computers, more electrical power. If you lose the data, through a security incident or otherwise, it is gone."

[1] Coping with the Threat of Computer Security Incidents, 1990, published on the Internet.

Mr. Brand made this comment in a paper on UNIX security several years ago; we think it summarizes the situation well, within limits.[2] Backups are one of the most critical aspects of your system operation. Having backups that are valid, complete, and up to date may make the difference between a minor incident and a catastrophe.

[2] We actually think personnel are more important than data. We've seen disaster response plans that call for staff members to dash into burning rooms to close tape closets or rescue tapes; these are stupid. Plans that require such activity are not likely to work when needed, and are morally indefensible.

7.1.1 Why Make Backups?

Backups are important only if you value the work that you do on your computer. If you use your computer as a paperweight, then you don't need to make backups.

You also don't need to back up a computer that only uses read-only storage, such as a CD-ROM . A variant of this is Sun's "dataless" client workstations, in which the operating system is installed from a CD-ROM and never modified. When configured this way, the computer's local hard disk is used as an accelerator and a cache, but it is not used to store data you would want to archive. Sun specifically designs its operating system for these machines so that they do not need backups.

On the other hand, if you ever turn your computer on and occasionally modify data on it, then you must make a copy of that information if you want to recover it in the event of a loss.

7.1.1.1 A taxonomy of computer failures

Years ago, making daily backups was a common practice because computer hardware would often fail for no obvious reason. A backup was the only protection against data loss.

Today, hardware failure is still a good reason to back up your system. In 1990, many hard-disk companies gave their drives two- or three-year guarantees; many of those drives are failing now. Even though today's state-of-the-art hard disk drives might come with five-year warranties, they too will fail one day!

Such a failure might not be years away, either. In the fall of 1993, one of the authors bought a new 1.7GB hard drive to replace a 1.0 GB unit. The files were copied from the older drive to the newer one, and then the older unit was reformatted and given to a colleague. The next week, the 1.7GB unit failed. Luckily, there was a backup.

Backups are important for a number of other reasons as well:

User error: Users - especially novice users - accidentally delete their files. A user might type rm * -i instead of typing rm -i *. Making periodic backups protects users from their own mistakes, because the deleted files can be restored. Mistakes aren't limited to novices, either. More than one expert has accidentally overwritten a file by issuing an incorrect editor or compiler command, or accidentally trashed an entire directory by mistyping a wildcard to the shell.
System-staff error: Sometimes your system staff may make a mistake. For example, a system administrator deleting old accounts might accidentally delete an active one.
Hardware failure: Hardware breaks, often destroying data in the process: disk crashes are not unheard of. If you have a backup, you can restore the data on a different computer system.
Software failure: Application programs occasionally have hidden flaws that destroy data under mysterious circumstances.[3] If you have a backup and your application program suddenly deletes half of your 500 x 500-cell spreadsheet, you can telephone the vendor and provide them with the dataset that caused the program to misbehave. You can also reload your data to try a different approach (and a different spreadsheet!).

[3] Unfortunately, more than "occasionally." Quality control is not job #1 for most vendors - even the big ones. It's probably not even job # 10. This fact is why most software is sold with the tiny print disclaimers of liability and waiver of warranty. When your vendors don't have that much confidence in their own software, you shouldn't either. Caveat emptor .
Electronic break-ins and vandalism: Computer crackers sometimes alter or delete data. Unfortunately, they seldom leave messages telling you whether they changed any information - and even if they do, you can't trust them! If you suffer a break-in, you can compare the data on your computer after the break-in with the data on your backup to determine if anything was changed. Items that have changed can be replaced with originals.
Theft: Computers are expensive and easy to sell. For this reason, small computers - especially laptops - are often stolen. Cash from your insurance company can buy you a new computer, but it can't bring back your data. Not only should you make a backup, you should take it out of your computer and store it in a safe place, so that if the computer is stolen, at least you'll have your data.
Natural disaster: Sometimes rain falls and buildings are washed away. Sometimes the earth shakes and buildings are demolished. Fires are also very effective at destroying the places where we keep our computers. Mother Nature is inventive and not always kind. As with theft, your insurance company can buy you a new computer, but it can't bring back your data.
Other disasters: Sometimes Mother Nature isn't to blame: planes crash into buildings; gas pipes leak and cause explosions; and sometimes building-maintenance people use disk-drive cabinets as temporary saw horses (really!). We even know of one instance in which EPA inspectors came into a building and found asbestos in the A/C ducts, so they forced everyone to leave within 10 minutes, and then sealed the building for several months!
Archival information: Backups provide archival information that lets you compare current versions of software and databases with older ones. This capability lets you determine what you've changed - intentionally or by accident. It also provides an invaluable resource if you ever need to go back and reconstruct the history of a project, either as an academic exercise, or to provide evidence in a court case.

7.1.2 What Should You Back Up?

There are two schools of thought concerning computer-backup systems:

Back up everything that is unique to your system, including all user files, any system databases that you might have modified (such as /etc/passwd and /etc/tty) , and important system directories (such as /bin and /usr/bin) that are especially important or that you may have modified.
Back up everything, because restoring a complete system is easier than restoring an incomplete one, and tape is cheap.

We recommend the second school of thought. While some of the information you back up is already "backed up" on the original distribution disks or tape you used to load them onto your hard disk, distribution disks or tapes sometimes get lost. Furthermore, as your system ages, programs get installed in reserved directories such as /bin and /usr/bin , security holes get discovered and patched, and other changes occur. If you've ever tried to restore your system after a disaster,[4] you know how much easier the process is when everything is in the same place.

[4] Imagine having to reapply 75 vendor "jumbo patches" by hand, plus all the little security patches you got off the net and derived from this book, plus all the tweaks to optimize performance - for each system you manage. Ouch!

For this reason, we recommend that you store everything from your system (and that means everything necessary to reinstall the system from scratch - every last file) onto backup media at regular, predefined intervals. How often you do this depends on the speed of your backup equipment and the amount of storage space allocated for backups. You might want to do a total backup once a week, or you might want to do it only twice a year.

But please do it!

7.1.3 Types of Backups

There are three basic types of backups:

A day-zero backup

Makes a copy of your original system. When your system is first installed, before people have started to use it, back up every file and program on the system. Such backups can be invaluable after a break-in.[5]

[5] We recommend that you also do such a backup immediately after you restore your system after recovering from a break-in. Even if you have left a hole open and the intruder returns, you'll save a lot of time if you are able to fix the hole in the backup, rather than starting from scratch again.
A full backup

Makes a copy of every file on your computer to the backup device. This method is similar to a day-zero backup, except that you do it on a regular basis.
An incremental backup

Makes a copy to the backup device of only those items in a filesystem that have been modified after a particular event (such as application of a vendor patch) or date (such as the date of the last full backup).

Full backups and incremental backups work together. One common backup strategy is:

Make a full backup on the first day of every other week.
Make an incremental backup every evening of everything that has been modified since the last full backup.

Most UNIX administrators plan and store their backups by partition. Different partitions usually require different backup strategies. Some partitions, like the root filesystem and the /etc filesystem (if it is separate), should probably be backed up whenever you make a change to them, on the theory that every change that you make to them is too important to lose. You should use full backups with these systems, rather than incremental backups, because they are only usable in their entirety.

On the other hand, partitions that are used for keeping user files are more amenable to incremental backups. Partitions that are used solely for storing application programs really only need to be backed up when new programs are installed or when the configuration of existing programs are changed.

When you make incremental backups, use a rotating set of backup tapes.[6] The backup you do tonight shouldn't write over the tape you used for your backup last night. Otherwise, if your computer crashes in the middle of tonight's backup, you would lose the data on the disk, the data in tonight's backup (because it is incomplete), and the data in last night's backup (because you partially overwrote it with tonight's backup). Ideally, perform an incremental backup once a night, and have a different tape for every night of the week, as shown in

[6] Yes, all tapes rotate. We mean that the tapes are rotated with each other according to a schedule, rather than being rotated around a spindle.

Figure 7.1: An incremental backup

7.1.4 Guarding Against Media Failure

You can use two distinct sets of backup tapes to create a tandem backup . With this backup strategy, you create two complete backups (call them A and B) on successive backup occasions. Then, when you perform your first incremental backup, the A incremental, you back up all of the files that were created or modified after the original A backup (even if they are on the B full backup tape). The second time you perform an incremental backup, your B incremental, you write out all of the files that were created or modified since the B backup (even if they are on the A incremental backup.) This system protects you against media failure, because every file is backed up in two locations. It does, however, double the amount of time that you will spend performing backups.

Some kinds of tapes - in particular, 4mm or 8mm video tape and Digital Audio Tape ( DAT ) - cannot be reused repeatedly without degrading the quality of the backup. If you use the same tape cartridge for more than a fixed number of backups (usually, 50 or 100), you should get a new one. Be certain to see what the vendor recommends - and don't push that limit. The few pennies you may save by using a tape beyond its useful range will not offset the cost of a major loss.

Try to restore a few files chosen at random from your backups each time, to make sure that your equipment and software are functioning properly. Stories abound about computer centers that have lost disk drives and gone to their backup tapes, only to find them all unreadable. This scenario can occur as a result of bad tapes, improper backup procedures, faulty software, operator error (see the sidebar below), or other problems.

Sometimes, the weakest link in the backup chain is the human responsible for making the backup. Even when everything is automated and requires little thought, things can go badly awry. The following was presented to one of us as a true story. The names and agency have been omitted for obvious reasons.

It seems that a government agency had hired a new night operator to do the backups of the UNIX systems. The operator indicated that she had prior computer operations experience. Even if she hadn't, that was okay - little was needed in this job because the backup was largely the result of an automated script. All the operator had to do was log in at the terminal in the machine room located next to the tape cabinet, start up a command script, and follow the directions. The large disk array would then be backed up with the correct options.

All went fine for several months, until one morning, the system administrator met the operator leaving. She was asked how the job was going. "Fine," she replied. Then the system administrator asked if she needed some extra tapes to go with the tapes she was using every night - he noticed that the disks were getting nearer to full capacity as they approached the end of the fiscal year. He was met by a blank stare and the chilling reply "What tapes?"

Further investigation revealed that the operator didn't know she was responsible for selecting tapes from the cabinet and mounting them. When she started the command file (using UNIX dump), it would pause while mapping the sectors on disk that it needed to write to tape. She would wait a few minutes, see no message, and assume that the backup was proceeding. She would then retire to the lounge to read.

Meanwhile, the tape program would, after some time, begin prompting the operator to mount a tape and press the return key. No tape was forthcoming, however, and the mandatory security software installed on the system logged out the terminal and cleared the screen after 60 minutes of no typing. The operator would come back some hours later and see no error messages of any kind.

Murphy was kind: the system did not crash in the next few hours that it took the panicked system administrator to make a complete level zero dump of the system. Procedures were changed, and the operator was given more complete training.

How do you know if the people doing your backups are doing them correctly?

At least once a year, you should attempt to restore your entire system completely from backups to ensure that your entire backup system is working properly. Starting with a different, unconfigured computer, see if you can restore all of your tapes and get the new computer operational. Sometimes you will discover that some critical file is missing from your backup tapes. These practice trials are the best times to discover a problem and fix it.

It's possible that your computer vendor may let you borrow or rent a computer of the appropriate configuration to let you perform this test. The whole process should take only a few hours, but it will do wonders for your peace of mind and will verify that your backup procedure is working correctly (or illustrate any problems, if it isn't). If you have business continuation insurance, you might even get a break on your premiums by doing this on a regular basis!

A related exercise that can prove valuable is to pick a file at random, once a week or once a month, and try to restore it. Not only will this reveal if the backups are comprehensive, but the exercise of doing the restoration may also provide some insight.

NOTE: We have heard many stories about how the tape drive used to make the backup tapes had a speed or alignment problem. Such a problem results in the tapes being readable by the drive that made them, but unreadable on every other tape drive in the world! Be sure that you load your tapes on other drives when you check them.

7.1.5 How Long Should You Keep a Backup?

It may take a week or a month to realize that a file has been deleted. Therefore, you should keep some backup tapes for a week, some for a month, and some for several months.

Many organizations make yearly backups that they archive indefinitely. After all, tape or CD-ROM is cheap, and rm is forever. Keeping a yearly or a biannual backup "forever" is a small investment in the event that it should ever be needed again.

You may wish to keep on your system an index or listing of the names of files on your backup tapes. This way, if you ever need to restore a file, you can find the right tape to use by scanning the index, rather than reading in every single tape. Having a printed copy of these indexes is also a good idea, especially if you keep the online index on a system that may need to be restored!

NOTE: If you keep your backups for a long period of time, you should be sure to migrate the data on your backups each time you purchase a new backup system. Otherwise, you might find yourself stuck with a lot of tapes that can't be read by anyone, anywhere. This happened in the late 1980s to the MIT Artificial Intelligence Laboratory, which had a collection of research reports and projects from the 1970s on seven-track tape. One day, the lab started a project to put all of the old work online once more. The only problem was that there didn't appear to be a working seven-track tape drive anywhere in the country that the lab could use to restore the data.

7.1.6 Security for Backups

Backups pose a double problem for computer security. On the one hand, your backup tape is your safety net: ideally, it should be kept far away from your computer system so that a local disaster cannot ruin both. But on the other hand, the backup contains a complete copy of every file on your system, so the backup itself must be carefully protected.

7.1.6.1 Physical security for backups

If you use tape drives to make backups, be sure to take the tape out of the drive! One company in San Francisco that made backups every day never bothered removing the cartridge tape from their drive: when their computer was stolen over a long weekend by professional thieves who went through a false ceiling in their office building, they lost everything. "The lesson is that the removable storage media is much safer when you remove it from the drive," said an employee after the incident.

Do not store your backup tapes in the same room as your computer system! Any disaster that might damage or destroy your computers is likely to damage or destroy anything in the immediate vicinity of those computers as well. This rule applies to fire, flood, explosion, and building collapse.

You may wish to consider investment in a fireproof safe to protect your backup tapes. However, the safe should be placed off site , rather than right next to your computer system. While fireproof safes do protect against fire and theft, they don't protect your data against explosion, many kinds of water damage, and building collapse.

NOTE: Be certain that any safe you use for storing backups is actually designed for storing your form of media. One of the fireproof lockboxes from the neighborhood discount store might not be magnetically safe for your tapes. It might be heat-resistant enough for storing paper, but not for storing magnetic tape, which cannot withstand the same high temperatures. Also, some of the generic fire-resistant boxes for paper are designed with a liquid in the walls that evaporates or foams when exposed to heat, to help protect paper inside. Unfortunately, these chemicals can damage the plastic in magnetic tape or CD-ROM s.

7.1.6.2 Write-protect your backups

After you have removed a backup tape from a drive, do yourself a favor and flip the write-protect switch. A write-protected tape cannot be accidentally erased.

If you are using the tape for incremental backups, you can flip the write-protect switch when you remove the tape, and then flip it again when you reinsert the tape later. If you forget to unprotect the tape, your software will probably give you an error and let you try again. On the other hand, having the tape write-protected will save your data if you accidentally put the wrong tape in the tape drive, or run a program on the wrong tape.

7.1.6.3 Data security for backups

File protections and passwords protect the information stored on your computer's hard disk, but anybody who has your backup tapes can restore your files (and read the information contained in them) on another computer. For this reason, keep your backup tapes under lock and key.

Several years ago, an employee at a computer magazine pocketed a 4mm cartridge backup tape that was on the desk of the system manager. When the employee got the tape home, he discovered that it contained hundreds of megabytes of personal files, articles in progress, customer and advertising lists, contracts, and detailed business plans for a new venture that the magazine's parent company was planning. The tape also included tens of thousands of dollars worth of computer application programs, many of which were branded with the magazine's name and license numbers. Quite a find for an insider who is setting up a competing publication.

When you transfer your backup tapes from your computer to the backup location, protect the tapes at least as well as you normally protect the computers themselves. Letting a messenger carry the tapes from building to building may not be appropriate if the material on the tapes is sensitive. Getting information from a tape by bribing an underpaid courier, or by knocking him unconscious and stealing it, is usually easier and cheaper than breaching a firewall, cracking some passwords, and avoiding detection online.

One apocryphal story we saw on the net several years ago concerned a system administrator who tried to do the right thing. Every week, he'd make a full backup of his system. He would test the backups to be certain the tapes were readable. Then he would throw them into the passenger seat of his car and he'd drive home, where he would store the tapes in a fireproof safe.

All proceeded without incident for many months until there was a major disaster of some sort that destroyed the computer room. Our would-be hero felt quite relieved that he had these valuable backups in a safe place. So, after an alternate site was identified, he drove home and retrieved some of the tapes.

Unfortunately, the tapes contained no usable data. All the data was lost.

The problem? It seems that these events occurred in the far north, during the winter. The administrator's car had heater wires in the car seats, and the current passing through the wires degaussed the data on the tapes that he stored on the passenger seat during the long ride home.

Is it true? Possibly. But even if not, the lessons are sound ones: test your backup tapes after subjecting them to whatever treatment you would expect for a tape you would need to use, and exercise great care in how you store and transport the tapes

The use of encryption can dramatically improve the security for backup tapes. However, if you do choose to encrypt your backup tapes, be sure that the encryption key is known by more than one person. You may wish to escrow your key (see the sidebar entitled "A Note About Key Escrow" in Chapter 6, Cryptography ). Otherwise, the backups may be worthless if the only person with the key forgets it, becomes incapacitated, or decides to hold your data for ransom.

Here are some ideas for storing a backup tape's encryption key:

Always use the same key. Physical security of your backup tape should be your first line of defense.
Store copies of the key on pieces of paper in envelopes. Give the envelopes to each member of the organization's board of directors, or chief officers.
If your organization uses an encryption system such as PGP that allows a message to be encrypted for multiple recipients, encrypt and distribute the backup encryption key so that it can be decrypted by anyone on the board.
Alternatively, you might consider a secret-sharing system, so that the key can be decrypted by any two or three board members working together, but not by any board member on his own.

7.1.7 Legal Issues

Finally, some firms should be careful about backing up too much information, or holding it for too long. Recently, backup tapes have become targets in lawsuits and criminal investigations. Backup tapes can be obtained by subpoena or during discovery in lawsuits. If your organization has a policy regarding the destruction of old paper files, you should extend this policy to backup tapes as well.

You may wish to segregate potentially sensitive data so that it is stored on separate backup tapes. For example, you can store applications on one tape, pending cases on another tape, and library files and archives on a third.

Back up your data, but back up with caution.