Those who forget the past are condemned to fulfill it.
George Santayana, Life of Reason
Those who do not archive the past are condemned to retype it!
Garfinkel and Spafford, Practical UNIX Security (first edition)
When we were working on the first edition of Practical UNIX Security , we tried in vain to locate a source of the quotation "those who forget the past are condemned to repeat it." But no matter where we searched, we couldn't find the source. What we found instead was the reference above in George Santayana's book, Life of Reason . So we printed it and added our own variation.
What's the point? Few people take the time to verify their facts, be they the words of Santayana's oft-misquoted statement, or the contents of an unlabeled backup tape. In the years since Practical UNIX Security was first published, we have heard of countless instances in which people or whole organizations have had their files lost to computer failure or vandalism. Often, the victims are unable to restore their systems from full and compete backups. Instead, restoration is often a piecemeal and lengthy project - a few files from this tape, a few files from that one, and a few from the original CD-ROM distribution.
Even if backup tapes exist, there are still problems. In one case, a researcher at Digital Equipment Corporation lost a decade's worth of personal email because of a bad block at the beginning of a 2GB DAT tape. The contents of the tape had never been verified.
In another case, we know of a project group that had to manually recreate a system from printouts (that is, they had to retype the entire system) because the locally written backup program was faulty. Although the staff had tested the program, they had only tested the program with small, random files - and the program's bug was that it only backed up the first 1024 bytes of each real file. Unfortunately, this fault was not discovered until after the program had been in use several months.
Making backups and verifying them may be the most important things that you can do to protect your data - other than reading this book, of course!
Bugs, accidents, natural disasters, and attacks on your system cannot be predicted. Often, despite your best efforts, they can't be prevented. But if you have backups, you can compare your current system and your backed-up system, and you can restore your system to a stable state. Even if you lose your entire computer - to fire, for instance - with a good set of backups you can restore the information after you have purchased or borrowed a replacement machine. Insurance can cover the cost of a new CPU and disk drive, but your data is something that in many cases can never be replaced.
Russell Brand writes: "To me, the user data is of paramount importance. Anything else is generally replaceable. You can buy more disk drives, more computers, more electrical power. If you lose the data, through a security incident or otherwise, it is gone."
Mr. Brand made this comment in a paper on UNIX security several years ago; we think it summarizes the situation well, within limits. Backups are one of the most critical aspects of your system operation. Having backups that are valid, complete, and up to date may make the difference between a minor incident and a catastrophe.
Backups are important only if you value the work that you do on your computer. If you use your computer as a paperweight, then you don't need to make backups.
You also don't need to back up a computer that only uses read-only storage, such as a CD-ROM . A variant of this is Sun's "dataless" client workstations, in which the operating system is installed from a CD-ROM and never modified. When configured this way, the computer's local hard disk is used as an accelerator and a cache, but it is not used to store data you would want to archive. Sun specifically designs its operating system for these machines so that they do not need backups.
On the other hand, if you ever turn your computer on and occasionally modify data on it, then you must make a copy of that information if you want to recover it in the event of a loss.
Today, hardware failure is still a good reason to back up your system. In 1990, many hard-disk companies gave their drives two- or three-year guarantees; many of those drives are failing now. Even though today's state-of-the-art hard disk drives might come with five-year warranties, they too will fail one day!
Such a failure might not be years away, either. In the fall of 1993, one of the authors bought a new 1.7GB hard drive to replace a 1.0 GB unit. The files were copied from the older drive to the newer one, and then the older unit was reformatted and given to a colleague. The next week, the 1.7GB unit failed. Luckily, there was a backup.
Backups are important for a number of other reasons as well:
There are two schools of thought concerning computer-backup systems:
We recommend the second school of thought. While some of the information you back up is already "backed up" on the original distribution disks or tape you used to load them onto your hard disk, distribution disks or tapes sometimes get lost. Furthermore, as your system ages, programs get installed in reserved directories such as /bin and /usr/bin , security holes get discovered and patched, and other changes occur. If you've ever tried to restore your system after a disaster, you know how much easier the process is when everything is in the same place.
For this reason, we recommend that you store everything from your system (and that means everything necessary to reinstall the system from scratch - every last file) onto backup media at regular, predefined intervals. How often you do this depends on the speed of your backup equipment and the amount of storage space allocated for backups. You might want to do a total backup once a week, or you might want to do it only twice a year.
But please do it!
There are three basic types of backups:
Full backups and incremental backups work together. One common backup strategy is:
Most UNIX administrators plan and store their backups by partition. Different partitions usually require different backup strategies. Some partitions, like the root filesystem and the /etc filesystem (if it is separate), should probably be backed up whenever you make a change to them, on the theory that every change that you make to them is too important to lose. You should use full backups with these systems, rather than incremental backups, because they are only usable in their entirety.
On the other hand, partitions that are used for keeping user files are more amenable to incremental backups. Partitions that are used solely for storing application programs really only need to be backed up when new programs are installed or when the configuration of existing programs are changed.
When you make incremental backups, use a rotating set of backup tapes. The backup you do tonight shouldn't write over the tape you used for your backup last night. Otherwise, if your computer crashes in the middle of tonight's backup, you would lose the data on the disk, the data in tonight's backup (because it is incomplete), and the data in last night's backup (because you partially overwrote it with tonight's backup). Ideally, perform an incremental backup once a night, and have a different tape for every night of the week, as shown in
You can use two distinct sets of backup tapes to create a tandem backup . With this backup strategy, you create two complete backups (call them A and B) on successive backup occasions. Then, when you perform your first incremental backup, the A incremental, you back up all of the files that were created or modified after the original A backup (even if they are on the B full backup tape). The second time you perform an incremental backup, your B incremental, you write out all of the files that were created or modified since the B backup (even if they are on the A incremental backup.) This system protects you against media failure, because every file is backed up in two locations. It does, however, double the amount of time that you will spend performing backups.
Some kinds of tapes - in particular, 4mm or 8mm video tape and Digital Audio Tape ( DAT ) - cannot be reused repeatedly without degrading the quality of the backup. If you use the same tape cartridge for more than a fixed number of backups (usually, 50 or 100), you should get a new one. Be certain to see what the vendor recommends - and don't push that limit. The few pennies you may save by using a tape beyond its useful range will not offset the cost of a major loss.
Try to restore a few files chosen at random from your backups each time, to make sure that your equipment and software are functioning properly. Stories abound about computer centers that have lost disk drives and gone to their backup tapes, only to find them all unreadable. This scenario can occur as a result of bad tapes, improper backup procedures, faulty software, operator error (see the sidebar below), or other problems.
At least once a year, you should attempt to restore your entire system completely from backups to ensure that your entire backup system is working properly. Starting with a different, unconfigured computer, see if you can restore all of your tapes and get the new computer operational. Sometimes you will discover that some critical file is missing from your backup tapes. These practice trials are the best times to discover a problem and fix it.
It's possible that your computer vendor may let you borrow or rent a computer of the appropriate configuration to let you perform this test. The whole process should take only a few hours, but it will do wonders for your peace of mind and will verify that your backup procedure is working correctly (or illustrate any problems, if it isn't). If you have business continuation insurance, you might even get a break on your premiums by doing this on a regular basis!
A related exercise that can prove valuable is to pick a file at random, once a week or once a month, and try to restore it. Not only will this reveal if the backups are comprehensive, but the exercise of doing the restoration may also provide some insight.
Many organizations make yearly backups that they archive indefinitely. After all, tape or CD-ROM is cheap, and rm is forever. Keeping a yearly or a biannual backup "forever" is a small investment in the event that it should ever be needed again.
You may wish to keep on your system an index or listing of the names of files on your backup tapes. This way, if you ever need to restore a file, you can find the right tape to use by scanning the index, rather than reading in every single tape. Having a printed copy of these indexes is also a good idea, especially if you keep the online index on a system that may need to be restored!
Backups pose a double problem for computer security. On the one hand, your backup tape is your safety net: ideally, it should be kept far away from your computer system so that a local disaster cannot ruin both. But on the other hand, the backup contains a complete copy of every file on your system, so the backup itself must be carefully protected.
If you use tape drives to make backups, be sure to take the tape out of the drive! One company in San Francisco that made backups every day never bothered removing the cartridge tape from their drive: when their computer was stolen over a long weekend by professional thieves who went through a false ceiling in their office building, they lost everything. "The lesson is that the removable storage media is much safer when you remove it from the drive," said an employee after the incident.
Do not store your backup tapes in the same room as your computer system! Any disaster that might damage or destroy your computers is likely to damage or destroy anything in the immediate vicinity of those computers as well. This rule applies to fire, flood, explosion, and building collapse.
You may wish to consider investment in a fireproof safe to protect your backup tapes. However, the safe should be placed off site , rather than right next to your computer system. While fireproof safes do protect against fire and theft, they don't protect your data against explosion, many kinds of water damage, and building collapse.
If you are using the tape for incremental backups, you can flip the write-protect switch when you remove the tape, and then flip it again when you reinsert the tape later. If you forget to unprotect the tape, your software will probably give you an error and let you try again. On the other hand, having the tape write-protected will save your data if you accidentally put the wrong tape in the tape drive, or run a program on the wrong tape.
File protections and passwords protect the information stored on your computer's hard disk, but anybody who has your backup tapes can restore your files (and read the information contained in them) on another computer. For this reason, keep your backup tapes under lock and key.
Several years ago, an employee at a computer magazine pocketed a 4mm cartridge backup tape that was on the desk of the system manager. When the employee got the tape home, he discovered that it contained hundreds of megabytes of personal files, articles in progress, customer and advertising lists, contracts, and detailed business plans for a new venture that the magazine's parent company was planning. The tape also included tens of thousands of dollars worth of computer application programs, many of which were branded with the magazine's name and license numbers. Quite a find for an insider who is setting up a competing publication.
When you transfer your backup tapes from your computer to the backup location, protect the tapes at least as well as you normally protect the computers themselves. Letting a messenger carry the tapes from building to building may not be appropriate if the material on the tapes is sensitive. Getting information from a tape by bribing an underpaid courier, or by knocking him unconscious and stealing it, is usually easier and cheaper than breaching a firewall, cracking some passwords, and avoiding detection online.
The use of encryption can dramatically improve the security for backup tapes. However, if you do choose to encrypt your backup tapes, be sure that the encryption key is known by more than one person. You may wish to escrow your key (see the sidebar entitled "A Note About Key Escrow" in Chapter 6, Cryptography ). Otherwise, the backups may be worthless if the only person with the key forgets it, becomes incapacitated, or decides to hold your data for ransom.
Here are some ideas for storing a backup tape's encryption key:
Finally, some firms should be careful about backing up too much information, or holding it for too long. Recently, backup tapes have become targets in lawsuits and criminal investigations. Backup tapes can be obtained by subpoena or during discovery in lawsuits. If your organization has a policy regarding the destruction of old paper files, you should extend this policy to backup tapes as well.
You may wish to segregate potentially sensitive data so that it is stored on separate backup tapes. For example, you can store applications on one tape, pending cases on another tape, and library files and archives on a third.
Back up your data, but back up with caution.