Which Unix System?

An unfortunate side effect of Unix's popularity is that there are many different versions of Unix; today, nearly every computer manufacturer has its own. When we wrote the first edition of this book, there were two main families of Unix: AT&T System V and Berkeley's BSD. There was a sharp division between these systems. System V was largely favored by industry and government because of its status as a well-supported, "official" version of Unix. BSD, meanwhile, was largely favored by academic sites and developers because of its flexibility, scope, and additional features.

When we wrote the first edition of this book, only Unix operating systems sold by AT&T could be called "Unix" because of licensing restrictions. Other manufacturers adopted names such as SunOS (Sun Microsystems), Solaris (also Sun Microsystems), Xenix (Microsoft), HP-UX (Hewlett-Packard), A/UX (Apple), Dynix (Sequent), OSF/1 (Open Software Foundation), Linux (Linus Torvalds), Ultrix (Digital Equipment Corporation), and AIX (IBM)—to name a few. Practically every supplier of a Unix or Unix-like operating system made its own changes to the operating system. Some of these changes were small, while others were significant. Some of these changes had dramatic security implications and, unfortunately, many of these implications are usually not evident. Not every vendor considers the security implications of its changes before making them.

In recent years, Unix has undergone a rapid evolution. Most of the commercial versions of the operating system have died off, while there has simultaneously been an explosion of "free" Unix systems. Security has grown more important in recent years, and now all companies, organizations, and individuals distributing Unix claim to take the subject of security quite seriously. However, it is clear that some take the subject far more seriously than others.

Versions Covered in This Book

The third edition of this book covers Unix security as it relates to the four most common versions of Unix today: Solaris, Linux, FreeBSD, and MacOS X. Solaris and Linux are generally thought of as System V-based operating systems, while FreeBSD and MacOS X are generally seen as BSD-based systems. However, there has been so much mingling of concepts and code in recent years that these distinctions may no longer be relevant. In many cases, the underlying theory and commands on these systems are similar enough that we can simply use the word "Unix" to stand for all of these systems. In cases where we cannot, we note individual operating system differences.

Particular details in this book concerning specific Unix commands, options, and side effects are based upon the authors' experience with AT&T System V Release 3.2 and 4.0, Berkeley Unix Release 4.3 and 4.4, Digital Unix, FreeBSD 3.0 through 4.5, Linux (various versions), MacOS X, NeXTSTEP 0.9 through 4.0, Solaris 2.3 through 8, SunOS 4.0 and 4.1, and Ultrix 4.0. We've also had the benefit of our technical reviewers' long experience with other systems, such as AIX and HP-UX. As these systems are representative of the majority of Unix machines in use, it is likely that these descriptions will suffice for most machines to which readers will have access.

Throughout this book, we generally refer to System V Release 4 as SVR4. When we refer to SunOS without a version number, assume that we are referring to SunOS 4.1.x. When we refer to Solaris without a version number, assume that we are referring to Solaris 7 and above.

We also refer to operating systems that run on top of the Linux kernel as Linux, even though many Linux systems contain significant components that were developed by readily identifiable third parties. (For example, the Free Software Foundation was responsible for the creation of the GNU development tools, without which the Linux system could not have been built, while MIT and the X Windows Consortium were responsible for the creation and initial development of the X Window system.)

Many Unix vendors have modified the basic behavior of some of their system commands, and there are dozens upon dozens of Unix vendors. As a result, we don't attempt to describe every specific feature offered in every version issued by every manufacturer—that would only make the book longer, as well as more difficult to read. It would also make this book inaccurate, as some vendors change their systems frequently. Furthermore, we are reluctant to describe special-case features on systems we have not been able to test thoroughly ourselves. Whether you're a system administrator or an ordinary user, it's vital that you read the reference pages of your own particular Unix system to understand the differences between what is presented in this volume and the actual syntax of the commands that you're using. This is especially true in situations in which you depend upon the specific output or behavior of a program to verify or enhance the security of your system.

The Many Faces of "Open Source" Unix

One of the difficulties in writing this book is that there are many, many versions of Unix. All of them have differences: some minor, some significant. Our problem, as you shall see, is that even apparently minor differences between two operating systems can lead to dramatic differences in overall security. Simply changing the protection settings on a single file can turn a secure operating system into an unsecure one.

The Linux operating system makes things even more complicated. That's because Linux is a moving target. There are many different distributions of Linux. Some have minor differences, such as the installation of a patch or two. Others are drastically different, with different kernels, different driver software, and radically different security models.

Furthermore, Linux is not the only free form of Unix. After the release of Berkeley 4.3, the Berkeley Computer Systems Research Group (CSRG) (and a team of volunteers across the Internet) worked to develop a system that was devoid of all AT&T code; this release was known as Berkeley 4.4. Somewhere along the line the project split into several factions, eventually producing four operating systems: BSD 4.4 Lite, NetBSD, FreeBSD, and OpenBSD. Today there are several versions of each of these operating systems. There are also systems based on the Mach kernel and systems that employ Unix-like utilities from a number of sources. (Chapter 2 covers this history.)

The world of free Unix is less of a maelstrom today than it was when the second edition of this book was published. However, it remains true that if you want to run Linux, NetBSD, FreeBSD, or any other such system securely, it is vitally important that you know exactly which version of which distribution of which operating system with which software you are running on your computer. Merely reading your manual may not be enough! You may have to read the source code. You may also have to verify that the source code you are reading actually compiles to produce the binaries you are running!

Also, please note that we cannot possibly describe (or even know) all the possible variations and implications, so don't assume that we have covered all the nuances of your particular system. When in doubt, check it out.

By writing this book, we hope to provide information that will help users and system administrators improve the security of their systems. We have tried to ensure the accuracy and completeness of everything within this book. However, as we noted previously, we can't be sure that we have covered everything, and we can't know about all the quirks and modifications made to every version and installation of Unix-derived systems. Thus, we can't promise that your system security will never be compromised if you follow all our advice, but we can promise that successful attacks will be less likely. We encourage readers to tell us of significant differences between their own experiences and the examples presented in this book; those differences may be noted in future editions.

"Secure" Versions of Unix

Over time, several vendors have developed "secure" versions of Unix, sometimes known as "trusted Unix." These systems embody mechanisms, enhancements, and restraints described in various government standards documents. These enhanced versions of Unix are designed to work in Multilevel Security (MLS) and Compartmented-Mode Workstation (CMW) environments—where there are severe constraints designed to prevent the mixing of data and code with different security classifications, such as Secret and Top Secret. In 2001, Chris I. Dalton and Tse Huong Choo at HP Labs released a system called Trusted Linux. The National Security Agency has also released a Linux variant called Security Enhanced Linux (SE Linux).^[6]

^[6] Security Enhanced Linux is a misleading name, however, as the release does not address all of the underlying architectural and implementation flaws. Instead, SE Linux adds a form of mandatory access control to a vanilla Linux. Assuming that there are no major bugs and that you configure it correctly, you can achieve better security—but it doesn't come automatically, nor does it provide a comprehensive security solution.

Secure Unix systems generally have extra features added to them, including access control lists, data labeling, enhanced auditing, and mutual authentication between separate components. They also remove some traditional features of Unix, such as the superuser's special access privileges and access to some device files. Despite these changes, the systems still bear a resemblance to standard Unix. Trusted Solaris still functions basically like Solaris.

These systems are not in widespread use outside of selected government agencies, their contractors, and the financial industry. It seems doubtful to us that they will ever enjoy widely popular acceptance because many of the features make sense only within the context of a military security policy. On the other hand, some of these enhancements are useful in the commercial environment as well, and C2 security features are already common in many modern versions of Unix.

Today, trusted Unix systems are often more difficult to use in a wide variety of environments, more difficult to port programs to, and more expensive to obtain and maintain. Thus, we haven't bothered to describe the quirks and special features of these systems in this book. If you have such a system, we recommend that you read the vendor documentation carefully and repeatedly.