Startup Files (TCP/IP Network Administration, 3rd Edition)

5.2.1. Startup Runlevels

To understand System V startup, you need to understand runlevels, which are used to indicate the state of the system when the init process is complete. There is nothing inherent in the system hardware that recognizes runlevels; they are purely a software construct. init and /etc/inittab -- the file used to configure init -- are the only reasons why the runlevels affect the state of the system. We use Red Hat Linux as an example of how runlevels are used.

Linux defines several runlevels that run the full gamut of possible system states from not-running (halted) to running multiple processes for multiple users:

Runlevel 0 shuts down all running processes and halts the system.
Runlevel 1 places the system in single-user mode. Single-user mode is used by the system administrator to perform maintenance that cannot be done when users are logged in. This runlevel may also be indicated by the letter S instead of the number 1. Solaris uses S for single-user mode.
Runlevel 2 is a special multiuser mode that does not support file sharing.
Runlevel 3 provides full multiuser support with the full range of services, including NFS file sharing. It is the default mode used on Solaris systems.
Runlevel 4 is unused. You can design your own system state and implement it through runlevel 4.
Runlevel 5 initializes the system as a dedicated X Windows terminal. Linux systems use this to provide an X Windows console login. When Linux systems boot at runlevel 3, they provide a text-based console login. Solaris does not use this runlevel. Entering runlevel 5 on a Solaris system causes a system shutdown.
Runlevel 6 shuts down all running processes and reboots the system.

As these notes make clear, different systems use the same runlevels in different ways. That is because runlevels are software. They are boot command arguments that tell init which startup scripts should be run. The scripts that are run can contain any valid commands. init maps runlevels to startup scripts using the inittab file.

5.2.1.1. Understanding /etc/inittab

All of the lines in the inittab file that begin with a sharp sign (#) are comments. A liberal dose of comments is needed because the syntax of inittab configuration lines is terse and arcane. An inittab entry has this general format:

label:runlevel:action:process

The label is a one- to four-character tag that identifies the entry. Because some systems support only two-character labels, most configurations limit all labels to two characters. The labels can be any arbitrary character string; they have no intrinsic meaning.

The runlevel field indicates the runlevels to which the entry applies. For example, if the field contains a 3, the process identified by the entry must be run for the system to initialize runlevel 3. More than one runlevel can be specified. Entries that have an empty runlevel field are not involved in initializing specific runlevels. For example, Linux systems have an inittab entry to handle the three-finger salute (Ctrl+Alt+Del); it does not have a value in the runlevel field.

The action field defines the conditions under which the process is run. Table 5-1 lists the action values used on Red Hat, Mandrake, and Caldera Linux systems.

Table 5-1. Linux inittab action values

Action	Meaning
Boot	Runs when the system boots. Runlevels are ignored.
Bootwait	Runs when the system boots, and init waits for the process to complete. Runlevels are ignored.
Ctrlaltdel	Runs when Ctrl+Alt+Del is pressed, which passes the SIGINT signal to init. Runlevels are ignored.
Initdefault	Doesn't execute a process. It sets the default runlevel.
Kbrequest	Runs when init receives a signal from the keyboard. This requires that a key combination be mapped to KeyBoardSignal.
Off	Disables the entry so the process is not run.
Once	Runs one time for every runlevel.
Ondemand	Runs when the system enters one of the special runlevels A, B, or C.
Powerfail	Runs when init receives the SIGPWR signal.
Powerokwait	Runs when init receives the SIGPWR signal and the file /etc/powerstatus contains the word OK.
Powerwait	Runs when init receives the SIGPWR signal, and init waits for the process to complete.
Respawn	Restarts the process whenever it terminates.
sysinit	Runs before any boot or bootwait processes.
wait	Runs the process upon entering the run mode, and init waits for the process to complete.

The last field in an inittab entry is process. It contains the process that init executes. The process appears in the exact format that it is executed from the command line. Therefore the process field starts with the name of the program that is to be executed followed by the arguments that will be passed to that process. For example, /sbin/shutdown -t3 -r now, which is the process executed by some Linux systems when Ctrl+Alt+Del is pressed, is the same command that could be typed at the shell prompt to reboot the system. On most inittab entries, the process field contains the name of a startup script. Two main types of startup scripts are used: the system initialization script and the runlevel initialization scripts. These sample lines from a Red Hat Linux system show both:

# System initialization.
si::sysinit:/etc/rc.d/rc.sysinit

l0:0:wait:/etc/rc.d/rc 0
l1:1:wait:/etc/rc.d/rc 1
l2:2:wait:/etc/rc.d/rc 2
l3:3:wait:/etc/rc.d/rc 3
l4:4:wait:/etc/rc.d/rc 4
l5:5:wait:/etc/rc.d/rc 5
l6:6:wait:/etc/rc.d/rc 6

These seven lines are the real heart of the inittab file -- they invoke the startup scripts. The first line tells init to run the boot script located at /etc/rc.d/rc.sysinit to initialize the system. This entry has no runlevel value. It is run every time the system starts. The system initialization script performs certain essential tasks. For example, the Red Hat rc.sysinit script:

Initializes the swap space
Runs the filesystem check
Mounts the /proc filesystem
Mounts the root filesystem as read-write after the fsck completes
Loads the loadable kernel modules

Other initialization scripts may look different than Red Hat's, but they perform very similar functions. For example, a Caldera system begins by loading the loadable modules. It then activates the swap space, does the filesystem check, and remounts the root filesystem as read-write. The order is different, but the major functions are the same.

After the system initialization script is run, init runs a script for the specific runlevel. The remaining six lines in the sample are used to invoke the startup scripts for individual runlevels. Except for the runlevel involved, each line is identical.

Let's use the line with label l3 as an example. This line starts all of the processes and services needed to provide the full multiuser support. The runlevel is 3. The action wait directs init to wait until the startup script terminates before going on to any other entries in the inittab file that relate to runlevel 3. init executes the script /etc/rc.d/rc and passes that script the command-line argument 3.

The control script, /etc/rc.d/rc, then runs all the scripts that are appropriate for the runlevel. It does this by running the scripts that are stored in the directory /etc/rcn.d, where n is the specified runlevel. In our example, the rc script is passed a 3, so it runs the scripts found in the directory /etc/rc.d/rc3.d. A listing of that directory from a Red Hat system shows that there are lots of scripts:

$ ls /etc/rc.d
init.d  rc0.d  rc2.d  rc4.d  rc6.d     rc.sysinit
rc      rc1.d  rc3.d  rc5.d  rc.local
$ ls /etc/rc.d/rc3.d
K03rhnsd      K35smb       K74ntpd    S05kudzu     S25netfs       S85httpd
K16rarpd      K45arpwatch  K74ypserv  S06reconfig  S26apmd        S90crond
K20nfs        K45named     K74ypxfrd  S08ipchains  S28autofs      S90xfs
K20rstatd     K50snmpd     K75gated   S09isdn      S40atd         S95anacron
K20rusersd    K50tux       K84bgpd    S10network   S55sshd        S99linuxconf
K20rwalld     K55routed    K84ospf6d  S12syslog    S56rawdevices  S99local
K20rwhod      K61ldap      K84ospfd   S13portmap   S56xinetd
K28amd        K65identd    K84ripd    S14nfslock   S60lpd
K34yppasswdd  K73ypbind    K84ripngd  S17keytable  S80sendmail
K35dhcpd      K74nscd      K85zebra   S20random    S85gpm

The scripts that begin with a K are used to kill processes when exiting a specific runlevel. In the listing above, the K scripts would be used when terminating runlevel 3. The scripts that start with an S are used when starting runlevel 3. None of the items in rc3.d, however, is really a startup script. They are logical links to the real scripts, which are located in the /etc/rc.d/init.d directory. For example, S80sendmail is linked to init.d/sendmail. This raises the question of why the scripts are executed from the directory rc3.d instead of directly from init.d where they actually reside. The reasons are simple. The same scripts are needed for several different runlevels. Using logical links, the scripts can be stored in one place and still be accessed by every runlevel from the directory used by that runlevel.

Scripts are executed in alphabetical order. Thus S10network is executed before S80sendmail. This allows the system to control the order in which scripts are executed through simple naming conventions. Different runlevels can execute the scripts in different orders while still allowing the real scripts in init.d to have simple, descriptive names. A listing of the init.d directory shows these descriptive names:

$ ls /etc/rc.d/init.d
amd       functions  kdcrotate  network  rarpd       rwalld    xfs
anacron   gated      keytable   nfs      rawdevices  rwhod     xinetd
apmd      gpm        killall    nfslock  reconfig    sendmail  ypbind
arpwatch  halt       kudzu      nscd     rhnsd       single    yppasswdd
atd       httpd      ldap       ntpd     ripd        smb       ypserv
autofs    identd     linuxconf  ospf6d   ripngd      snmpd     ypxfrd
bgpd      ipchains   lpd        ospfd    routed      sshd      zebra
crond     iptables   named      portmap  rstatd      syslog
dhcpd     isdn       netfs      random   rusersd     tux

It is possible to place a customized configuration command directly in the applicable script in the init.d directory. A better alternative on a Red Hat system is to place any local changes in rc.local.

Like BSD systems, Linux systems provide an rc.local script for local customization. In general, you do not directly edit boot scripts. The exception to this rule is the rc.local script located in the /etc/rc.d directory. It is the one customizable startup file, and it is reserved for your use; you can put anything you want in there. After the system initialization script executes, the runlevel scripts execute in alphabetical order. The last of these is S99local, which is a link to rc.local. Since it is executed last, the values set in the rc.local script override other configuration values.

Solaris also uses the System V startup model, but it makes things a little more difficult than Linux does. First off, it does not provide an rc.local script. If you want to use one, you need to add your own to the runlevel directories. Secondly, Solaris does not use many logical links in the runlevel directories. Therefore, there is no guarantee of a central place to modify scripts that are used for all runlevels. Additionally, each runlevel has a separate controlling script that can introduce differences in the startup process for each runlevel. For example, /sbin/rc2 is the controlling script for runlevel 2 and /sbin/rc3 is the controlling script for runlevel 3. All of these differences make the Solaris startup process more complex to analyze.

On a Solaris 8 system, runlevel 3 is the default runlevel for a multiuser system offering network services. The /sbin/rc3 controlling script runs the scripts in /etc/rc2.d and then those in /etc/rc3.d. Basic network configuration is handled in /etc/rc2.d by the S69inet script and the S72inetsvc script. Several other scripts in both /etc/rc2.d and /etc/rc3.d are involved in launching network services.

For troubleshooting purposes it is important to understand where and how things happen during the system startup. When the network fails to initialize properly, it is good to know where to look. However, when you configure the network you should stick with the standard tools and procedures provided with your system. Directly modifying startup scripts can cause problems during the startup and can lead to lots of confusion for the other people who help you maintain your systems.

Of course, not all network services are started by a boot script. ost network services are started on demand. The most widely used tool for starting network services on demand is inetd, the Internet Daemon.

5.2. Startup Files

5.2.1. Startup Runlevels

5.2.1.1. Understanding /etc/inittab

Table 5-1. Linux inittab action values