Chapter 7. Input/Output and Command-Line Processing

7.1.1. Here-Documents

The << label redirector essentially forces the input to a command to be the shell program's text, which is read until there is a line that contains only label. The input in between is called a here-document. Here-documents aren't very interesting when used from the command prompt. In fact, it's the same as the normal use of standard input except for the label. We could have used a here-document in the previous example of >>, like this (EOF, for "end of file," is an often-used label):

$ cat >> .mailrc << EOF
> alias fred frederick@longmachinename.longcompanyname.com
> EOF
$

Here-documents are meant to be used from within shell scripts; they let you specify "batch" input to programs. A common use of here-documents is with simple text editors like ed(1). Task 7-1 uses a here-document in this way.

Task 7-1 The s file command in mail(1) saves the current message in file. If the message came over a network (such as the Internet), it has several prepended header lines that give information about network routing. You need this information because you're trying to solve some network routing problems. Write a shell script that extracts just the header lines from the file.

We can use ed to delete the body lines, leaving just the header. To do this, we need to know something about the syntax of mail messages, specifically, that there is always a blank line between the header lines and the message text. The ed command /^$/,$d does the trick: it means, "Delete from the first blank line[91] through the last line of the file." We also need the ed commands w (write the changed file) and q (quit). Here is the code that solves the task:

[91] The line has to be completely empty; no spaces or TABs. That's OK: mail message headers are separated from their bodies by exactly this kind of blank line.

ed $1 << \EOF
/^$/,$d
w
q
EOF

Normally, the shell does parameter (variable) substitution, command substitution, and arithmetic substitution on text in a here-document, meaning that you can use shell variables and commands to customize the text. This evaluation is disabled if any part of the delimiter is quoted, as done in the previous example. (This prevents the shell from treating $d as a variable substitution.)

Often though, you do want the shell to perform its evaluations: perhaps the most common use of here-documents is for providing templates for form generators or program text for program generators. Task 7-2 is a simple task for system administrators that shows how this works.

Task 7-2 Write a script that sends a mail message to a set of users saying that a new version of a certain program has been installed in a certain directory.

You can get a list of all users on the system in various ways; perhaps the easiest is to use cut to extract the first field of /etc/passwd, the file that contains all user account information. Fields in this file are separated by colons (:).[92]

[92] There are a few possible problems with this; for example, /etc/passwd usually contains information on "accounts" that aren't associated with people, like uucp, lp, and daemon. We'll ignore such problems for the purpose of this example.

Given such a list of users, the following code does the trick:

pgmname=$1
for user in $(cut -f1 -d: /etc/passwd); do
    mail $user << EOF
Dear $user,

A new version of $pgmname has been installed in $(whence pgmname).

Regards,
Your friendly neighborhood sysadmin.
EOF
done

The shell substitutes the appropriate values for the name of the program and its directory.

The redirector << has two variations. First, you can prevent the shell from doing parameter, command and arithmetic substitution by surrounding the label in single or double quotes. (Actually, it's enough to quote just one character in the label.) We saw this in the solution to Task 7-1.

The second variation is <<-, which deletes leading TABs (but not spaces) from the here-document and the label line. This allows you to indent the here-document's text, making the shell script more readable:

pgmname=$1
for user in $(cut -f1 -d: /etc/passwd); do
    mail $user <<- EOF
        Dear $user,

        A new version of $pgmname has been installed in $(whence pgmname).

        Regards,

        Your friendly neighborhood sysadmin.
        EOF
done

Of course, you need to choose your label so that it doesn't appear as an actual input line.

7.1.2. Here-Strings

A common idiom in shell programming is to use print to generate some text to be further processed by one or more commands:

# start with a mild interrogation
print -r "$name, $rank, $serial_num" | interrogate -i mild

This could be rewritten to use a here-document, which is slightly more efficient, although not necessarily any easier to read:

# start with a mild interrogation
interrogate -i mild << EOF
$name, $rank, $serial_num
EOF

Starting with ksh93n,[93] the Korn shell provides a new form of here-document, using three less-than signs:

[93] Thanks to David Korn for providing me prerelease access to the version with this feature. ADR.

program <<< WORD

In this form, the text of WORD (followed by a trailing newline) becomes the input to the program. For example:

# start with a mild interrogation
interrogate -i mild <<< "$name, $rank, $serial_num"

This notation first originated in the Unix version of the rc shell, where it is called a "here string." It was later picked up by the Z shell, zsh (see Appendix A), from which the Korn shell borrowed it. This notation is simple, easy to use, efficient, and visually distinguishable from regular here-documents.

7.1.3. File Descriptors

The next few redirectors in Table 7-1 depend on the notion of a file descriptor. This is a low-level Unix I/O concept that is vital to understand when programming in C or C++. It appears at the shell level when you want to do anything that doesn't involve standard input, standard output and standard error. You can get by with a few basic facts about them; for the whole story, look at the open(2), creat(2), read(2), write(2), dup(2), dup2(2), fcntl(2), and close(2) entries in the Unix manual. (As the manual entries are aimed at the C programmer, their relationship to the shell concepts won't necessarily be obvious.)

File descriptors are integers starting at 0 that index an array of file information within a process. When a process starts, it has three file descriptors open. These correspond to the three standards: standard input (file descriptor 0), standard output (1), and standard error (2). If a process opens Unix files for input or output, they are assigned to the next available file descriptors, starting with 3.

By far the most common use of file descriptors with the Korn shell is in saving standard error in a file. For example, if you want to save the error messages from a long job in a file so that they don't scroll off the screen, append 2> file to your command. If you also want to save standard output, append > file1 2> file2.

This leads to Task 7-3.

Task 7-3 You want to start a long job in the background (so that your terminal is freed up) and save both standard output and standard error in a single log file. Write a function that does this.

We'll call this function start. The code is very terse:

function start {
    "$@" > logfile 2>&1 &
}

This line executes whatever command and parameters follow start. (The command cannot contain pipes or output redirectors.) It first sends the command's standard output to logfile.

Then, the redirector 2>&1 says, "Send standard error (file descriptor 2) to the same place as standard output (file descriptor 1)." 2>&1 is actually a combination of two redirectors in Table 7-1: n> file and >&n. Since standard output is redirected to logfile, standard error will go there too. The final & puts the job in the background so that you get your shell prompt back.

As a small variation on this theme, we can send both standard output and standard error into a pipe instead of a file: command 2>&1 | ... does this. (Why this works is described shortly.) Here is a function that sends both standard output and standard error to the logfile (as above) and to the terminal:

function start {
    "$@" 2>&1 | tee logfile &
}

The command tee(1) takes its standard input and copies it to standard output and the file given as argument.

These functions have one shortcoming: you must remain logged in until the job completes. Although you can always type jobs (see Chapter 1) to check on progress, you can't leave your office for the day unless you want to risk a breach of security or waste electricity. We'll see how to solve this problem in Chapter 8.

The other file-descriptor-oriented redirectors (e.g., <&n) are usually used for reading input from (or writing output to) more than one file at the same time. We'll see an example later in this chapter. Otherwise, they're mainly meant for systems programmers, as are <&- (force standard input to close) and >&- (force standard output to close), <&n- (move file descriptor n to standard input) and >&n- (move file descriptor n to standard output).

Finally, we should just note that 0< is the same as <, and 1> is the same as >. (In fact, 0 is the default for any operator that begins with <, and 1 is the default for any operator that begins with >.)

7.1.4. Special Filenames

Normally, when you provide a pathname after an I/O redirector such as < or >, the shell tries to open an actual file that has the given filename. However, there are two kinds of pathnames where the shell instead treats the pathnames specially.

The first kind of pathname is /dev/fd/N, where N is the file descriptor number of an already open file. For example:

# assume file descriptor 6 is already open on a file
print 'something meaningful' > /dev/fd/6   # same as 1>&6

This works even on systems that don't have a /dev/fd directory. This kind of pathname may also be used with the various file attribute test operators of the [[...]] command.

The second kind of pathname allows access to Internet services via either the TCP or UDP protocol. The pathnames are:

/dev/tcp/host/port

Using TCP, connect to remote host host on remote port port. The host may be given as an IP address in dotted-decimal notation (1.2.3.4) or as a hostname (www.oreilly.com). Similarly, the port for the desired service may be a symbolic name (typically as found in /etc/services) or a numeric port number.[95]

[95] The ability to use hostnames was added in ksh93f; use of service names was added in ksh93m.

/dev/udp/host/port

This is the same, but using UDP.

To use these files for two-way I/O, open a new file descriptor using the exec command (which is described in Chapter 9), using the "read and write" operator, <>. Then use read -u and print -u to read from and write to the new file descriptor. (The read command and the -u option to read and print are described later in this chapter.)

The following example, courtesy of David Korn, shows how to do this. It implements the whois(1) program, which provides information about the registration of Internet domain names:

host=rs.internic.net
port=43
exec 3<> /dev/tcp/$host/$port
print -u3 -f "%s\r\n" "$@"
cat <&3

Using the exec built-in command (see Chapter 9), this program uses the "read-and-write" operator, <>, to open a two-way connection to the host rs.internic.net on TCP port 43, which provides the whois service. (The script could have used port=whois as well.) It then uses the print command to send the argument strings to the whois server. Finally, it reads the returned result using cat. Here is a sample run:

$ whois.ksh kornshell.com

Whois Server Version 1.3

Domain names in the .com, .net, and .org domains can now be registered
with many different competing registrars. Go to http://www.internic.net
for detailed information.

   Domain Name: KORNSHELL.COM
   Registrar: NETWORK SOLUTIONS, INC.
   Whois Server: whois.networksolutions.com
   Referral URL: http://www.networksolutions.com
   Name Server: NS4.PAIR.COM
   Name Server: NS0.NS0.COM
   Updated Date: 02-dec-2001


>>> Last update of whois database: Sun, 10 Feb 2002 05:19:14 EST <<<

The Registry database contains ONLY .COM, .NET, .ORG, .EDU domains and
Registrars.

Network programming is beyond the scope of this book. But for most things, you will probably want to use TCP connections instead of UDP connections if you do write any networking programs in ksh.