home | O'Reilly's CD bookshelfs | FreeBSD | Linux | Cisco | Cisco Exam    

sed & awk

sed & awkSearch this book
Previous: 10.4 A Menu-Based Command Generator Chapter 10
The Bottom Drawer
Next: 10.6 Generating Columnar Reports

10.5 Directing Output to Files and Pipes

The output of any print or printf statement can be directed to a file, using the output redirection operators ">" or ">>". For example, the following statement writes the current record to the file data.out :

print > "data.out"

The filename can be any expression that evaluates to a valid filename. A file is opened by the first use of the redirection operator, and subsequent uses append data to the file. The difference between ">" and ">>" is the same as between the shell redirection operators. A right-angle bracket (">") truncates the file when opening it while ">>" preserves whatever the file contains and appends data to it.

Because the redirection operator ">" is the same as the relational operator, there is the potential for confusion when you specify an expression as an argument to the print command. The rule is that ">" will be interpreted as a redirection operator when it appears in an argument list for any of the print statements. To use ">" as a relational operator in an expression that appears in the argument list, put either the expression or the argument list in parentheses. For example, the following example uses parentheses around the conditional expression to make sure that the relational expression is evaluated properly:

print "a =", a, "b =", b, "max =", (a > b ? a : b) > "data.out"

The conditional expression evaluates whether a is greater than b ; if it is, then the value of a is printed as the maximum value; otherwise, b 's value is used.

10.5.1 Directing Output to a Pipe

You can also direct output to a pipe. The command

print | command

opens a pipe the first time it is executed and sends the current record as input to that command. In other words, the command is only invoked once, but each execution of the print command supplies another line of input.

The following script strips troff macros and requests from the current input line and then sends the line as input to wc to determine how many words are in the file:

{# words.awk - strip macros then get word count
print | "wc -w" 

By removing formatting codes, we get a truer word count.

In most cases, we prefer to use a shell script to pipe the output of the awk command to another command rather than do it inside the awk script. For instance, we'd write the previous example as a shell script invoking awk and piping its output to wc :

awk '{ # words -- strip macros 
}' $* | 
# get word count
wc -w

This method seems simpler and easier to understand. Nonetheless, the other method has the advantage of accomplishing the same thing without creating a shell script.

Remember that you can only have so many pipes open at a time. Use the close() function to close the pipe when you are done with it.

10.5.2 Working with Multiple Files

A file is opened whenever you read from or write to a file. Every operating system has some limit on the number of files a running program may have open. Furthermore, each implementation of awk may have an internal limit on the number of open files; this number could be smaller than the system's limit.[4] So that you don't run out of open files, awk provides a close() function that allows you to close an open file. Closing files that you have finished processing allows your program to open more files later on.

[4] Gawk will attempt to appear to have more files open than the system limit by closing and reopening files as needed. Even though gawk is "smart," it is still more efficient to close your files when you're done with them.

A common use for directing output to files is to split up a large file into a number of smaller files. Although UNIX provides utilities, split and csplit , that do a similar job, they do not have the ability to give the new file a useful filename.

Similarly, sed can be used to write to a file, but you must specify a fixed filename. With awk, you can use a variable to specify the filename and pick up the value from a pattern in the file. For instance, if $1 provided a string that could be used as a filename, you could write a script to output each record to its own file:

print $0 > $1

You should perhaps test the filename, either to determine its length or to look for characters that cannot be used in a filename.

If you don't close your files, such a program would eventually run out of available open files, and have to give up. The example we are going to look at works because it uses the close() function so that you will not run into any open-file limitations.

The following script was used to split up a large file containing dozens of manpages. Each manual page began by setting a number register and ended with a blank line:

.nr X 0

(Although they used the -man macros for the most part, the beginning of a manpage was strangely coded, making things a little harder.) The line that provides the filename looks like this:

.if \nX=0 .ds x}  XDrawLine "" "Xlib - Drawing Primitives"

The fifth field on this line, "XDrawLine," contains the filename. Perhaps the only difficulty in writing the script is that the first line is not the one that provides the filename. Therefore, we collect the lines in an array until we get a filename. Once we get the filename, we output the array, and from that point on we simply write each input line to the new file. Here's the man.split script:

# man.split -- split up a file containing X manpages. 
BEGIN { file = 0; i = 0; filename = "" }

# First line of new manpage is ".nr X 0"
# Last line is blank
/^\.nr X 0/,/^$/ {
	# this conditional collects lines until we get a filename.
	if (file == 0)
		line[++i] = $0
		print $0 > filename

	# this matches the line that gives us the filename
	if ($4 == "x}") {
		# now we have a filename
		filename = $5 
		file = 1
		# output name to screen 
		print filename 
		# print any lines collected
		for (x = 1; x <= i; ++x){
			print line[x] > filename
		i = 0

	# close up and clean up for next one
	if ($0 ~ /^$/) {
		filename = ""
		file = 0
		i = 0

As you can see, we use the variable file as a flag to convey whether or not we have a valid filename and can write to the file. Initially, file is 0, and the current input line is stored in an array. The variable i is a counter used to index the array. When we encounter the line that sets the filename, then we set file to 1. The name of the new file is printed to the screen so that the user can get some feedback on the progress of the script. Then we loop through the array and output it to the new file. When the next input line is read, file will be set to 1 and the print statement will output it to the named file.

Previous: 10.4 A Menu-Based Command Generator sed & awk Next: 10.6 Generating Columnar Reports
10.4 A Menu-Based Command Generator Book Index 10.6 Generating Columnar Reports

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System