Using awk (sed & awk, Second Edition)

2.4.1. Running awk

awk 'instructions' files

Input is read a line at a time from one or more files or from standard input. The instructions must be enclosed in single quotes to protect them from the shell. (Instructions almost always contain curly braces and/or dollar signs, which are interpreted as special characters by the shell.) Multiple command lines can be entered in the same way as shown for sed: separating commands with semicolons or using the multiline input capability of the Bourne shell.

Awk programs are usually placed in a file where they can be tested and modified. The syntax for invoking awk with a script file is:

awk -f script files

The -f option works the same way as it does with sed.

While awk instructions have the same structure as sed, consisting of pattern and procedure sections, the procedures themselves are quite different. Here is where awk looks less like an editor and more like a programming language. There are statements and functions instead of one- or two-character command sequences. For instance, you use the print statement to print the value of an expression or to print the contents of the current input line.

Awk, in the usual case, interprets each input line as a record and each word on that line, delimited by spaces or tabs, as a field. (These defaults can be changed.) One or more consecutive spaces or tabs count as a single delimiter. Awk allows you to reference these fields, in either patterns or procedures. $0 represents the entire input line. $1, $2, ... refer to the individual fields on the input line. Awk splits the input record before the script is applied. Let's look at a few examples, using the sample input file list.

The first example contains a single instruction that prints the first field of each line in the input file.

$ awk '{ print $1 }' list
John
Alice
Orville
Terry
Eric
Hubert
Amy
Sal

"$1" refers to the value of the first field on each input line. Because there is no pattern specified, the print statement is applied to all lines. In the next example, a pattern "/MA/" is specified but there is no procedure. The default action is to print each line that matches the pattern.

$ awk '/MA/' list
John Daggett, 341 King Road, Plymouth MA
Eric Adams, 20 Post Road, Sudbury MA
Sal Carpenter, 73 6th Street, Boston MA

Three lines are printed. As mentioned in the first chapter, an awk program can be used more like a query language, extracting useful information from a file. We might say that the pattern placed a condition on the selection of records to be included in a report, namely that they must contain the string "MA". Now we can also specify what portion of a record to include in the report. The next example uses a print statement to limit the output to the first field of each record.

$ awk '/MA/ { print $1 }' list
John
Eric
Sal

It helps to understand the above instruction if we try to read it aloud: Print the first word of each line containing the string "MA". We can say "word" because by default awk separates the input into fields using either spaces or tabs as the field separator.

In the next example, we use the -F option to change the field separator to a comma. This allows us to retrieve any of three fields: the full name, the street address, or the city and state.

$ awk -F, '/MA/ { print $1 }' list
John Daggett
Eric Adams
Sal Carpenter

Do not confuse the -F option to change the field separator with the -f option to specify the name of a script file.

In the next example, we print each field on its own line. Multiple commands are separated by semicolons.

$ awk -F, '{ print $1; print $2; print $3 }' list
John Daggett
 341 King Road
 Plymouth MA
Alice Ford
 22 East Broadway
 Richmond VA
Orville Thomas
 11345 Oak Bridge Road
 Tulsa OK
Terry Kalkas
 402 Lans Road
 Beaver Falls PA
Eric Adams
 20 Post Road
 Sudbury MA
Hubert Sims
 328A Brook Road
 Roanoke VA
Amy Wilde
 334 Bayshore Pkwy
 Mountain View CA
Sal Carpenter
 73 6th Street
 Boston MA

Our examples using sed changed the content of incoming data. Our examples using awk rearrange the data. In the preceding awk example, note how the leading blank is now considered part of the second and third fields.

2.4. Using awk

2.4.1. Running awk

2.4.2. Error Messages

2.4.3. Summary of Options

Table 2.2. Command-Line Options for awk