Understanding Basic Operations (sed & awk, Second Edition)

If you are starting out to learn sed and awk, you can benefit from looking at how much they have in common.

2.1. Awk, by Sed and Grep, out of Ed

You can trace the lineage of awk to sed and grep, and through those two programs to ed, the original UNIX line editor.

Have you ever used a line editor? If so, it will be much easier for you to understand the line orientation of sed and awk. If you have used vi, a full-screen editor, then you are familiar with a number of commands that are derived from its underlying line editor, ex (which in turn is a superset of the features in ed).

Let's look at some basic operations using the line editor ed. Don't worry--this is an exercise intended to help you learn sed and awk, not an attempt to convince you of the wonders of line editors. The ed commands that are shown in this exercise are identical to the sed commands you'll learn later on. Feel free to experiment with ed on your own to get a sense of how it works. (If you're already familiar with ed, feel free to skip to the next section.)

To use a line editor, you work on one line at a time. It is important to know what line you are positioned at in the file. When you open a file using ed, it displays the number of characters in the file and positions you at the last line.

$ ed test
339

There is no prompt. If you enter a command that ed does not understand, it prints a question mark as an error message. You can enter the print command, p, to display the current line.

p
label on the first box.

By default, a command affects only the current line. To make an edit, you move to the line that you want to edit and then apply the command. To move to a line, you specify its address. An address might consist of a line number, a symbol indicating a specific position in the file, or a regular expression. You can go to the first line by entering the line number 1. Then you can enter the delete command to remove that line.

1
You might think of a regular expression
d

Entering "1" makes the first line the current line, displaying it on the screen. The delete command in ed is d and here it deletes the current line. Rather than moving to a line and then editing it, you can prefix an editing command with an address that indicates which line or range of lines is the object of the command. If you enter "1d", the first line would be deleted.

You can also specify a regular expression as an address. To delete a line containing the word "regular," you could issue this command:

/regular/d

where slashes delimit the regular expression and "regular" is the string you want to match. This command deletes the first line containing "regular" and makes the line following it the current line.

NOTE: Make sure you understand that the delete command deletes the whole line. It does not just delete the word "regular" on the line.

To delete all the lines that contain the regular expression, you'd prefix the command with the letter g for global.

g/regular/d

The global command makes all lines that match the regular expression the object of the specified command.

Deleting text can take you only so far. Substituting text (replacing one bit of text with another) is much more interesting. The substitution command, s, in ed is:

[address]s/pattern/replacement/flag

pattern is a regular expression that matches a string in the current line to be replaced by replacement. For example, the following command replaces the first occurrence of "regular" with "complex" on the current line.

s/regular/complex/

No address is specified, so it affects only the first occurrence on the current line. It is an error if "regular" is not found on the current line. To look for multiple occurrences on the same line, you must specify g as a flag:

s/regular/complex/g

This command changes all occurrences on the current line. An address must be specified to direct this command to act upon more than the current line. The following substitution command specifies an address:

/regular/s/regular/complex/g

This command affects the first line that matches the address in the file. Remember, the first "regular" is an address and the second is a pattern to match for the substitution command. To make it apply to all lines, use the global command, putting g before the address.

g/regular/s/regular/complex/g

Now the substitution is made everywhere--all occurrences on all lines.

NOTE: Note the different meanings of "g." The "g" at the beginning is the global command that means make the changes on all lines matched by the address. The "g" at the end is a flag that means change each occurrence on a line, not just the first.

The address and the pattern need not be the same.

g/regular expression/s/regular/complex/g

On any line that contains the string "regular expression," replace "regular" with "complex." If the address and the pattern are the same, you can tell ed by specifying two consecutive delimiters (//).

g/regular/s//complex/g

In this example, "regular" is specified as the address and the pattern to be matched for substitution is the same. If it seems that we've covered these commands quickly and that there is a lot to absorb, don't worry. We will be covering these commands again later on.

The familiar UNIX utility grep is derived from the following global command in ed:

g/re/p

which stands for "global regular expression print." Grep is a line-editing command that has been extracted from ed and made available as an external program. It is hard-wired to perform one editing command. It takes the regular expression as an argument on the command line and uses it as the address of lines to print. Here's an example, looking for lines matching "box":

$ grep 'box' test
You are given a series of boxes, the first one labeled "A",
label on the first box.

It prints all lines matching the regular expression.

One more interesting feature of ed is the ability to script your edits, placing them in a separate file and directing them as input to the line editor. For instance, if a series of commands were put in a file named ed-script, the following command executes the script:

ed test < ed-script

This feature makes ed a programmable editor--that is, you can script any action that you might perform manually.

Sed was created as a special-purpose editor that was meant to execute scripts exclusively; unlike ed, it cannot be used interactively. Sed differs from ed primarily in that it is stream-oriented. By default, all of the input to sed passes through and goes to standard output. The input file itself is not changed. If you actually do want to alter the input file, you typically use the shell mechanism for output redirection, and when you are satisfied with the edits you've made, replace the original file with the modified version.

ed is not stream-oriented and changes are made to the file itself. An ed script must contain commands to save the file and quit the editor. It produces no output to the screen, except what may be generated by a specific command.

The stream orientation of sed has a major impact on how addressing is applied. In ed, a command without an address affects only the current line. Sed goes through the file, a line at a time, such that each line becomes the current line, and the commands are applied to it. The result is that sed applies a command without an address to every line in the file.

Look at the following substitution command:

s/regular/complex/

If you entered this command interactively in ed, you'd substitute "complex" for the first occurrence of "regular" on the current line. In an ed script, if this was the first command in the script, it would be applied only to the last line of the file (ed's default current line). However, in a sed script, the same command applies to all lines. That is, sed commands are implicitly global. In sed, the previous example has the same result as the following global command in ed:

g/regular/s//complex/

NOTE: Understanding the difference between current-line addressing in ed and global-line addressing in sed is very important. In ed you use addressing to expand the number of lines that are the object of a command; in sed, you use addressing to restrict the number of lines affected by a command.

Sed also was designed with a number of additional commands that support script writing. We will look at many of these commands in Chapter 6, "Advanced sed Commands".

Awk was developed as a programmable editor that, like sed, is stream-oriented and interprets a script of editing commands. Where awk departs from sed is in discarding the line-editor command set. It offers in its place a programming language modeled on the C language. The print statement replaces the p command, for example. The concept of addressing is carried over, such that:

/regular/ { print }

prints those lines matching "regular". The braces ({}) surround a series of one or more statements that are applied to the same address.

The advantage of using a programming language in scripts is that it offers many more ways to control what the programmable editor can do. Awk offers expressions, conditional statements, loops, and other programming constructs.

One of the most distinctive features of awk is that it parses, or breaks up, each input line and makes individual words available for processing with a script. (An editor such as vi also recognizes words, allowing you to move word by word, or make a word the object of an action, but these features can only be used interactively.)

Although awk was designed as a programmable editor, users found that awk scripts could do a wide range of other tasks as well. The authors of awk never imagined it would be used to write large programs. But, recognizing that awk was being used in this way, the authors revised the language, creating nawk to offer more support for writing larger programs and tackling general-purpose programming problems. This new version, with minor improvements, is now codified by the POSIX standard.

Chapter 2. Understanding Basic Operations

Contents:

2.1. Awk, by Sed and Grep, out of Ed