adj--Adjust Lines for Text Files (sed & awk, Second Edition)

13.5. adj--Adjust Lines for Text Files

Contributed by Norman Joseph

        Well, I decided to take you up on your offer.  I'm sure there are
more sophisticated gurus out there than me, but I do have a nawk script
that I'm kind of fond of, so I'm sending it in.

        Ok, here's the low down.  When I'm writing e-mail, I often make a
lot of changes to the text (especially if I'm going to post on the net).
So what starts out as a nicely adjusted letter or posting usually ends up
looking pretty sloppy by the time I'm done adding and deleting lines.  So
I end up spending a lot of time joining and breaking lines all through my
document so as to get a nice right-hand margin.  So I say to myself,
"This is just the kind of tedious work a program would be good for."

        Now, I know I can use nroff to filter my document through and
adjust the lines, but it has lousy defaults (IMHO) for simple text like
this.  So, with a view to sharpening my nawk skills I wrote adj.nawk
and the accompanying shell script wrapper adj.  

Here's the syntax for the nawk filter adj:


adj [-l|c|r|b] [-w n] [-i n] [files ...]


The options are:

-l: Lines are left adjusted, right ragged (default).
-c: Lines are centered.
-r: Lines are right adjusted, left ragged.
-b: Lines are left and right adjusted.
-w n: Sets line width to n characters (default is 70).
-i n: Sets initial indent to n characters (default is 0).

        So, whenever I'm finished with this letter (I'm using vi) I will
give the command :%!adj -w73 (I like my lines a little longer) and
all the breaking and joining will be done by a program (the way the Good
Lord intended :-).  Indents and blank lines are preserved, and two spaces
are given after any end-of-sentence punctuation.

        The program is naive about tabs, and when computing line lengths,
it considers a tab character to be one space wide.

        The program is notable for its use of command-line parameter
assignment, and some of the newer features of awk (nawk), such as the
match and split built-in functions, and for its use of support functions.

#! /bin/sh
#
# adj - adjust text lines
#
# usage: adj [-l|c|r|b] [-w n] [-i n] [files ...]
#
# options:
#    -l    - lines are left adjusted, right ragged (default)
#    -c    - lines are centered
#    -r    - lines are right adjusted, left ragged
#    -b    - lines are left and right adjusted
#    -w n  - sets line width to <n> characters (default: 70)
#    -i n  - sets initial indent to <n> characters (default: 0)
#
# note:
#    output line width is -w setting plus -i setting
#
# author:
#    Norman Joseph (amanue!oglvee!norm)

adj=l
wid=70
ind=0

set -- `getopt lcrbw:i: $*`
if test $? != 0
then
    printf 'usage: %s [-l|c|r|b] [-w n] [-i n] [files ...]' $0
    exit 1
fi

for arg in $*
do
    case $arg in
    -l) adj=l;  shift;;
    -c) adj=c;  shift;;
    -r) adj=r;  shift;;
    -b) adj=b;  shift;;
    -w) wid=$2;  shift 2;;
    -i) ind=$2;  shift 2;;
    --) shift;  break;;
    esac
done

exec nawk -f adj.nawk type=$adj linelen=$wid indent=$ind $*

Here's the adj.nawk script that's called by the shell script adj.

# adj.nawk -- adjust lines of text per options
#
# NOTE:  this nawk program is called from the shell script "adj"
#    see that script for usage & calling conventions
#
# author:
#    Norman Joseph (amanue!oglvee!norm)

BEGIN  {
    FS = "\n"
    blankline  = "^[ \t]*$"
    startblank = "^[ \t]+[^ \t]+"
    startwords = "^[^ \t]+"
}

$0 ~ blankline {
    if ( type == "b" )
        putline( outline "\n" )
    else
        putline( adjust( outline, type ) "\n" )
    putline( "\n" )
    outline = ""
}

$0 ~ startblank {
    if ( outline != "" ) {
        if ( type == "b" )
            putline( outline "\n" )
        else
            putline( adjust( outline, type ) "\n" )
    }

    firstword = ""
    i = 1
    while ( substr( $0, i, 1 ) ~ "[ \t]" ) {
        firstword = firstword substr( $0, i, 1 )
        i++
    }
    inline = substr( $0, i )
    outline = firstword

    nf = split( inline, word, "[ \t]+" )

    for ( i = 1;  i <= nf;  i++ ) {
        if ( i == 1 ) {
            testlen = length( outline word[i] )
        } else {
            testlen = length( outline " " word[i] )
            if ( match( ".!?:;", "\\" substr( outline,
                    length( outline ), 1 )) )
                testlen++
        }

        if ( testlen > linelen ) {
            putline( adjust( outline, type ) "\n" )
            outline = ""
        }

        if ( outline == "" )
            outline = word[i]
        else if ( i == 1 )
            outline = outline word[i]
        else {
            if ( match( ".!?:;", "\\" substr( outline,
                   length( outline ), 1 )) )
                outline = outline "  " word[i]     # 2 spaces
            else
                outline = outline " " word[i]      # 1 space
        }
    }
}

$0 ~ startwords  {
    nf = split( $0, word, "[ \t]+" )

    for ( i = 1;  i <= nf;  i++ ) {
        if ( outline == "" )
            testlen = length( word[i] )
        else {
            testlen = length( outline " " word[i] )
            if ( match( ".!?:;", "\\" substr( outline,
                   length( outline ), 1 )) )
                testlen++
        }

        if ( testlen > linelen ) {
            putline( adjust( outline, type ) "\n" )
            outline = ""
        }

        if ( outline == "" )
            outline = word[i]
        else {
            if ( match( ".!?:;", "\\" substr( outline,
                   length( outline ), 1 )) )
                outline = outline "  " word[i]     # 2 spaces
            else
                outline = outline " " word[i]      # 1 space
        }
    }
}

END  {
    if ( type == "b" )
        putline( outline "\n" )
    else
        putline( adjust( outline, type ) "\n" )
}


#
# -- support functions --
#

function putline( line,    fmt )
{
    if ( indent ) {
        fmt = "%" indent "s%s"
        printf( fmt, " ", line )
    } else
        printf( "%s", line )
}


function adjust( line, type,    fill, fmt )
{
    if ( type != "l" )
        fill = linelen - length( line )

    if ( fill > 0 ) {
        if        ( type == "c" ) {
            fmt = "%" (fill+1)/2 "s%s"
            line = sprintf( fmt, " ", line )
        } else if ( type == "r" ) {
            fmt = "%" fill "s%s"
            line = sprintf( fmt, " ", line )
        } else if ( type == "b" ) {
            line = fillout( line, fill )
        }
    }

    return line
}


function fillout( line, need,    i, newline, nextchar, blankseen )
{
    while ( need ) {
        newline = ""
        blankseen = 0

        if ( dir == 0 ) {
            for ( i = 1;  i <= length( line );  i++ ) {
                nextchar = substr( line, i, 1 )
                if ( need ) {
                    if ( nextchar == " " ) {
                        if ( ! blankseen ) {
                            newline = newline " "
                            need--
                            blankseen = 1
                        }
                    } else {
                        blankseen = 0
                    }
                }
                newline = newline nextchar
            }

        } else if ( dir == 1 ) {
            for ( i = length( line );  i >= 1;  i-- ) {
                nextchar = substr( line, i, 1 )
                if ( need ) {
                    if ( nextchar == " " ) {
                        if ( ! blankseen ) {
                            newline = " " newline
                            need--
                            blankseen = 1
                        }
                    } else {
                        blankseen = 0
                    }
                }
                newline = nextchar newline
            }
        }

        line = newline

        dir = 1 - dir
    }

    return line
}

13.5.1. Program Notes for adj

This small text formatter is a nifty program for those of us who use text editors. It allows you to set the maximum line width and justify paragraphs and thus can be used to format mail messages or simple letters.

The adj shell script does all the option setting, although it could have been done by reading ARGV in the BEGIN action. Using the shell to establish command-line parameters is probably easier for those who are already familiar with the shell.

The lack of comments in the adj.awk script makes this script more difficult to read than some of the others. The BEGIN procedure assigns three regular expressions to variables: blankline, startblank, startwords. This is a good technique (one that you'll see used in lex specifications) because regular expressions can be difficult to read and the name of the variable makes it clear what it matches. Remember that modern awks lets you supply a regular expression as a string, in a variable.

There are three main procedures, which can be named by the variable they match. The first is blankline, a procedure which handles collected text once a blank line is encountered. The second is startblank, which handles lines that begin with whitespace (spaces or tabs). The third is startwords, which handles a line of text. The basic procedure is to read a line of text and determine how many of the words in that line will fit, given the line width, outputting those that will fit and saving those that will not in the variable outline. When the next input line is read, the contents of outline must be output before that line is output.

The adjust() function does the work of justifying the text based on a command-line option specifying the format type. All types except "l" (left-adjusted, right-ragged) need to be filled. Therefore, the first thing this function does is figure out how much "fill" is needed by subtracting the length of the current line from the specified line length. It makes excellent use of the sprintf() function to actually do the positioning of the text. For instance, to center text, the value of fill (plus 1) is divided by 2 to determine the amount of padding needed on each side of the line. This amount is passed through the fmt variable as the argument to sprintf():

fmt = "%" (fill+1)/2 "s%s"
            line = sprintf( fmt, " ", line )

Thus, the space will be used to pad a field that is the length of half the amount of fill needed.

If text is right-justified, the value of fill itself is used to pad the field. Finally, if the format type is "b" (block), then the function fillout is called to determine where to add spaces that will fill out the line.

In looking over the design of the program, you can see, once again, how the use of functions helps to clarify what a program is doing. It helps to think of the main procedure as controlling the flow of input through the program while procedures handle the operations performed on the input. Separating the "operations" from the flow control makes the program readable and more easily maintained.

In passing, we're not sure why FS, the field separator, is set to newline in the BEGIN procedure. This means that the field and record separators are the same (i.e., $0 and $1 are the same). The split() function is called to break the line into fields using tabs or spaces as the delimiter.

nf = split( $0, word, "[ \t]+" )

It would seem that the field separator could have been set to the same regular expression, as follows:

FS = "[ \t]+"

It would be more efficient to use the default field parsing.

Finally, using the match() function to find punctuation is inefficient; it would have been better to use index().