29.7 Count How Many Times Each Word Is Used

Here's this book's Preface run through wordfreq :

% wordfreq ch00


 141 the
  98 to
  84 and
  84 of
  71 a
  55 in
  44 that
  38 book
  32 we
  ...

The script was taken from a long-ago Usenet (1.33 ) posting by Carl Brandauer. Here is Carl's original script (with a few small edits):

tr
 


sort
 
uniq
 


-4
 

cat $* |   # tr reads the standard input
tr "[A-Z]" "[a-z]" |   # Convert all uppercase to lowercase
tr -cs "a-z'" "\012" |   # replace all characters not a-z or '
   # with a new line. i.e. one word per line
sort |   # uniq expects sorted input
uniq -c |   # Count number of times each word appears
sort +0nr +1d |   # Sort first from most to least frequent,
   # then alphabetically
pr -w80 -4 -h "Concordance for $*"     # Print in four columns

The version on the disc is somewhat different. It adjusts the tr commands for the script's -i option. The disc version also doesn't use pr to make output in four columns, though you can add that to your copy of the script - or just pipe the wordfreq output through pr on the command line when you need it.

The second tr command above (with the -cs options) is for the Berkeley version of tr . For System V tr , the command should be:

tr -cs "[a-z]'" "[\012*]"

If you aren't sure which version of tr you have, see article 35.11 . You could use deroff (29.10 ) instead.

One of the beauties of a simple script like this is that you can tweak it if you don't like the way it counts. For example, if you want hyphenated words like copy-editor to count as one, add a hyphen to the tr -cs expression: "[a-z]'-" (System V) or "-a-z'" (Berkeley).