[Chapter 7] 7.11 Information Retrieval

7.11 Information Retrieval

An awk program can be used to retrieve information from a database, the database basically being any kind of text file. The more structured the text file, the easier it is to work with, although the structure might be no more than a line consisting of individual words.

The list of acronyms below is a simple database.

$ cat acronyms


BASIC	Beginner's All-Purpose Symbolic Instruction Code
CICS	Customer Information Control System
COBOL	Common Business Oriented Language
DBMS	Data Base Management System
GIGO	Garbage In, Garbage Out
GIRL 	Generalized Information Retrieval Language

A tab is used as the field separator. We're going to look at a program that takes an acronym as input and displays the appropriate line from the database as output. (In the next chapter, we're going to look at two other programs that use the acronym database. One program reads the list of acronyms and then finds occurrences of these acronyms in another file. The other program locates the first occurrence of these acronyms in a text file and inserts the description of the acronym.)

The shell script that we develop is named acro . It takes the first argument from the command line (the name of the acronym) and passes it to the awk script. The acro script follows:

$ cat acro


#! /bin/sh
# assign shell's $1 to awk search variable
awk '$1 == search' search=$1 acronyms

The first argument specified on the shell command line ($1) is assigned to the variable named search ; this variable is passed as a parameter into the awk program. Parameters passed to an awk program are specified after the script section. (This gets somewhat confusing, because $1 inside the awk program represents the first field of each input line, while $1 in the shell represents the first argument supplied on the command line.)

The example below demonstrates how this program can be used to find a particular acronym on our list.

$ acro CICS


CICS Customer Information Control System

Notice that we tested the parameter as a string ($1 == search ). We could also have written this as a regular expression match ($1 ~ search ).

7.11.1 Finding a Glitch

A net posting was once forwarded to one of us because it contained a problem that could be solved using awk. Here's the original posting by Emmett Hogan:

I have been trying to rewrite a sed/tr/fgrep script that we use quite
a bit here in Perl, but have thus far been unsuccessful...hence this
posting.  Having never written anything in perl, and not wishing to
wait for the Nutshell Perl Book, I figured I'd tap the knowledge of this
group.

Basically, we have several files which have the format:

item	info line 1
	info line 2
	  .
	  .
	  .
	info line n

Where each info line refers to the item and is indented by either
spaces or tabs.  Each item "block" is separated by a blank line.

What I need to do, is to be able to type:

info glitch filename

Where info is the name of the perl script, glitch is what I want to
find out about, and filename is the name of the file with the
information in it.  The catch is that I need it to print the entire
"block" if it finds glitch anywhere in the file, i.e.:

machine		Sun 3/75
		8 meg memory
		Prone to memory glitches
		more info
		more info

would get printed if you looked for "glitch" along with any other
"blocks" which contained the word glitch.

Currently we are using the following script:

#!/bin/csh -f
#
sed '/^ /\!s/^/@/' $2 | tr '\012@' '@\012' | fgrep -i $1 | tr '@' '\012'

Which is in a word....SLOW.

I am sure Perl can do it faster, better, etc...but I cannot figure it out.

Any, and all, help is greatly appreciated.

Thanks in advance,
Emmett

-------------------------------------------------------------------
Emmett Hogan              Computer Science Lab, SRI International

The problem yielded a solution based on awk. You may want to try to tackle the problem yourself before reading any further. The solution relies on awk's multiline record capability and requires that you be able to pass the search string as a command-line parameter.

Here's the info script using awk:

awk 'BEGIN { FS = "\n"; RS = "" }
$0 ~ search { print $0 }' search=$1 $2

Given a test file with multiple entries, info was tested to see if it could find the word "glitch."

$ info glitch glitch.test


machine		Sun 3/75
		8 meg memory
		Prone to memory glitches
		more info
		more info

In the next chapter, we look at conditional and looping constructs, and arrays.