Expect (Unix Power Tools, 3rd Edition)

28.18. Expect

Go to http://examples.oreilly.com/upt3 for more information on: expect

Expect is a program to control interactive applications such as telnet (Section 1.21) and passwd. These and many other applications interactively prompt and expect a user to enter keystrokes in response. But you can write simple Expect scripts to automate these interactions. Then the Expect program can run the "interactive" program noninteractively. Expect can also be used to automate only parts of a dialogue, since control can be passed from the script to the keyboard and vice versa. This allows a script to do the drudgery and a user to do the fun stuff.

Go to http://examples.oreilly.com/upt3 for more information on: tcl, tk

Expect programs can be written in any language but are almost always written in Tcl. Tcl is an interpreted language that is widely used in many other applications. If you already use a Tcl-based application, you won't have to learn a new language for Expect.

Tcl is a very typical-looking shell-like language. There are commands to set variables (set), control flow (if, while, foreach, etc.), and perform the usual math and string operations. Of course, Unix programs can be called, too.

Expect is integrated on top of Tcl and provides additional commands for interacting with programs. Expect is named after the specific command that waits for output from a program. The expect command is the heart of the Expect program. It describes a list of patterns to watch for. Each pattern is followed by an action; if the pattern is found, the action is executed.

For example, the following fragment is from a script that involves a login. When executed, the script waits for the strings welcome, failed, or busy, and then it evaluates [(executes) -- JP] one of the corresponding actions. The action associated with busy shows how multiple commands can be evaluated. The timeout keyword is a special pattern that matches if no other patterns match in a certain amount of time.

expect {
   "welcome" break
   "failed"  abort
   timeout   abort
   "busy"    {
        puts "I'll wait - the system is busy!"
        continue
    }
}

28.18.1. Dialback

It is surprising how little scripting is necessary to produce something useful. Below is a script that dials a phone. It is used to reverse the charges so that long-distance phone calls are charged to the computer. It is invoked with the phone number as its argument.

spawn tip modem
expect "connected"
send "ATD$argv\r"
# modem takes a while to connect
set timeout 60
expect "CONNECT"

The first line runs the tip program so that the output of a modem can be read by expect and its input written by send. Once tip says it is connected, the modem is told to dial using the command ATD followed by the phone number. The phone number is retrieved from argv, which is a variable predefined to contain the original argument with which the script was called.

The fourth line is just a comment noting that the variable being set in the next line controls how long expect will wait before giving up. At this point, the script waits for the call to complete. No matter what happens, expect terminates. If the call succeeds, the system detects that a user is connected and prompts with login:.

Actual scripts do more error checking, of course. For example, the script could retry if the call fails. But the point here is that it does not take much code to produce useful scripts. This six-line script replaced a 60 KB executable (written in C) that did the same thing!

28.18.2. Automating /bin/passwd

Earlier I mentioned some programs that cannot be automated with the shell. It is difficult to imagine why you might even want to embed some of these programs in shell scripts. Certainly the original authors of the programs did not conceive of this need. As an example, consider passwd.

passwd is the command to change a password. The passwd program does not take the new password from the command line.[90] Instead, it interactively prompts for it -- twice. Here is what it looks like when run by a system administrator. (When run by users, the interaction is slightly more complex because they are prompted for their old passwords as well.)

[90]Newer versions will accept input from STDIN, however.

# passwd libes
Changing password for libes on thunder.
New password:
Retype new password:

This is fine for a single password. But suppose you have accounts of your own on a number of unrelated computers and you would like them all to have the same password. Or suppose you are a system administrator establishing 1,000 accounts at the beginning of each semester. All of a sudden, an automated passwd makes a lot of sense. Here is an Expect script to do just that: automate passwd so that it can be called from a shell script.

spawn passwd [lindex $argv 0]
set password [lindex $argv 1]
expect "password:"
send "$password\r"
expect "password:"
send "$password\r"
expect eof

The first line starts the passwd program with the username passed as an argument. The next line saves the password in a variable for convenience. As in shell scripts, variables do not have to be declared in advance.

In the third line, the expect command looks for the pattern password:. expect waits until the pattern is found before continuing.

After receiving the prompt, the next line sends a password to the current process. The \r indicates a carriage return. (Most of the usual C string conventions are supported.) There are two expect-send sequences because passwd asks the password to be typed twice as a spelling verification. There is no point to this in a noninteractive passwd, but the script has to do it because passwd assumes it is interacting with a human who does not type consistently.

The final command expect eof causes the script to wait for the end-of-file character in the output of passwd. Similar to timeout, eof is another keyword pattern. This final expect effectively waits for passwd to complete execution before returning control to the script.

Take a step back for a moment. Consider that this problem could be solved in a different way. You could edit the source to passwd (should you be so lucky as to have it) and modify it so that given an optional flag, it reads its arguments from the command line just the way that the Expect script does. If you lack the source and have to write passwd from scratch, of course, then you will have to worry about how to encrypt passwords, lock and write the password database, etc. In fact, even if you only modify the existing code, you may find it surprisingly complicated code to look at. The passwd program does some very tricky things. If you do get it to work, pray that nothing changes when your system is upgraded. If the vendor adds NIS, NIS+, Kerberos, shadow passwords, a different encryption function, or some other new feature, you will have to revisit the code.

Expect comes with several example scripts that demonstrate how you can do many things that are impossible with traditional shells. For example, the passmass script lets you update your password on many unrelated machines simultaneously. The rftp script provides your regular ftp client with additional commands to do recursive FTP in either direction. The cryptdir script encrypts all the files in a directory. And an amusing script is provided that lets two chess processes play each other. Expect has no limit to the number of interactive programs it can drive at the same time. The Unix system may limit Expect, though, by controlling the maximum number of processes or other system resources available.

28.18.3. Testing: A Story

Many people use Expect for testing. You can test interactive programs as easily as you can automate them. And hardware lends itself to testing with Expect, too. For example, we solved a thorny problem when we had to deal with an unreliable bank of modems. We were receiving dozens of calls each week reporting "the modem is hung." No indication of which modem, of course. And it was always too late for us to ask the user to try something to investigate the problem. The connection was gone by then. Our solution was an Expect script that connected to each modem hourly and exercised it. Any problems were recorded so that we had a clear and full history of each modem's behavior. As soon as a defective or hung modem was encountered, the Expect script would send email to the system administrator. With this script in place, reports of modem problems from our users dropped to zero.

28.18.4. Other Problems

These are just a few of the problems that can be solved with Expect. And as with all Expect solutions, recompilation of the original programs is unnecessary. You don't even need the source code! Expect handles many other problems as well. For example, Expect can wrap existing interactive tools with GUI wrappers. This means you can wrap interactive programs with graphic frontends to control applications by buttons, scrollbars, and other graphic elements. And Expect scripts work great as CGI scripts or from cron (Section 25.2) or inetd [the daemon that controls Internet services provided by a system -- JP]. Finally, learning Expect may be easier than you think. Expect can watch you interact and then produce an Expect script for you. Interaction automation can't get much easier than this!

More information on Expect is available in Exploring Expect, by Don Libes, from O'Reilly & Associates.

-- DL