Chapter 17. Efficiency and Optimization

17.1.1. Benchmark Your Code

First, here's the simplest way to benchmark:

$start = (times)[0];

## your code goes here

$end = (times)[0];

printf "Elapsed time: %.2f seconds!\n", $end - $start;

This determines the elapsed user time needed to execute your code in seconds. It is important to consider a few rules when benchmarking:

• Try to benchmark only the relevant piece(s) of code.

• Don't accept the first benchmark value. Benchmark the code several times and take the average.

• If you are comparing different benchmarks, make sure they are tested under comparable conditions. For example, make sure that the load on the machine doesn't differ between tests because another user happened to be running a heavy job during one.

Second, we can use the Benchmark module. The Benchmark module provides us with several functions that allow us to compare multiple pieces of code and determine elapsed CPU time as well as elapsed real-world time.

Here's the easiest way to use the module:

use Benchmark;
$start = new Benchmark;

## your code goes here

$end = new Benchmark;

$elapsed = timediff ($end, $start);
print "Elapsed time: ", timestr ($elapsed), "\n";

The result will look similar to the following:

Elapsed time:  4 wallclock secs (0.58 usr +  0.00 sys =  0.58 CPU)

You can also use the module to benchmark several pieces of code. For example:

use Benchmark;
timethese (100, {
                    for => <<'end_for',
                        my   $loop;
                        for ($loop=1; $loop <= 100000; $loop++) { 1 }
end_for
                    foreach => <<'end_foreach'
                        my      $loop;
                        foreach $loop (1..100000) { 1 }
end_foreach
                } );

Here, we are checking the for and foreach loop constructs. As a side note, you might be interested to know that, in cases where the loop iterator is great, foreach is much less efficient than for in versions of Perl older than 5.005.

The resulting output of timethese will look something like this:

Benchmark: timing 100 iterations of for, foreach...
       for: 49 wallclock secs (49.07 usr +  0.01 sys = 49.08 CPU)
   foreach: 69 wallclock secs (68.79 usr +  0.00 sys = 68.79 CPU)

One thing to note here is that Benchmark uses the time system call to perform the actual timing, and therefore the granularity is still limited to one second. If you want higher resolution timing, you can experiment with the Time::HiRes module. Here's an example of how to use the module:

use Time::HiRes;
my $start = [ Time::HiRes::gettimeofday(  ) ];

## Your code goes here

my $elapsed = Time::HiRes::tv_interval( $start );
print "Elapsed time: $elapsed seconds!\n";

The gettimeofday function returns the current time in seconds and microseconds; we place these in a list, and store a reference to this list in $start. Later, after our code has run, we call tv_interval, which takes $start and calculates the difference between the original time and the current time. It returns a floating-point number indicating the number of seconds elapsed.

17.1.2. Benchmark Modules, Too

You should also determine whether using a module make sense. For example, a popular module is IO::File, which provides a set of functions to deal with file I/O:

use IO::File;
$fh = new IO::File;
if ($fh->open ("index.html")) {
    print <$fh>;
    $fh->close;
}

There are advantageous to using an interface like IO::File. Unfortunately, due to module loading and method-call overhead, this code is, on the average, ten times slower than:

if (open FILE, "index.html") {
    print <FILE>;
    close FILE;
}

So the bottom line is, pay very careful attention to modules that you use.

17.1.3. Localize Variables with my

sub name {
    local $my_name = shift;
    greeting(  );
}

sub greeting {
    print "Hello $my_name, how are you!\n";
}

If you run this simple program, you can see that $my_name isn't exactly local to the name function. In fact, it is also visible in greeting. This behavior can produce unexpected results if you are not careful. Thus, most Perl developers avoid using local and use my instead for everything except global variables, file handles, and Perl's built-in global punctuation variables like $_ or $/.

17.1.4. Avoid Slurping

local $/;
open FILE, "large_index.html" or die "Could not open file!\n";
$large_string = <FILE>;
close FILE;

open FILE, "large_index.html" or die "Could not open file!\n";
while (<FILE>) {
    # Split fields by whitespace, output as HTML table row
    print $q->tr( $q->td( [ split ] ) );
}
close FILE;

Of course, there are situations when you cannot process a line at a time. For example, you may be looking for data that crosses line boundaries. In this case, you may fall back to slurping for small files. Try benchmarking your code to see what kind of penalty is imposed by slurping in the entire file.

17.1.5. undef Versus ( )

...
while (<FILE>) {
    chomp;
    $count++;  
    $some_large_array[$count] .= int ($_);
}
...
 
@some_large_array = (  );     ## Good
undef @some_large_array;    ## Not so good

If you undefine @some_large_array to clear it out, Perl will deallocate the space containing the data. And when you populate the array with new data, Perl will have to reallocate the necessary space again. This can slow things down.

17.1.6. SelfLoader

Let's look at how to convert your program to use self-loading, and then we can look at the internals of how it works. Here's a simple framework:

use SelfLoader;

## step 1: subroutine stubs

sub one;
sub two;
...

## your main body of code
...

## step 2: necessary/required subroutines

sub one {
    ...
}

__DATA__

## step 3: all other subroutines

sub two {
    ...
}
...
__END__

It's a three-step process:

1. Create stubs for all the functions and subroutines in your application.

2. Determine which functions are used often enough that they should be loaded by default.

3. Take the rest of your functions and move them between the __DATA__ and __END__ tokens.

Congratulations, Perl will now load these functions only on demand!

Now, how does it actually work? The __DATA__ token has a special significance to Perl; everything after the token is available for reading through the DATA filehandle. When Perl reaches the __DATA__ token, it stops compiling, and all the subroutines defined after the token do not exist, as far as Perl is concerned.

When you call an unavailable function, SelfLoader reads in all the subroutines from the DATA filehandle, and caches them in a hash. This is a one-time process, and is performed the first time you call an unavailable function. It then checks to see if the specified function exists, and if so, will eval it within the caller's namespace. As a result, that function now exists in the caller's namespace, and any subsequent calls to that function are handled via symbol table lookups.

The costs of this process are the one time reading and parsing of the self-loaded subroutines, and a eval for each function that is invoked. Despite this overhead, the performance of large programs with many functions and subroutines can improve dramatically.

17.1.7. autouse

use autouse DB_File;

You have to be very careful when using this feature, since a portion of the chain of execution will shift from compile time to runtime. Also, if a module needs to execute a particular sequence of steps early on in the compile phase, using autouse can potentially break your applications.

If the modules you need behave as expected, using autouse for modules can yield a big savings when it comes time to "load" your application.

17.1.8. Avoid the Shell

system( "/bin/rm", $file );                     ## External command
unlink $file or die "Cannot remove $file: $!";  ## Internal function

system( "/bin/grep", $expr, $file );

Note however, that the circumstances under which you might need to do this are rare. First, Perl must do a lot of extra work to invoke a system call, so the performance difference gained by an external command is seldom worth the overhead. Second, if you only were interested in the first match and not all the matches, then Perl gains speed because your script can exit the loop as soon as it finds a match:

my $match;
open FILE, $file or die "Could not open $file: $!";
while (<FILE>) {
    chomp;
    if ( /$expr/ ) {
        $match = $_;
        last;
    }
}

grep will always read the entire file. Third, if you find yourself needing to resort to using grep to handle text files, it likely means that the problem isn't so much with Perl as with the structure of your data. You should probably consider a different data format, such as a DBM file or a RDBMS.

Instead, use Perl's opendir, readdir, and closedir functions. Here is an example:

@files = </usr/local/apache/htdocs/*.html>;      ## Uses the shell
....
$directory = "/usr/local/apache/htdocs";         ## A better solution
if (opendir (HTDOCS, $directory)) {
    while ($file = readdir (HTDOCS)) {
        push (@files, "$directory/$file") if ($file =~ /\.html$/);
    }
}

17.1.9. Find Existing Solutions for Your Problems

You should also check out the Perl newsgroups. news:comp.lang.perl.modules is a good place to go to check in with new module announcements or to get help with particular modules. news:comp.lang.perl and news:comp.lang.perl.misc are more general newsgroups.

Finally, there are many very good books available that discuss algorithms or useful tricks and tips. The Perl Cookbook by Tom Christiansen and Nathan Torkington and Mastering Algorithms with Perl by Jon Orwant, Jarkko Hietaniemi, and John Macdonald are full of gems specifically for Perl. Of course, don't overlook books whose focus is not Perl. Programming Pearls by John Bentley, The C Programming Language by Brian Kernighan and Dennis Ritchie, and Code Complete by Steve McConnell are also all excellent references.

17.1.10. Regular Expressions

Second, the following type of regular expressions are highly inefficient:

while (<FILE>) {
    next if (/^(?:select|update|drop|insert|alter)\b/);     
    ...  
}

Instead, use the following syntax:

while (<FILE>) {
    next if (/^select/);
    next if (/^update/);
    ...
}

Or, consider building a runtime compile pattern if you do not know what you are searching against at compile time:

@keywords = qw (select update drop insert);
$code = "while (<FILE>) {\n";

foreach $keyword (@keywords) {
    $code .= "next if (/^$keyword/);\n";
}

$code .= "}\n";
eval $code;

This will build a code snippet that is identical to the one shown above, and evaluate it on the fly. Of course, you will incur an overhead for using eval, but you will have to weigh that against the savings you will gain.

Third, consider using o modifier in expressions to compile the pattern only once. Take a look at this example:

@matches = (  );
...
while (<FILE>) {
    push @matches, $_ if /$query/i;
}
...

Code like this is typically used to search for a string in a file. Unfortunately, this code will execute very slowly, because Perl has to compile the pattern each time through the loop. However, you can use the o modifier to ask Perl to compile the regex just once:

push @matches, $_ if /$query/io;

If the value of $query changes in your script, this won't work, since Perl will use the first compiled value. The compiled regex features introduced in Perl 5.005 address this; refer to the perlre manpage for more information.

These tips are general optimization tips. You'll get a lot of mileage from some, and not so much from the others, depending on your application. Now, it's time to look at more complicated ways to optimize our CGI applications.