Guidelines for Better CGI Applications (CGI Programming with Perl)

Like many forms of programming, CGI programming with Perl is a balance between art and science. As an art form, Perl is such a uniquely expressive language that you have the freedom to accomplish the same tasks in many different ways. However, thinking of Perl as a science, you will want to choose methods based on balancing such real-world requirements as performance, security, and team development.

Furthermore, any program that is useful in one context will generally evolve to be useful in others. This requires that a program be flexible and have the capability to grow. Unfortunately, programs do not grow by themselves. They require the dreaded m-word: maintenance. Maintenance is usually difficult, but it can be made easier by taking steps to make sure that the code is readable as well as flexible.

Because of these concerns, seasoned CGI developers typically end up sticking to a set of guidelines that help their code live up to these expectations. In a corporate setting, these guidelines tend to become the standards through which teams of developers understand how to easily read the code that their neighbors produce.

16.1. Architectural Guidelines

The first step in learning any language consists of being able to accomplish small tasks without the compiler complaining. However, larger programs are made up of more than just syntactically correct statements. The details of how the small parts of a program fit together is just as important as making sure that those same small parts compile successfully.

In other words, a program is literally more than the sum of its parts. Attention must be paid to developing the program in order to accommodate design goals such as flexibility and future maintainability. Sometimes this is referred to as " programming in the large" or "strategic programming." This section emphasizes specific tips on how to architect a CGI application for these design goals.

16.1.1. Plan for Future Growth

Web sites may start small, but they typically grow and evolve over time. You may start out working on a small site without many developers where it is easy to coordinate work. However, as web sites grow and the staff that develops and supports the web site grows, it becomes more critical that it is designed well. Developers should have a development site where they can work on their own copies of the web site without affecting the production web server.

As web sites grow and multiple developers share work on projects, a system to track changes to your applications is crucial. If you are not using a revision control system, you should be planning for one. There are numerous commercial products available for revision control in addition to open source implementations of CVS and RCS. Supporting for a revision control system is an important consideration when making architectural decisions.

You can configure this a number of different ways. Here are a few examples:

Web developers share a common development web server. This is the simplest solution and can work for small groups but quickly becomes unwieldy for large projects. This does not support revision control at a user level, and there is no stable code base because everything is in flux. One developer would be unable to test a component with another developer's code while the second developer is making changes to that code.
Web developers have their own directory tree on the web server. In this example, each developer has a home directory on the web server and can access a copy of web server's contents beneath this directory. This is relatively easy to set up and works if HTML links are relative to the current directory. This supports revision control systems because developers can periodically check in (preferably stable) snapshots of their code. Other developers can update their directories with these snapshots and even develop code in parallel.
Web developers have their own copy of the web server running on a separate port. This requires the most configuration because the web server must be reconfigured each time a port is added for a developer. This works for all relative URLs, whether they contain full paths or paths relative to the current directory. This also supports revision control.

16.1.2. Use Directories to Organize Your Projects

CGI applications often consist of several related files, including one or more CGI scripts, HTML forms, template files -- if you are generating output with templates, data files, configuration files, etc. If your development system is separate from your production server (as it should be), then these systems may have different directory structures.

On your development system you should develop a directory structure that helps you organize this information easily. On systems that support pointers to directories,[22] it is a good idea to place all the files for a given CGI application within one directory. For example, if you had an web storefront application, you might store the components in subdirectories within /usr/local/projects/web_store like so:

[22]Such pointers could include symlinks on Unix or aliases on MacOS; Windows shortcuts are not transparent to applications and thus will not work in this context.

/usr/local/projects/web_store/
    cgi/
    conf/
    data/
    html/
    templates/

You could then create the following symlinks that map this content into the corresponding directories your web server uses:

/usr/local/apache/htdocs/web_store    -> /usr/local/projects/web_store/html/
/usr/local/apache/cgi-bin/web_store   -> /usr/local/projects/web_store/cgi/

You may also wish to add global directories for data, configuration, and template files:

/usr/local/apache/data/web_store      -> /usr/local/projects/web_store/conf/
/usr/local/apache/conf/web_store      -> /usr/local/projects/web_store/data/
/usr/local/apache/templates/web_store -> /usr/local/projects/web_store/templates/

Besides making it easier to locate all of the components that are part of the web store application, placing all of your content beneath a common directory such as /usr/local/projects/web_store makes it easier to manage this application with a revision control system.

Note that it is slower for the web server to follow a symlink than to stay in the document root, so this structure makes more sense on a development system than on a production system.

16.1.3. Use Relative URLs

Your web site will be most flexible if you use relative URLs instead of absolute URLs. In other words, do not include the domain name of your web server when you do not need to. If your development and production web servers have different names, you want your code to work on either system with very little reconfiguration.

Whether these relative URLs contain fully qualified paths or paths that are relative to the current directory depends on how you have configured your development system, as we previously discussed. However, primary navigation elements, such as navigation bars, almost always use fully qualified paths, so configuring your development environment to support this allows the development environment to better mirror the production environment.

16.1.4. Separate Configuration from Your Primary Code

Information that is likely to change in the program or that is dependent upon the environment should be placed in a separate setup file. With Perl, setup files are easy because you can write the file in Perl; they simply need to set one or more global variables. To access these variables in a CGI script, first use Perl's require function to import the configuration file.

In some scenarios, each web developer may need different configuration parameters. By storing file paths in a configuration file, web developers can test their applications with their own copies of data and HTML files. However, that does not mean that CGI scripts need to require multiple files; another advantage to using Perl for setup files is that they are easily extended. A CGI script can require a single configuration file that requires other files. This easily supports configuration files for both applications and developers. Likewise, if a CGI application grows so large that a single application configuration file is difficult to manage, you can break it into smaller files and have the primary configuration file require these smaller sections.

16.1.5. Separating Display from Your Primary Code

The display associated with a CGI script is one of the most likely things to change in the lifetime of an application. Most Web sites undergo some look and feel change during their evolution, and an application that will be used across several web sites needs to be flexible enough to accommodate all of their individual cosmetic guidelines. We discussed many of the arguments for separating display from logic in Chapter 6, "HTML Templates".

However, even beyond keeping HTML separate from code so that HTML maintainers have an easier time, it is a good idea to develop the code that handles display (such as template parsing calls, CGI.pm methods, etc.) separated from the rest of your program logic. This allows you to change the solution you use for generating display with as little effort as possible. You may at some point decide you want to port all your CGI scripts from CGI.pm to templates or vice versa.

Another reason for separating display from the main program logic is that you may not want to limit your program to displaying HTML. As your program evolves, you may want to provide other interfaces. You may wish to convert from basic HTML to the new XHTML standard. Or you might want to add an XML interface to allow other systems programs to grab and process the output of your CGI script as data.

16.1.6. Separating Storage from Your Primary Code

The manner of storing and retrieving data is a key architecture decision that every application encounters. A simple shopping cart might start out using flat text files to store shopping cart data throughout the user's shopping experience. However, a more sophisticated one will probably want to take advantage of relational databases such as MySQL or Oracle. Other applications may use DBM hash files.

Separating the code that is responsible for data storage from your core program logic is good architectural design. In practice, this can be more difficult to achieve than separating other components of your programs such as display. Often your logic is closely tied to your data. Sometimes you must also make trade-offs with performance; SQL for example is such an expressive language, it is possible to embed logic into your queries, and this is typically much faster and more memory efficient than duplicating this functionality in your programs.

However, it is a good idea to strive towards a separation, especially if your application is using simpler storage mechanisms such as text files. Because applications grow, you may easily find yourself adopting a full RDBMS down the road. The least amount of change required in your code, the better.

One strategy is to simply allow DBI to be your layer of abstraction. If you are not ready for a database, you can use DBD::CSV to store your data in text files. Later, if you move to a relational database, most of your code that is built around DBI will not need to change. Keep in mind that not all DBI drivers are equal. DBD::CSV, for example, only supports limited SQL queries; while on the other extreme, complex drivers like DBD::Oracle allow you to use stored procedures written in Oracle's PL/SQL programming language. Thus, even with DBI, you must balance the portability of writing simple, vanilla SQL against the performance advantages that you can get by taking advantage of particular features available to you with your current storage mechanism, as well as the likelihood that you will want to change storage mechanisms in the future.

16.1.7. Number of Scripts per Application

CGI applications often consist of many different tasks that must work together. For example, in a basic online store you will have code to display a product catalog, code to update a shopping cart, code to display the shopping cart, and code to accept and process payment information. Some CGI developers would argue that all of this should be managed by a single CGI script, possibly breaking some functionality out into modules that can be called by this script. Others would argue that a separate CGI scripts should support each page or functional group of pages, possibly moving common code into modules that can be shared by this script. There are reasons for pursuing either approach; let's take a look at them.

16.1.7.1. Using one CGI program rather than many for each major application

Having one file makes things simple; there is only one file one must edit to make changes. One doesn't need to look through multiple files in order to find a particular section of code. Imagine you saw a directory with multiple applications:

web_store.cgi
order.cgi
display_cart.cgi
maintain_cart.cgi

Without delving into the source code, you might pick out that web_store.cgi is the primary application. Furthermore, you might conclude that the program probably prints out a front page inviting the user to shop and provides links to the other CGI programs. You would also be able to tell which scripts have to do with ordering, displaying, and maintaining cart information.

However, without actually going into the source code of all these CGI scripts, it is difficult to tell how they relate to one another. For example, can you add or delete shopping cart items from the order page?

Instead, you can make just one CGI program: web_store.cgi. This combined script can then import the functionality of order forms, cart data display and maintenance using libraries or modules.

Second, different components often need to share code. It is much simpler for one component to access code in another component if they are in the same file. Moving shared code into modules is certainly an alternative that works well for applications distributed into multiple CGI scripts. However, using modules to share common code requires a greater degree of planning to know what code can be shared and what code will not. A single file is more amenable to making simple changes.

It is possible to use modules with this single CGI program approach. In fact, you can keep file sizes small if you want by making the primary CGI script a basic interface, or a wrapper, that routes requests to other modules. In this scenario, you create multiple modules that handle the different tasks. In many ways it is like having multiple files except that all HTTP requests are directed through a common front-end.

If you write CGI scripts that you distribute so that others may download and install them on their own systems, then you may certainly want to reduce the number of files in your application. In this scenario, the focus is on making the application easy to install and configure. People installing software care more about what the package does than one individual tasks are handled by which component, and it is easier for them to avoid accidentally deleting a file they didn't realize was important if the number of files is minimized.

The final reason you may wish to combine CGI scripts is if you are running FastCGI. FastCGI runs a separate process for each of your CGI scripts, so the fewer scripts you have, the fewer separate processes are running.

16.1.7.2. Using multiple CGI scripts for each major application

There are also several reasons to keep applications distributed. First of all, it does keep files smaller and more manageable. This also helps with projects that have multiple developers, because reconciling changes made by multiple developers working on the same file at the same time can be complicated to say the least.

Of course, as we stated before, one can keep files small and separated when using the single CGI program approach by shifting code into modules and restricting the single CGI program to being a simple interface that routes requests to the appropriate modules. However, creating a general front-end that uses modules for specific tasks is a rather backward approach for Perl. Typically, Perl modules contain general code that can be shared across multiple programs that do specific tasks. Keeping general code within modules also allows them to be potentially shared across different CGI applications.

It is true that creating multiple files requires more architectural planning when different components need to share the same code because the common code must be placed in a module. You should always plan your architecture carefully and be wary of quick and simple solutions. The problem with quick and simple solutions is that too many of them begin to bloat an application and create an inferior overall solution. It may require a bit more work in the short term to shift code in one component into a module because another component needs to access it; however, in the long run the application may be much more flexible and easier to maintain with this module than it would be if all the code is simply dumped into a common file.

There are some cases when it is clear that code should be kept separate. Some applications, such as our web store example, may have administrative pages where employees can update product information, modify product categories, etc. Those tasks that require a different level of authorization should certainly be kept separate from the public code for security reasons. Administrative code should be placed in a separate subdirectory, such as /usr/local/apache/cgi-bin/web_store/admin/ that is restricted by the web server.

If you do choose to separate a CGI application into multiple scripts, then you should certainly create a separate directory within /cgi-bin for each application. Placing lots of files from lots of different applications together in one directory guarantees confusion later.

16.1.8. Using Submit Buttons to Control Flow

Whether or not you break your applications into multiple scripts, you will still encounter situations where one form may allow the user to choose very different actions. In this case, your CGI script can determine what action to take by looking at the name of the submit button that was chosen. The name and value of submit buttons is only included within form query requests if they were clicked by the user. Thus, you can have multiple submit buttons on the HTML form with different names indicating different paths of logic that the program should follow.

For example, a simple shopping cart CGI script may begin with code like the following:

#!/usr/bin/perl -wT

use strict;
use CGI;

my $q        = new CGI;
my $quantity = $q->param( "quantity" );
my $item_id  = $q->param( "item_id" );
my $cart_id  = $q->cookie( "cart_id" );

# Remember to handle exceptional cases
defined( $item_id ) or die "Invalid input: no item id";
defined( $cart_id ) or die "No cookie";

if ( $q->param( "add_item" ) ) {
    add_item( $cart_id, $item_id );
} elsif ( $q->param( "delete_item" ) ) {
    delete_item( $cart_id, $item_id );
} elsif ( $q->param( "update_quantity" ) ) {
    update_quantity( $cart_id, $item_id, $quantity );
} else {
    display_cart( $cart_id );
}

# That's it; subroutines follow...

From looking at this section of code, it is easily apparent how the entire script reacts to input and the role of each of the subroutines. If we clicked on a submit button represented in an HTML form with <INPUT TYPE="submit" NAME="add_item"VALUE="Add Item to Cart">, the script would call the add_item subroutine. Furthermore, it is clear that the default behavior is defined as displaying the shopping cart.

Note that we are branching based on the name of the submit button and not the value; this allows HTML designers to alter the text on the button displayed to users without affecting our script.