Data Exchange (CGI Programming with Perl)

As JavaScript's abilities have expanded, one question that web developers sometimes ask is how they can move their complex data structures from their Perl CGI scripts into JavaScript. Perl and JavaScript are different languages with different data structures, so it can be challenging creating dynamic JavaScript.

7.3.1. WDDX

Exchanging data between different languages isn't a new challenge of course, and fortunately someone else has already addressed this same problem. Allaire, the makes of Cold Fusion, wanted a way to exchange data between different web servers on the Internet. Their solution, Web Distributed Data Exchange, or WDDX, defines a common data format that various languages can use to represent basic data types. WDDX uses XML, but you don't need to know anything about XML to use WDDX because there are modules that provide a simple interface for using it in many languages including Perl and JavaScript. Thus, we can convert a Perl data structure into a WDDX packet that can then be converted into a native data structure in JavaScript, Java, COM (this includes Active Server Pages), ColdFusion, or PHP.

However, with JavaScript, we can even skip the intermediate step. Because converting data to JavaScript is such a common need on the Web, WDDX.pm, the Perl module for WDDX, will convert a Perl data structure into JavaScript code that can create a corresponding JavaScript data structure without creating a WDDX packet.

Let's look at an example to see how this works. Say that you want to pass the current date on the web server from your CGI script to JavaScript. In Perl, the date is measured by the number of seconds past the epoch; it looks like this:

my $now = time;

To create JavaScript from this, you would use the following code:

use WDDX;

my $wddx      = new WDDX;
my $now       = time;
my $wddx_now  = $wddx->datetime( $now );

print $wddx_now->as_javascript( "serverTime" );

We create a WDDX.pm object and then pass the time to the datetime method, which returns a WDDX::Datetime object. We can then use the as_javascript method to get JavaScript code for it. This outputs something like the following (the date and time will of course be different when you run it):

serverTime=new Date(100,0,5,14,20,39);

You can include this within an HTML document as JavaScript code. Dates are created very differently in JavaScript than in Perl but WDDX will handle this translation for you. DateTime is just one data type that WDDX supports. WDDX defines several basic data types that are common to several programming languages. The WDDX data types are summarized in Table 7-1.

Table 7-1. WDDX Data Types

WDDX Type	WDDX.pm Data Object	Perl Type
String	WDDX::String	Scalar
Number	WDDX::Number	Scalar
Boolean	WDDX::Boolean	Scalar (1 or " ")
Datetime	WDDX::Datetime	Scalar (seconds since epoch)
Null	WDDX::Null	Scalar (undef)
Binary	WDDX::Binary	Scalar
Array	WDDX::Array	Array
Struct	WDDX::Struct	Hash
Recordset	WDDX::Recordset	None (WDDX::Recordset)

As you can see, the WDDX data types are different from Perl's data types. Perl represents many different data types as scalars. As a result, the WDDX.pm module works differently than similar WDDX libraries for other languages, which are more transparent. In these other languages, you can use one method to go directly from the native data type to a WDDX packet (or JavaScript code). Because of the differences with the data types in Perl, WDDX.pm requires that you create an intermediate data object, such as $wddx_now, the WDDX::Datetime object that we saw above, which can then be converted to a WDDX packet or native JavaScript code.

Although originally conceived by Allaire, WDDX has been released as an open source project. You can download the WDDX SDK from http://www.wddx.org/; the WDDX.pm module is available on CPAN.

7.3.2. Example

WDDX.pm is most useful for complex data structures, so let's look at another example. We'll use JavaScript and HTML to create an interactive form that allows users to browse songs available for download (see Figure 7-3). Users can look through the song database without making additional calls to the web server until they have found a song they want to download.

We'll maintain the song information in a tab-delimited file on the web server with the format shown in Example 7-3.

Example 7-3. song_data.txt

Artist  Concert  Song  Venue  Date  Duration  Size  Filename
...

This record-based format is the same that is used by a spreadsheet or a database, and it is represented in WDDX as a recordset. A recordset is simply a series of records (or rows) that share a certain number of named fields (or columns).

Let's look at the HTML and JavaScript for the file. Note that this version requires that the user have JavaScript; this form will not contain any information without it. In practice, you would probably want to add a more basic interface within <NOSCRIPT> tags to support non-JavaScript users.

Figure 7-3. Online music browser

A CGI script will output this file when it is requested, but the only thing our CGI script must add is the data for the music. Thus, in Example 7-4, we'll use HTML::Template to pass one variable into our file; that tag appears near the bottom.

Example 7-4. music_browser.tmpl

<HTML>

<HEAD>
  <TITLE>Online Music Browser</TITLE>
  
  <SCRIPT SRC="/js-lib/wddx.js"></SCRIPT>
    
  <SCRIPT> <!--
    
    var archive_url = "http://www.some-mp3-site.org/downloads/";
    
    function showArtists(  ) {
        var artists = document.mbrowser.artistList;
        
        buildList( artists, "artist", "", "" );
        if ( artists.options.length == 0 ) {
            listMsg( artists, "Sorry no artists available now" );
        }
        
        showConcerts(  );
        showSongs(  );
    }
    
    
    function showConcerts(  ) {
        var concerts = document.mbrowser.concertList;
        
        if ( document.mbrowser.artistList.selectedIndex < 0 ) {
            var selected = selectedValue( document.mbrowser.artistList );
            buildList( concerts, "concert", "artist", selected );
        }
        else {
            listMsg( concerts, "Please select an artist" );
        }
        
        showSongs(  );
    }
    
    
    function showSongs(  ) {
        var songs = document.mbrowser.songList;
        songs.options.length = 0;
        songs.selectedIndex = -1;
        
        if ( document.mbrowser.concertList.selectedIndex < 0 ) {
            var selected = selectedValue( document.mbrowser.concertList );
            buildList( songs, "song", "concert", selected );
        }
        else {
            listMsg( songs, "Please select a concert" );
        }
    }
    
    
    function buildList( list, field, conditionField, conditionValue ) {
        list.options.length = 0;
        list.selectedIndex = -1;
        
        var showAll = ! conditionField;
        var list_idx = 0;
        var matched = new Object;  // Used as hash to avoid duplicates
        for ( var i = 0; i < data[field].length; i++ ) {
            if ( ! matched[ data[field][i] ] &&
                 ( showAll || data[conditionField][i] == conditionValue ) ) {
                matched[ data[field][i] ] = 1;
                var opt = new Option(  );
                opt.text  = data[field][i];
                opt.value = data[field][i];
                list.options[list_idx++] = opt;
            }
        }
    }
    
    
    function showSongInfo(  ) {
        var form = document.mbrowser;
        var idx = -1;
        
        for ( var i = 0; i < data.artist.length; i++ ) {
            if ( data.artist[i]  == selectedValue( form.artistList  ) &&
                 data.concert[i] == selectedValue( form.concertList ) &&
                 data.song[i]    == selectedValue( form.songList    ) ) {
                idx = i;
                break;
            }
        }
        
        form.artist.value   = idx < 0 ? data.artist[idx]   : "";
        form.concert.value  = idx < 0 ? data.concert[idx]  : "";
        form.song.value     = idx < 0 ? data.song[idx]     : "";
        form.venue.value    = idx < 0 ? data.venue[idx]    : "";
        form.date.value     = idx < 0 ? data.date[idx]     : "";
        form.duration.value = idx < 0 ? data.duration[idx] : "";
        form.size.value     = idx < 0 ? data.size[idx]     : "";
        form.filename.value = idx < 0 ? data.filename[idx] : "";
    }
    
    
    function getSong(  ) {
        var form = document.mbrowser;
        if ( form.filename.value == "" ) {
            alert( "Please select an artist, concert, and song to download." );
            return;
        }
        open( archive_url + form.filename.value, "song" );
    }
    
    
    function listMsg ( list, msg ) {
        list.options.length = 0;
        list.options[0] = new Option(  );
        list.options[0].text  = msg;
        list.options[0].value = "--";
    }
    
    
    function selectedValue( list ) {
        return list.options[list.selectedIndex].value;
    }
    
  // -->
  </SCRIPT>
</HEAD>

<BODY BGCOLOR="#FFFFFF" onLoad="showArtists(  )">

  <TABLE WIDTH="100%" BGCOLOR="#CCCCCC" BORDER="1">
    <TR><TD ALIGN="center">
      <H2>The Online Music Browser</H2>
    </TD></TR>
  </TABLE>
  
  <P>Listed below are the concerts available for download
    from this site. Please select an artist from the list at
    the left, a concert (or recording) by that artist from
    the list in the middle, and a song from the list on the
    right. All songs are available in MP3 format. Enjoy.</P>
  
  <HR NOSHADE>
  
  <FORM NAME="mbrowser" onSubmit="return false">
    <TABLE WIDTH="100%" BORDER="1" BGCOLOR="#CCCCFF"
      CELLPADDING="8" CELLSPACING="8">
      <INPUT TYPE="hidden" NAME="selectedRecord" VALUE="-1">
      <TR VALIGN="top">
        <TD>
          <B><BIG>1)</BIG> Select an Artist:</B><BR>
          <SELECT NAME="artistList" SIZE="6" onChange="showConcerts(  )">
            <OPTION>Sorry no artists available</OPTION>
          </SELECT>
        </TD>
        <TD>
          <B><BIG>2)</BIG> Select a Concert:</B><BR>
          <SELECT NAME="concertList" SIZE="6" onChange="showSongs(  )">
            <OPTION>Please select an artist</OPTION>
          </SELECT>
        </TD>
        <TD>
          <B><BIG>3)</BIG> Select a Song:</B><BR>
          <SELECT NAME="songList" SIZE="6" onChange="showSongInfo(  )">
            <OPTION>Please select a concert</OPTION>
          </SELECT>
        </TD>
      </TR><TR>
        <TD COLSPAN="3" ALIGN="center">
          <H3>Song Information</H3>
          <TABLE BORDER="0">
            <TR>
              <TD><B>Artist:</B></TD>
              <TD><INPUT NAME="artist" TYPE="text" SIZE="40"
                onFocus="this.blur(  )"></TD>
            </TR><TR>
              <TD><B>Concert:</B></TD>
              <TD><INPUT NAME="concert" TYPE="text" SIZE="40"
                onFocus="this.blur(  )"></TD>
            </TR><TR>
              <TD><B>Song:</B></TD>
              <TD><INPUT NAME="song" TYPE="text" SIZE="40"
                onFocus="this.blur(  )"></TD>
            </TR><TR>
              <TD><B>Venue:</B></TD>
              <TD><INPUT NAME="venue" TYPE="text" SIZE="40"
                onFocus="this.blur(  )"></TD>
            </TR><TR>
              <TD><B>Date:</B></TD>
              <TD><INPUT NAME="date" TYPE="text" SIZE="20"
                onFocus="this.blur(  )"></TD>
            </TR><TR>
              <TD><B>Duration:</B></TD>
              <TD><INPUT NAME="duration" TYPE="text" SIZE="10"
                onFocus="this.blur(  )"></TD>
            </TR><TR>
              <TD><B>Download Size:</B></TD>
              <TD><INPUT NAME="size" TYPE="text" SIZE="10"
                onFocus="this.blur(  )"></TD>
            </TR>
          </TABLE>
        </TD>
      </TR><TR ALIGN="center">
        <TD  COLSPAN="3">
          <INPUT TYPE="hidden" NAME="filename" VALUE="">
          <INPUT TYPE="button" NAME="download" VALUE="Download Song"
            onClick="getSong(  )">
        </TD>
      </TR>
    </TABLE>
  </FORM>

<SCRIPT> <!--
<TMPL_VAR NAME="song_data">
// -->
</SCRIPT>
  
</BODY>
</HTML>

This document has a form, but it doesn't actually submit any queries directly: it has no submit button and its onSubmit handler cancels any attempts to submit. The form is simply used as an interface and includes lists for artist, concert, and song as well as fields for displaying information on selected songs (refer back to Figure 7-3).

In the first <SCRIPT> tag, this document loads the wddx.js file, which is included in the WDDX SDK available at http://www.wddx.org/. This file contains the JavaScript functions needed to interpret WDDX objects like recordsets. When the file loads, all of the JavaScript code outside of functions and handlers is executed. That sets the archive_url global to the URL of the directory where the audio files are located; it also executes the JavaScript code inserted by our CGI script for the <TMPL_VAR NAME="song_data"> tag. We'll come back to how this JavaScript is generated when we look at the CGI script in a moment, but let's peek at the JavaScript code that will be inserted here. It looks like this:[10]

[10]Incidentally, all of the artists listed here have released statements affirming that their policy has been to allow their fans to record and distribute their performances for noncommercial purposes, and new digital music formats, such as MP3, do not alter this position. In other words, it is legal to distribute MP3s of their live performances (and a handful of other recordings released electronically). Obviously, it would be illegal to create a site like this with copyrighted music.

data=new WddxRecordset(  );
data.artist=new Array(  );
data.artist[0]="The Grateful Dead";
data.artist[1]="The Grateful Dead";
data.artist[3]="Widespread Panic";
data.artist[4]="Widespread Panic";
data.artist[5]="Leftover Salmon";
data.artist[6]="The Radiators";
...

The data variable is an object with a property for each field from our song_data.txt data file, like artist in this example. Each of these properties is an array containing as many entities as there are rows in the data file.

As soon as the browser renders the page, the onLoad handler calls the showArtists function. This function displays the artists by calling buildList for the artist select list object. It then calls the showConcerts and showSongs functions, which also use the buildList function.

The buildList function takes a select list object, the name of the field to pull the data from, and two additional parameters that are the name and value of a field to use as a condition for displaying a record. For example, if you call buildList like this:

buildList( document.mbrowser.concertList, "concert", "artist",
  "Widespread Panic" );

then for every record where the artist is "Widespread Panic", the value of the concert field is added it to the concertList select list. If the conditional field name is not provided, then buildList adds the requested field for all records.

Initially, the artist list is populated, the concert list has one entry telling the user to select an artist, and the song list has one entry telling the user to select a concert. Once the user selects an artist, the concerts by that artist appear in the concert list. When the user selects a concert, the songs from that concert appear in the songs list. When the user selects a song, the song information is displayed in the lower text fields.

These text fields all have the same handler:

onFocus="blur(  )"

This handler essentially makes the text fields uneditable by the user. As soon as the user tries to click or tab to one of the fields, the cursor immediately leaves the field. This serves no purpose other than to indicate that these fields are not intended for user input. If the user is fast enough, it is actually possible to add text to these fields, but it won't affect anything. These fields are populated by the showSongInfo function. This function looks through the data to determine which song has been selected and then loads the information for this field into the text fields and also sets the hidden filename field.

When the user clicks on the Download Song button, its onClick handler calls the getSong function. getSong verifies that a song has been selected by checking the value of the filename field If no song has been selected, the user is notified. Otherwise, the requested song is downloaded in another window.

Let's look at the CGI script now. Our CGI script must read the data file, parse it into a WDDX::Recordset object, and add it as JavaScript to our template. The code appears in Example 7-5.

Example 7-5. music_browser.cgi

#!/usr/bin/perl -wT

use strict;
use WDDX;
use HTML::Template;

use constant DATA_FILE => "/usr/local/apache/data/music/song_data.txt";
use constant TEMPLATE  => "/usr/local/apache/templates/music/music_browser.tmpl";

print "Content-type: text/html\n\n";

my $wddx = new WDDX;
my $rec = build_recordset( $wddx, DATA_FILE );

# Create JavaScript code assigning recordset to variable named "data"
my $js_rec = $rec->as_javascript( "data" );

# Output, replacing song_data template var with the JavaScript code
my $tmpl = new HTML::Template( filename => TEMPLATE );
$tmpl->param( song_data => $js_rec );
print $tmpl->output;


# Takes WDDX object and file path; returns WDDX::Recordset object
sub build_recordset {
    my( $wddx, $file ) = @_;
    local *FILE;
    
    # Open file and read field names from first line
    open FILE, $file or die "Cannot open $file: $!";
    my $headings = <FILE>;
    chomp $headings;
    my @field_names = split /\t/, lc $headings;
    
    # Make each field a string
    my @types = map "string", @field_names;
    my $rec = $wddx->recordset( \@field_names, \@types );
    
    # Add each record to our recordset
    while (<FILE>) {
        	chomp;
        my @fields = split /\t/;
        $rec->add_row( \@fields );
    }
    
    close FILE;
    return $rec;
}

This CGI script starts like our previous examples: it adds the modules we need, defines constants to the files it uses, and outputs the HTTP header. Next, it creates a new WDDX object and constructs a recordset via the build_recordset function.

The build_recordset function takes a WDDX object and a file path. It opens the file and reads the first line into $headings to determine the names of the fields. It then splits these into an array, making sure that each field name is lowercase. The next line is a little more complex:

my @types = map "string", @field_names;

WDDX needs to know the data type for each field in the recordset. In this instance, we can treat each field as a string, so this script uses Perl's map function to create an array the same size as @field_names with every element set to "string" and assign it to @types. It then gets a new WDDX::Recordset object and loops through the file, adding each line to the recordset.

We then convert the recordset into JavaScript code and parse this into the template, replacing the song_data tag. That JavaScript code we discussed earlier takes over from WDDX there.