Tokenizing Walkthrough (Perl & LWP)

So far, I've been showing examples of data in a particular format, then presenting code for extracting the data out of that format, as an illustration of newly introduced HTML::TokeParser methods. But in real life, you do not proceed tidily from the problem to an immediate and fully formed solution. And ideally, the task of data extraction is simple: identify patterns surrounding the data you're after and write a program that matches those patterns and extracts the embedded data.

In practice, however, you write programs bit by bit and in fits and starts, and with data extraction specifically; this involves a good amount of trying one pattern, finding that its matching is too narrow or too broad, trying to amend it, possibly having to backtrack and try another pattern, and so on. Moreover, even equally effective patterns are not equal; some patterns are easier to capture in code than others, and some patterns are more temporary than others.

In this section, I'll try to make these points by walking though the implementation of a data extraction task, with all alternatives considered, and even a misstep or two.

8.1. The Problem

As a starting point, consider the task of harvesting a month's worth of listings and corresponding RealAudio URLs from the web site of the National Public Radio program Fresh Air, at http://freshair.npr.org. Fresh Air is on NPR stations each weekday, and on every show, different guests are interviewed. The show's web site lists which guests appear on the show each day and has links to the RealAudio files for each segment of each show. If your particular weekday schedule doesn't have you listening to Fresh Air every night or afternoon, you would find it useful to have a program tell you who had been on in the past month, so you could make a point of listening to the RealAudio files for the guests you find interesting. Such a data-extraction program could be scheduled with crontab to run on the first or second day of every month, to harvest the past month's program data.

Chapter 8. Tokenizing Walkthrough

Contents:

8.1. The Problem