20.18.1. Problem

20.18.2. Solution

# story is everything from <!-- story --> to <!-- /story -->
if ($html =~ m{<!-- story -->(.*?)<!-- /story -->}s) {
  my $story = $1;
  # ...
} else {
  warn "No story found in the page";
}

But for tables and data identifiable only by complex patterns of HTML, use a parser:

use HTML::TokeParser;

my $parser = HTML::TokeParser->new($FILENAME)
    or die "Can't open $FILENAME: $!\n";
while (my $token = $parser->get_token( )) {
    my $type = $token->[0];
    if    ($type eq 'S')  { ... }   # start tag
    elsif ($type eq 'E')  { ... }   # end tag
    elsif ($type eq 'T')  { ... }   # text
    elsif ($type eq 'C')  { ... }   # comment
    elsif ($type eq 'D')  { ... }   # declaration
    elsif ($type eq 'PI') { ... }   # processing instruction
    else { die "$type isn't a valid HTML token type" }
}

20.18.3. Discussion

Regular expressions are a convenient way to extract information from HTML. However, as the complexity of the HTML and the amount of information to be extracted go up, the maintainability of the regular expressions goes down. For a few well-defined fields, regular expressions are fine. For anything else, use a proper parser.

The argument to the new constructor of HTML::TokeParser is either a filename, a filehandle (or any object providing a read method), or a reference to the HTML text to be parsed:

$parser = HTML::TokeParser->new("foo.html") or die;
$parser = HTML::TokeParser->new(*STDIN) or die;
$parser = HTML::TokeParser->new(\$html) or die;

Starting tags have four more values in the token array: the tag name (in lowercase), a reference to a hash of attributes (lowercased attribute name as key), a reference to an array containing lowercased attribute names in the order they appeared in the tag, and a string containing the opening tag as it appeared in the text of the document. Parsing the following HTML:

<IMg SRc="/perl6.jpg" ALT="Steroidal Camel">

creates a token like this:

[ 'S',
  'img',
  { "src" => "/perl6.jpg",
    "alt" => "Steroidal Camel"
  },
  [ "src", "alt" ],
  '<IMg SRc="/perl6.jpg" ALT="Steroidal Camel">'
]

Since ending tags have fewer possibilities than opening tags, it follows that their tokens have a simpler structure. A token for an end tag contains "E" (identifying it as an end tag), the lowercased name of the tag being closed (e.g., "body"), and the tag as it appeared in the source (e.g., "</BODY>").

A token for a text tag has three values: "T" (to identify it as a text token), the text, and a flag identifying whether you need to decode entities on it (decode only if this flag is false).

use HTML::Entities qw(decode_entities);

if ($token->[0] eq "T") {
    $text = $token->[1];
    decode_entities($text) unless $token->[2];
    # do something with $text
}

Even simpler, a comment token contains only "C" (to indicate that it is a comment) followed by the comment text.

For a detailed introduction to parsing with tokens, see Perl & LWP by Sean Burke (O'Reilly).

20.18.4. See Also