Extracting or Removing HTML Tags (Perl Cookbook, 2nd Edition)

20.6.3. Discussion

As with almost everything else in Perl, there is more than one way to do it. Each solution attempts to strike a balance between speed and flexibility. Occasionally you may find HTML that's simple enough that a trivial command-line call works:

% perl -pe 's/<[^>]*>//g' file

However, this breaks with files whose tags cross line boundaries, like this:

<IMG SRC = "foo.gif"
     ALT = "Flurp!">

So, you'll see people doing this instead:

% perl -0777 -pe 's/<[^>]*>//gs' file

or its scripted equivalent:

{
    local $/;               # temporary whole-file input mode
    $html = <FILE>;
    $html =~ s/<[^>]*>//gs;
}

But even that isn't good enough except for simplistic HTML without any interesting bits in it. This approach fails for the following examples of valid HTML (among many others):

<IMG SRC = "foo.gif" ALT = "A > B">

<!-- <A comment> -->

<script>if (a<b && a>c)</script>

<# Just data #>

<![INCLUDE CDATA [ >>>>>>>>>>>> ]]>

If HTML comments include other tags, those solutions would also break on text like this:

<!-- This section commented out.
    <B>You can't see me!</B>
-->

The only solution that works well here is to use the HTML parsing routines from CPAN. The second code snippet shown in the Solution demonstrates this better technique.

For more flexible parsing, subclass the HTML::Parser class and record only the text elements you see:

package MyParser;
use HTML::Parser;
use HTML::Entities qw(decode_entities);

@ISA = qw(HTML::Parser);

sub text {
    my($self, $text) = @_;
    print decode_entities($text);
}

package main;
MyParser->new->parse_file(*F);

If you're only interested in simple tags that don't contain others nested inside, you can often make do with an approach like the following, which extracts the title from a non-tricky HTML document:

($title) = ($html =~ m#<TITLE>\s*(.*?)\s*</TITLE>#is);

Again, the regex approach has its flaws, so a more complete solution using LWP to process the HTML is shown in Example 20-4.

Example 20-4. htitle

  #!/usr/bin/perl
  # htitle - get html title from URL
  use LWP;
  die "usage: $0 url ...\n" unless @ARGV;
  foreach $url (@ARGV) {
      $ua = LWP::UserAgent->new( );
      $res = $ua->get($url);
      print "$url: " if @ARGV > 1;
      if ($res->is_success) {
          print $res->title, "\n";
      } else {
          print $res->status_line, "\n";
      }
  }

Here's an output example:

% htitle http://www.ora.com
www.oreilly.com -- Welcome to O'Reilly & Associates!

% htitle http://www.perl.com/ http://www.perl.com/nullvoid
http://www.perl.com/: The www.perl.com Home Page
http://www.perl.com/nullvoid: 404 File Not Found

Example 20-4. htitle

20.6. Extracting or Removing HTML Tags

20.6.1. Problem

20.6.2. Solution

20.6.3. Discussion

20.6.4. See Also