Adding Extra Request Header Lines (Perl & LWP)

11.2.1. Pretending to Be Netscape

For example, suppose you're looking at http://www.expreszo.nl/home.php and you see that it has interesting headlines. You'd like to write a headline detector for this site to go with the other headline detectors we've been producing throughout the book. You look at the source in Netscape and see that each headline link looks like this:

<A class=pink href="headlines.php?id=749">...text...</A>

So you write something quite simple to capture those links:

use strict;
use warnings;
use LWP;
my $browser = LWP::UserAgent->new;
 
my $url = 'http://www.expreszo.nl/home.php';
my $response = $browser->get($url);
die "Can't get $url: ", $response->status_line
 unless $response->is_success;
$_ = $response->content;
my %seen;
while( m{href="(headlines.php[^"]+)">(.*?)</A>}sg ) {
  my $this = URI->new_abs($1,$response->base);
  print "$this\n  $2\n" unless $seen{$this}++;
}
print "NO HEADLINES?!  Source:\n", $response->content unless keys %seen;

And you run it, and it quite stubbornly says:

NO HEADLINES?!  Source:
<html><body>
...
Je hebt minimaal Microsoft Internet Explorer versie 4 of hoger, of
Netscape Navigator versie 4 of hoger nodig om deze site te bekijken.
...
</body></html>

That is, "you need MSIE 4 or higher, or Netscape 4 or higher, to view this site." It seems to be checking the User-Agent string of whatever browser visits the site and throwing a fit unless it's MSIE or Netscape! This is easily simulated, by adding this line right after $browser is created:

$browser->agent('Mozilla/4.76 [en] (Win98; U)');

With that one small change, the server sends the same page you saw in Netscape, and the headline extractor happily sees the headlines, and everything works:

http://www.expreszo.nl/headlines.php?id=752 Meer syfilis en HIV bij homo's http://www.expreszo.nl/headlines.php?id=751 Imam hangt geldboete van 1200 boven het hoofd http://www.expreszo.nl/headlines.php?id=740 SGP wil homohuwelijk terugdraaien http://www.expreszo.nl/headlines.php?id=750 Gays en moslims worden vaak gediscrimineerd http://www.expreszo.nl/headlines.php?id=749 Elton's gaydar rinkelt bij bruidegom Minnelli http://www.expreszo.nl/headlines.php?id=746 Lekkertje Drew Barrymore liever met een vrouw?

This approach works fine when the web site is looking only at the User-Agent line, as you can most easily control it with $browser->agent(...). If you were dealing with some other site that insisted on seeing even more Netscape-like headers, that could be done, too:

my @netscape_like_headers = (
  'User-Agent' => 'Mozilla/4.76 [en] (Win98; U)',
  'Accept-Language' => 'en-US',
  'Accept-Charset' => 'iso-8859-1,*,utf-8',
  'Accept-Encoding' => 'gzip',
  'Accept' =>
   "image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*",
);
my $response = $browser->get($url, @netscape_like_headers);

11.2.2. Referer

For some sites, that's not enough: they want to see that your Referer header value is something they consider appropriate. A Referer header line signals the URL of a page that either linked to the item you're requesting (as with <a href="url">) or inlines that image item (as with <img src="url">).

For example, I am a big fan of the comic strip Dennis The Menace. I find it to be the truest realization of deep satire, and I admire how its quality has kept up over the past 50 years, quite undeterred by the retirement and eventual death of its auteur, the comic genius Hank Ketcham. And nothing brightens my day more than laughing over the day's Dennis The Menace strip and hardcopying a really good one now and then, so I can pin it up on my office door to amuse my colleagues and to encourage them to visit the DTM web site. However, the server for the strip's image files doesn't want it to be inlined on pages that aren't authorized to do so, so they check the Referer line. Unfortunately, they have forgotten to allow for when there is no Referer line at all, such as happens when I try to hardcopy the day's image file using my browser. But LWP comes to the rescue:

my $response = $browser->get(
  # The URL of the image:
  'http://pst.rbma.com/content/Dennis_The_Menace',

  'Referer' =>   # The URL where I see the strip:

  'http://www.sfgate.com/cgi-bin/article.cgi?file=/comics/Dennis_The_Menace.dtl',
);
open(OUT, ">today_dennis.gif") || die $!;
binmode(OUT);
print OUT $response->content;
close(OUT);

By giving a Referer value that passes the image server's test for a good URL, I get to make a local copy of the image, which I can then print out and put on my office door.

11.2. Adding Extra Request Header Lines

11.2.1. Pretending to Be Netscape

11.2.2. Referer