17.2.1 LWP::RobotUA sections

The Robot User Agent (LWP::RobotUA) is a subclass of LWP::UserAgent, and is used to create robot client applications. A robot application requests resources in an automated fashion. Robots perform such activities as searching, mirroring, and surveying. Some robots collect statistics, while others wander the Web and summarize their findings for a search engine.

The LWP::RobotUA module defines methods to help program robot applications and observes the Robot Exclusion Standards, which web server administrators can define on their web site to keep robots away from certain (or all) areas of the site.

The constructor for an LWP::RobotUA object looks like this:

$rob = LWP::RobotUA->new(
agent_name
, 
email
, [$
rules
]);

The first parameter, agent_name , is the user agent identifier that is used for the value of the User-Agent header in the request. The second parameter is the email address of the person using the robot, and the optional third parameter is a reference to a WWW::RobotRules object, which is used to store the robot rules for a server. If you omit the third parameter, the LWP::RobotUA module requests the robots.txt file from every server it contacts, and then generates its own WWW::RobotRules object.

Since LWP::RobotUA is a subclass of LWP::UserAgent, the LWP::UserAgent methods are used to perform the basic client activities. The following methods are defined by LWP::RobotUA for robot-related functionality:

• as_string
• delay
• host_wait
• no_visits
• rules

17.2.2 LWP::Simple

LWP::Simple provides an easy-to-use interface for creating a web client, although it is only capable of performing basic retrieving functions. An object constructor is not used for this class; it defines functions to retrieve information from a specified URL and interpret the status codes from the requests.

This module isn't named Simple for nothing. The following lines show how to use it to get a web page and save it to a file:

use LWP::Simple;

$homepage = 'oreilly_com.html';
$status = getstore('http://www.oreilly.com/', $homepage);
print("hooray") if is_success($status);

The retrieving functions get and head return the URL's contents and header contents respectively. The other retrieving functions return the HTTP status code of the request. The status codes are returned as the constants from the HTTP::Status module, which is also where the is_success and is_failure methods are obtained. See Section 17.3.4, "HTTP::Status " later in this chapter for a listing of the response codes.

The user-agent identifier produced by LWP::Simple is LWP::Simple /n.nn , where n.nn is the version number of LWP being used.

The following list describes the functions exported by LWP::Simple:

• get
• getprint
• getstore
• head
• is_error
• is_success
• mirror

17.2.3 LWP::UserAgent

Requests over the network are performed with LWP::UserAgent objects. To create an LWP::UserAgent object, use:

$ua = new LWP::UserAgent;

You give the object a request, which it uses to contact the server, and the information you requested is returned. The most often used method in this module is request , which contacts a server and returns the result of your query. Other methods in this module change the way request behaves. You can change the timeout value, customize the value of the User-Agent header, or use a proxy server.

The following methods are supplied by LWP::UserAgent:

• request
• agent
• clone
• cookie_jar
• credentials
• env_proxy
• from
• get_basic_credentials
• is_protocol_supported
• max_size
• mirror
• no_proxy
• parse_head
• proxy
• timeout
• use_alarm