Usenet News (Perl in a Nutshell, 2nd Edition)

Usenet is a collection of bulletin board-like newsgroups on the Internet, covering thousands of topics. Whatever your interest, chances are you'll find a newsgroup in which it is discussed .

Usenet has been around since late 1979. The current implementation is based on the Network NewsTransfer Protocol (NNTP), defined in RFC 977 and released in March 1986. Information is propagated through Usenet by a system of newsfeeds in which one site requests a newsfeed from another site, and a third site requests a newsfeed from the second site, etc. There is no central Usenet authority—like Perl, it runs on the spirit of cooperation and sharing. When you run a news reader, such as tin or the news reader in a web browser, your client software talks to the NNTP server on the news host. When you post a message to a newsgroup, this posting is received by your NNTP server and passed on to other servers throughout the distribution area you specified for the posting. Each server periodically receives updated newsgroup information and newly posted news articles.

This chapter explores NNTP commands and responses. It introduces Net::NNTP, which implements NNTP commands and simplifies the process of writing a Perl-based NNTP news client. It also describes News::Newsrc, a module that provides methods for managing a .newsrc file.

There are two kinds of NNTP commands: the official set of commands as defined in RFC 977 and a number of extensions that have been added since the RFC was written. The extensions are described in an IETF Internet Draft document, "Common NNTP Extensions" by Stan Barber, which can be found at ftp://ftp.academ.com/pub/nntp/ietf/nntpext.txt.

17.1. The NNTP Protocol

Before you write your own news client, you should have some idea of how the NNTP protocol works. Like other servers, an NNTP server is bound to a port (usually port 119). It listens for incoming connections, takes the appropriate action, and returns a response to the client. When a news client connects with an NNTP server, or to the port on which the NNTP server is running, a message such as the following is produced:

Trying 0.0.0.0...
Connected to hostname.mydomain.com.
Escape character is '^]'.
200 newshost.mydomain.com InterNetNews NNRP server INN 1.5.1 17-Dec-1996 ready
(posting ok).

Many NNTP servers understand the help (or HELP) command. When a client issues a help command, many NNTP servers respond with a list of all available commands. For example:

200 news.mydomain.com InterNetNews NNRP server INN 1.5.1 17-Dec-1996 ready
(posting ok).
HELP
100 Legal commands
  authinfo user Name|pass Password|generic <prog> <args>
  article [MessageID|Number]
  body [MessageID|Number]
  date
  group newsgroup
  head [MessageID|Number]
  help
  ihave
  last
  list [active|active.times|newsgroups|distributions|distrib.pats|
overview.fmt|subscriptions]
  listgroup newsgroup
  mode reader
  newgroups yymmdd hhmmss ["GMT"] [<distributions>]
  newnews newsgroups yymmdd hhmmss ["GMT"] [<distributions>]
  next
  post
  slave
  stat [MessageID|Number]
  xgtitle [group_pattern]
  xhdr header [range|MessageID]
  xover [range]
  xpat header range|MessageID pat [morepat...]
  xpath MessageID
Report problems to <usenet@news.mydomain.com>
.

After connecting to the NNTP server, you can get a list of available newsgroups with the list active command, and you can then select a newsgroup. If you were to connect directly to news.mydomain.com and use the group command to select the newsgroup local.test, your session might look like this:

200 news.mydomain.com InterNetNews NNRP server INN 1.5.1 17-Dec-1996 ready
(posting ok).
group local.test
211 4 1 4 local.test
QUIT
205 .

The four numbers (in this example, 211 4 1 4) preceding the group name (local.test) represent the success code, total number of articles, and the first and last article numbers, respectively. So in this example, 211 is the success code. There are four articles in the local.test group, starting with article number 1 and ending with number 4.

Every article in a newsgroup has two identifiers associated with it. The first is the message number, and the second is a message ID. The message number is the number of that article on the server to which you are connecting. The message ID is an identifier that is always associated with the article on every server that receives the article. An article that has the message number 4 on your server might be number 83 on someone else's server. The message ID, however, will be the same on both. So, for example, if you want to post an article that refers to another article, use the message ID.

Once the group command has successfully identified a newsgroup, the client can request a particular article by sending an article request to the server. The article command takes either a message ID or a message number as an argument. For example:

article 4
Path: newshost.mydomain.com!news-w.ans.net!newsfeeds.ans.net!philabs ...
From: user@mydomain.com (User Name)
Newsgroups: local.test
Subject: Is anybody there?
Date: 21 Apr 1997
...

If the article command is successful, the server returns the selected message.

The xhdr command returns the selected headers ("X-headers") from articles in a certain newsgroup. The X-headers contain all the information about the news article, including the poster's email address, the subject of the message, the date and time the message was posted, the newsgroup(s) the message appears in, and the message ID. For example, if you want to see the subjects of the articles in the local.test newsgroup, you can run xhdr subject 1-4, as follows:

xhdr subject 1-4
221 subject fields follow
1   Is anybody there?
2   Re: Is anybody there?
3   Re: Is anybody there?
4   Get a life!

The 221 status code indicates that the xhdr completed successfully and returned the requested headers. A Usenet client might display these headers to the user, who can then select the articles to read based on the subject lines.

A command similar to xhdr is xpat, which matches X-headers against a pattern string you give it. For example, executing xpat subject 1-9 *anybody* for local.test might look like this:

xpat subject 1-9 *anybody*
221 subject matches follow.
1   Is anybody there?
2   Re: Is anybody there?
3   Re: Is anybody there?

You can also post news articles with NNTP. When you issue the post command, the server returns a code of 340 if posting is permitted. If you get an error, such as 501, you cannot post articles with this command. Most servers require that you include Newsgroups, Subject, and From headers. You will receive a 240 reply if your message has been successfully posted. Here's an example of posting to local.test using NNTP:

200 news.mydomain.com InterNetNews NNRP server INN 1.5.1 17-Dec-1996 ready
(posting ok).
post
340 Ok
Newsgroups: local.test
Subject: talking NNTP
From: user@mydomain.com
This is a test posting.       
.
240 Article posted
QUIT
205 .

The server returns a code of 205 upon disconnecting. When the article arrives in local.test, it looks like this:

From news.mydomain.com!not-for-mail Sun Mar  9 19:18:37 1997
Path: news.mydomain.com!not-for-mail
From: user@mydomain.com
Newsgroups: local.test
Subject: talking NNTP
Date: 10 Mar 1997 00:09:33 GMT
Organization: Perl in a Nutshell
Lines: 2
Distribution: local
Message-ID: <5fvjft$k3$2@news.mydomain.com>
NNTP-Posting-Host: newshost.mydomain.com
Xref: newshost.mydomain.com local.test:6

This is a test posting.

Chapter 17. Usenet News

Contents:

17.1. The NNTP Protocol