20.3.1.1 Using this API

Perhaps your eyes are somewhat glazed and your mind is numbed, so we will relieve the tedium by writing a custom interpreter using the API we have seen so far. (For now, this is our idea of fun!) Example 20.2 shows a function called create_envt_vars that creates a scalar variable for every environment variable.

#include <EXTERN.h>
#include <perl.h>
void create_envt_vars (char **environ)
{
    /* 
     * Each element in environ is in the form <envt. var name>=<value>"
     */
    SV * sv = NULL;
    char **env = environ; /* for iterating through environ */
    char buf[1000];       /* will contain a copy of an envt variable */
    char *envt_var_name;  /* Name of the envt. variable, like PATH */
    char *envt_var_value; /* Its corresponding value */
    char var_name[100];   /* Fully qualified name of environment var */
    while (*env) {
        strcpy (buf, *env);
        /* Search for "=", replace it with '\0', thus splitting it into
         *  logical parts - envt variable name and the value 
         */
        envt_var_name = buf; envt_var_value = buf;
        while (*envt_var_value != '=') envt_var_value++;
        *envt_var_value++ = '\0';
        /* Qualify the environment var with the package name. 
         * PATH becomes $main::PATH 
         */
        strcpy (var_name, "main::"); strcat(var_name, envt_var_name);
        sv = perl_get_sv (var_name, TRUE); /* TRUE => Force Create */
        /* Set the string value for the sv);
        sv_setpv(sv, envt_var_value);
        env++;  /* On to the next environ variable */
    }
}

static PerlInterpreter *my_perl;  
main(int argc, char **argv, char **env) {
    my_perl = perl_alloc();
    perl_construct(my_perl);
    perl_parse(my_perl, NULL, argc, argv, env);
    create_envt_vars();
    perl_run(my_perl);
    perl_destruct(my_perl);
    perl_free(my_perl);
}

On a DEC Alpha box, you might compile and link it as follows:

% cc -o ex  -I/usr/local/lib/perl5/alpha-dec_osf/5.004/CORE \


            -L/usr/local/lib/perl5/alpha-dec_osf/5.004/CORE \


             ex.c -lperl -lsocket -lm

Now for the big test:

% ./ex -e 'print $USER'

sriram

Amazing, it works - try doing that with your regular Perl! Okay, so it's not a big deal, but you are definitely on your way to dirtying your hands more than you thought you could. Or would!

20.3.1.2 Inside SVs

An SV has the potential of being large, to accommodate the worst case of it morphing to any one of its subtypes. To avoid this, Perl keeps track of the information in two parts, as illustrated in Figure 20.3 : a generic structure called " sv ", which contains a bitmask flag, a reference count, and a pointer, sv_any , which refers to a "specific part."

Figure 20.3: An inside view of scalars. Each shaded box represents one SV.

The specific part is a structure of the appropriate type and is one of several structures called xpv , xpviv , xpvnv , and so on, depending on what the bitmask flag says is contained by the scalar. A scalar may start life as a number, but the moment it is used in a string context, it morphs to a structure containing both the number and a string. Figure 20.3 shows an example of the SV (the middle one) containing a double and a string at the same time. If you modify its value with, say, sv_setnv , it sets a bit flag in sv_flags to indicate that the string part of it is not valid any more. Perl doesn't morph a structure unless absolutely necessary.

The Devel::Peek module gives you script-level access to the internal information discussed previously:

use Devel::Peek 'Dump';  # Import the Dump subroutine
$a = 15602;
Dump ($a);

This outputs

SV = IV(0x663f1c)
  REFCNT = 1
  FLAGS = (IOK,pIOK)
  IV = 15602

Modify $a to a string and see what happens to it:

use Devel::Peek 'Dump';
$a = 10;           # Start off with an integer value
$a .= " Ten";      # Convert to a string
Dump ($a);

Its output is as follows:

SV = PVIV(0x7b2ef0)
  REFCNT = 1
  FLAGS = (POK,pPOK)
  IV = 10
  PV = 0x7b2f00 "10 Ten"
  CUR = 6
  LEN = 11

Note that the SV still contains the old integer value (10) but that field will be ignored because the FLAGS field indicates that only its string contents are valid.

A basic integer value costs you at least 28 bytes on a typical workstation ( sizeof(SV) + sizeof(XPVIV) + malloc overhead). Strings and arrays are more expensive than what you might infer from the length or count alone. The dump above shows that Perl allocated 11 bytes for the string (the xpv_len field) instead of the minimum 6 bytes (the length of the string, as stored in the xpv_cur field); this means that you can append 5 more bytes to the string without triggering a reallocation. Because it is geared for performance and convenience (being able to treat numbers and strings as one entity), it is not frugal with space at all. In fact, Perl applies this strategy of trading off space for performance for practically all data structures.[ 6 ]

[6] One notable exception being hashes, which share their key strings in one string table, thus minimizing space requirements but taking a small hit in performance.

There are no simple xiv or xnv structures that contain just an integer or a floating-point number. I do not know the reason for this, and idly speculate that it's so because a typical script requires numbers to morph into strings (while doing a print , for example) or vice versa (reading from files).

Figure 20.3 also shows the reference counts of the three scalars. The reference count of the middle scalar is 2 because of two arrows pointing to it; the arrows coming from the left imply a symbol table entry (for global and local variables) or a scratchpad for lexical variables, as we saw in Chapter 3 . Note that all pointers to an SV (to any Perl value, actually) refer to the outer structure, never to the "specific" part.

20.3.1.3 SVs and object pointers

As an extension writer, you are often interested in storing a pointer to a C or C++ object returned from an XSUB. Recall that the integer (IV) slot of a scalar is guaranteed to be big enough to hold a pointer. We use this facility as follows:

Matrix *m = new_matrix();
sv_setiv(sv, (IV) m);          # Cast pointer to an IV.  Ugh!

Crufty, to say the least, but that's the way it is.

In practice, C/C++ objects are always associated with blessed references because it allows the Perl programmer to use the arrow notation ( $matrix->transpose() ). Consider

RV *rv = newRV();
sv_setref_iv(rv, "Matrix", (IV) m);

This creates a new integer SV internally, sets it to the "integer" m , and makes rv point to this freshly allocated SV. It also blesses rv under the module Matrix. This is exactly as if you had said, in Perl space,

my $m = 0xfffa34a;     # Some pointer value, converted to an int
bless \$m, "Matrix";   # Return a blessed reference to $m.

We will use this snippet when discussing typemaps for objects, in the section "Object Interface Using XS Typemaps ."

20.3.2.1 Inside AVs

AVs, like SVs, are split into a generic part and a specific part. As it happens, this is true for the other value types also.

As shown in Figure 20.4 , the xav_alloc field points to a dynamically allocated array of SV*s, the real meat of the AV. av_fill contains the last valid (or filled) index in this array, and av_max contains the total number of SV*s allocated for the array. Perl always strives to make sure it allocates memory in some "reasonable" quantities so that it doesn't have to realloc every time you push an element into this array. xav_array points to the first valid element. It starts off by pointing to xav_alloc[0] , and gets incremented on an unshift to avoid having to move the rest of the elements to the left. In other words, the real contents of the AV are bounded by xav_array and av_fill .

Figure 20.4: AV internal structure. The shaded squares contain real data.

The xmg_magic pointer is typically NULL but refers to a "magic" structure if the array is special (such as @ISA) , represents a blessed object, or is tied to a package.[ 7 ] xav_arylen is an SV* that starts off being NULL but springs into existence as a magical scalar when you use the $# notation on the array (to get or set the array length).

[7] You don't have to understand this paragraph until we have covered magical variables.

Devel::Dump gives you script-level access to the internal details of an array and of all its constituent scalars. Dump expects nonscalar values to be passed by reference:

use Devel::Dump;
@l = (1,2,3,4);
Dump(\@l);  # Pass @l by reference

20.3.3.1 Inside HVs

The HV is a straightforward implementation of a hashing technique called collision chaining . The basic idea is to reduce a string key to an integer and use this number as an index into an ordinary dynamic array. Clearly, we cannot expect to reduce all possible string keys to unique array indices, so each element of this dynamic array points instead to a linked list of all hash entries that reduced to that index. Figure 20.5 shows this arrangement.

Figure 20.5: Hash value, containing hash entries

xhv_array is the dynamic array we mentioned above, xhv_fill indicates the number of elements that have linked lists hanging off them, and xhv_keys contains the total number of hash entries. Given a string, hv_fetch() computes the appropriate index and traverses the corresponding linked list, comparing the key to each hash entry's key value.

The translation of a string to an array index is a two-step process (for reasons to be discussed soon). First the string is run through an algorithm called a hash function , which computes an integer from a string without worrying about whether or not that number is practical to use as an array index. Perl's hash function is implemented as follows:

int i = klen;
unsigned int hash = 0;
char *s = key;
while (i--)
    hash = hash * 33 + *s++;

The resulting number is known as a hash . There is no guarantee that different strings hash to different hash values. Note that if you have a different hash algorithm, you can compute the hash yourself and supply it to hv_store (refer to Table 20.3 ).

To convert the hash to a practical array index, Perl folds it into the array's maximum size:

index = hash & xhv_max;

Ideally, we want the entries to spread out evenly around the array to keep the linked lists short. Perl's hashing algorithm does this surprisingly well for typical ASCII strings, but as we mentioned earlier, there's no guarantee that a given set of strings disperses well. So if xhv_keys exceeds xhv_fill , Perl takes it as an indication that one or more linked lists are unnecessarily long and that hv_fetch is likely to spend a considerable amount of time traversing these lists. Therefore, when such a condition occurs, Perl immediately reorganizes the hash table: the dynamic array xhv_array is doubled in size, and the entries are reindexed. Each hash entry stores its hash value, so it doesn't have to be hashed again when the table is reorganized.

You can get an idea of the hash efficiency by printing an associative array in a scalar context, as follows:

# Create a hash
for (1 .. 1000) {$h{'foo' . $_} = 1;} # Create 1000 entries
print scalar(%h);

This prints "406/1024" on my machine, which is simply a ratio of xhv_fill and xhv_max . The lower the ratio, the faster the hash access, because, on average, the linked lists are short.

If you know you are going to be performing a large number of insert operations on a hash, you can improve its efficiency in script space by telling it to preallocate a certain-sized dynamic array, like this:

keys %h = 400; # set xhv_max

Perl rounds it up to the next higher power of two: 512.

The xhv_riter and xhv_eiter fields are used by the iterator functions, hv_iterinit and hv_iternextsv , and constitute a cursor over the hash entries. xhv_riter contains the current row index, and xhv_eiter contains the pointer to the current entry.

Most object-oriented Perl implementations use hash tables for storing object attributes, which means that all instances of a given class would typically have the same set of key strings. To prevent unnecessary duplication, the actual key strings are maintained in a systemwide shared string table ( strtab in strtab.h ). strtab is a simplified HV: each value here keeps a reference count of the number of uses of that string. When you say " $h{'foo'} ", the string foo is first entered into strtab if it is not already present. Then the hash entry for $h{foo} is created in the HV for %h . It turns out that performance suffers very little; if there are lots of duplicates, shared storage saves time because the key is malloc 'ed only once. Also, since the hash algorithm needs to be executed only once, the performance is pretty good even when there aren't too many duplicates.

The shared string table is used only for immutable strings (remember that hash key strings cannot be changed). User-defined SVs containing strings do not get to use this table.

20.3.4.1 Inside glob values and symbol tables

Figure 20.6 shows most of the interesting components of a GV.

Figure 20.6: Glob value structure

The xgv_name field stores the name of the variable (without the prefix). The pointers to the contained values ( $foo , @foo , and so on) are encapsulated in a separate structure called gp to enable fast aliasing. In the typical case in which you don't reuse the same name for different types of variables, all but one of the value pointers are NULL .

Symbol tables are HVs that map variable names to GVs. But aren't HVs supposed to store SVs only? Well, you may have noticed that all value types have identical- looking wrapper structures that maintain the reference count, flags, and the pointer to an internal structure. Because they are identical, you can cast an AV* , HV* , CV* to an SV* and thus fool the HV into storing anything you want. If you were to do this yourself, be careful of the HV calls that decrement the reference count of a contained "SV" ( hv_delete , for example). This is because they will trigger an sv_free() if the reference count becomes 0, and if it is not an SV, you are in trouble.

20.3.5.1 Inside CV

A CV has the same overall structure as the other value types: a generic part and a specific part. Consider the following piece of code, which defines a function in another package (by fully qualifying the name) and examines the function using Devel::Peek:

package Foo;
sub main::bar {   #Introduce a function in a different package
   my $a = 10;
}
use Devel::Peek;
Dump(\&main::bar);

The dump looks like this:

SV = PVCV(0x774300)
  REFCNT = 2
  FLAGS = ()
  IV = 0
  NV = 0
  COMP_STASH = 0x6635f0 "Foo"
  START = 0x7744d0
  ROOT = 0x774650
  XSUB = 0x0
  XSUBANY = 0
  GVGV::GV = 0x66365c   "main" :: "bar"
  FILEGV = 0x660418     "_<foo.pl"
  DEPTH = 0
  PADLIST = 0x66362c

The COMP_STASH field indicates that the "Foo" stash would be active when bar() executes, although this subroutine is defined in package main . The ROOT field indicates the root opcode of the syntax subtree for the CV, and START is the address of the opcode to get control when the function starts. The XSUB field either is NULL or contains a pointer to a C subroutine. The DEPTH field indicates the depth of recursion, and PADLIST refers to a list of scratchpads for storing lexical variables defined inside that subroutine. More on this next.

20.3.5.2 How local and my work

Perl variables, as we are well aware, can be global, dynamic (tagged with local ), or lexical ( my ). Global variables are accessible via the stash and the corresponding typeglob. When Perl encounters the global variable $a , it produces the opcode gvsv , which places the corresponding GV's scalar value on the stack, at run-time.

When Perl parses " local $a ," it still outputs the same gvsv opcode, but this time it sets a special flag in that opcode to "localize" the scalar. At run-time, the corresponding opcode function pp_gvsv checks this flag and, if it is set, replaces the GV's scalar value with a new scalar value and pushes this new value onto the argument stack. Meanwhile, the old SV sits safely in something called a savestack (discussed later, in the section "Inside Other Stacks "). Subsequent accesses of $a within that scope (or a nested scope) lead you, via a 's GV, to the newly allocated scalar value.

my variables are stored and treated very differently. We mentioned earlier that each CV contains a padlist , a list of scratchpads, as illustrated in Figure 20.7 .

Figure 20.7: Inside look at my variables

The padlist is an ordinary AV. Its 0th element points to an AV containing all lexical variable names used within that subroutine (not just declared within that subroutine). The names contain their prefix symbols, so $a and %a have their own unique entries. The padlist's first element points to a scratchpad array (also an AV), whose elements contain the values corresponding to the lexical variables named in the 0th row. As you can see, the padlist is an alternative symbol table because it contains a logical pairing of variable names and values.

When the subroutine recurses, a new scratchpad is allocated for that recursion level. You'll notice that a CV requires at least three AVs (one for xcv_padlist , one for storing the names, and one or more for storing values).

When multithreading is introduced into Perl (Version 5.005 onward), each thread will get its own scratchpad,[ 9 ] which means that lexical variables will continue to be completely private to a recursion level and thread. (Package global variables will continue to be global, of course.)

[9] This is according to Malcolm Beattie's current prototype patches for supporting POSIX threads.

my variables are a shade faster than local variables. The reason is that local allocates a new value at run-time to temporarily occlude the global value. In contrast, my variables are already unique to a CV, so they are typically allocated once, at parse time. The only reason to create a fresh my value is in case of recursion, which is not a typical occurrence. In future releases of Perl, multiple threads executing the same CV will also require run-time allocation of lexical variables.

When you access a lexical variable, Perl's code generator outputs an opcode called padsv , equivalent to gvsv (which is used for global or local variables). padsv remembers the offset of the variable inside the scratchpad (1 for %a in Figure 20.7 ). At run-time, Perl wastes no time at all fetching the corresponding value and pushing it on the stack.

20.3.5.3 Closures

This brief introduction to CVs and lexical variables leads us to the subject of closures. When a closure is created, Perl allocates a CV, points it to the starting opcode for the subroutine, and supplies it with its own private padlist. The padlist contains pointers to all lexical variables used by that closure, whether or not they were created within that block, as illustrated by Figure 20.8 .

Figure 20.8: Scratchpad picking up lexicals from containing CVs' scratchpads

For those lexicals picked up from the CV containing the closure ( $a in Figure 20.8 ), the scratchpad contains direct pointers to the appropriate values, and the reference count of those values is incremented. Lexicals such as $b , created within the closure, are freshly allocated. Either way, the space allocated for a lexical variable is not deallocated as long as any subroutine using it can be called.

20.3.5.4 Objects versus closures

In Chapter 4, Subroutine References and Closures , we noted the similarity between objects and closures: both represent a binding between code and data. In other words, an object with three member functions can instead be represented as three closures acting on private variables borrowed from their containing environment.

Clearly, the closure approach is a lot more expensive in space; to represent 100 objects' worth of information, you require 300 unique closures, which works out to 900 AVs. In contrast, if you use a hash representation for storing object attributes, you need 100 hash tables and 9 AVs (three per subroutine).

On the other hand, calling a closure is faster than invoking an object's method. This is because a closure's variables are ready to be used as soon as the procedure is called, whereas an object's method has to dereference the object reference and then make a hash access for each attribute. The following benchmark compares the speed of an object accessor method to an equivalent closure - the latter approach is two to three times faster on my PC:

#--------------------------------------------------------------
package OBJECT;                    # Pkg for timing object accessors
sub new {
   bless {'abc' => 10};
}
sub abc {                          # Fetch the abc attribute
    $_[0]->{'abc'};
    }
sub increment {                    # Increment the abc attribute
    $_[0]->{'abc'}++;
}
#--------------------------------------------------------------
package CLOSURE;                   # Pkg for timing closures
sub new {
  my $abc = 10;                    # member data.
  $rs_increment = sub {$abc++};    # equivalent of OBJECT::increment
  $rs_abc       = sub {$abc}  ;    # equivalent of OBJECT::abc
  ($rs_increment, $rs_abc);
}
#--------------------------------------------------------------
package main;
use Benchmark;
$a = OBJECT->new();                # Create a new object
($inc, $fetch) = CLOSURE->new();   # Create two closures
timethese(1000000, {
   Object  =>  '$a->increment',    # call an object method
   Closure =>  '&$inc'             # call a closure
});

On my PC, this prints

Benchmark: timing 1000000 iterations of Closure, Object...
   Closure: 13 secs (14.39 usr  0.00 sys = 14.39 cpu)
    Object: 45 secs (45.14 usr  0.00 sys = 45.14 cpu)