XML::Parser Tutorial

Prev

http://perlmonks.thepen.com/62782.html

by OeufMayo on Mar 07, 2001

Introduction

We all agree that Perl does a really good job when it comes to text extraction, particulary with regular expressions.

The XML is based on text, so one might think that it would be dead easy to take any XML input and have it converted in the way one wants.

Unfortunately, that is wrong. If you think you'll be able to parse a XML file with your own homegrown parser you did overnight, think again, and look at the XML specs closely. It's as complex as the CGI specs, and you'll never want to waste precious time trying to do something that will surely end up wrong anyway. Most of the background discussions on why you have to use CGI.pm instead of your own CGI-parser apply here.

The aim of this tutorial is not to show you how XML should be structured and why you shouldn't parse it by hand but how to use the proper tool to do the right job. I'll focus on the most basic XML module you can find, XML::Parser. It's written by Larry Wall and Clark Cooper, and I'm sure we can trust the former to make good software (rn and patch are his most famous programs)

Okay, enough talk, let's jump into the module!

This tutorial will only show you the basics of XML parsing, using the easiest (IMHO) methods. Please refer to the perldoc XML::Parser for more detailed info. I'm aware that there are a lot of XML tools available, but knowing how to use XML::Parser can surely help you a lot when you don't have any other module to work with, and it also helped me to understand how other XML modules worked, since most of them are built on top of XML::Parser.

The data

The example I'll use for this tutorial is the Perlmonks Chatterbox ticker that some of you may have already used. It looks like this:

<CHATTER><INFO site="http://perlmonks.org[]" sitename="Perl Monks">
Rendered by the Chatterbox XML Ticker</INFO>
    <message author="OeufMayo" time="20010228112952">
test</message>
    <message author="deprecated" time="20010228113142">
pong</message>
    <message author="OeufMayo" time="20010228113153">
/me test again; :)</message>
    <message author="OeufMayo" time="20010228113255">
&lt;a href="#"&gt;please note the use of HTML
tags&lt;/a&gt;</message>
</CHATTER>

The code

Let's assume we want to output this file in a readable way (though it'll still be barebone). It doesn't handles links and internal HTML entities. It only gets the CB ticker, parses it and prints it, you have to launch it again to follow the wise meditations and the brilliant rethoric of the other fine monks present at the moment.

00001: !/usr/bin/perl -w
00002: se strict;
00003: se XML::Parser;
00004: se LWP::Simple;  # used to fetch the chatterbox ticker
00005:
00006: y $message;      # Hashref containing infos on a message
00007:
00008: y $cb_ticker = get("http://perlmonk?s.org/index.pl?node=?chatterbox+xml+ticke?r[]");
00009:  we should really check if it succeeded or not
00010:
00011: my $parser = new XML::Parser ( Handlers => {   # Creates our parser object
00012:                             Start   => \&hdl_start,
00013:                             End     => \&hdl_end,
00014:                             Char    => \&hdl_char,
00015:                             Default => \&hdl_def,
00016:                           });
00017: $parser->parse($cb_t?icker);
00018:
00019: # The Handlers
00020: sub hdl_start{
00021:     my ($p, $elt, %atts) = @_;
00022:     return unless $elt eq 'message';  # We're only interrested in what's said
00023:     $atts{'_str'} = '';
00024:     $message = \%atts;
00025: }
00026:
00027: sub hdl_end{
00028:     my ($p, $elt) = @_;
00029:     format_message($mess?age) if $elt eq 'message' && $message && $message->{'_str'} =~ /\S/;
00030: }
00031:
00032: sub hdl_char {
00033:     my ($p, $str) = @_;
00034:     $message->{'_str'} .= $str;
00035: }
00036:
00037: sub hdl_def { }  # We just throw everything else
00038:
00039: sub format_message { # Helper sub to nicely format what we got from the XML
00040:     my $atts = shift;
00041:     $atts->{'_str'} =~ s/\n//g;
00042:
00043:     my ($y,$m,$d,$h,$n,$s) = $atts->{'time'} =~ m/^(\d{4})(\d{2})(\d?{2})(\d{2})(\d{2})(\?d{2})$/;
00044:
00045:     # Handles the /me
00046:     $atts->{'_str'} = $atts->{'_str'} =~ s/^\/me// ?
00047:     "$atts->{'author'} $atts->{'_str'}"   :
00048:     "<$atts->{'author'}>?: $atts->{'_str'}";
00049:     $atts->{'_str'} = "$h:$n " . $atts->{'_str'};
00050:     print "$atts->{'_str'}\n";
00051:     undef $message;
00052: }

Step-by-step code walkthrough:

Lines 1 to 4

Initialisation of the basics needed for this snippet, XML::Parser, of course, and LWP::Simple to get the chatterbox ticker.

Line 8

LWP::Simple get the requested URL, and put the content of the page in the $cb_ticker scalar.

Lines 11 to 16

The most interesting part, no doubt. We create here a new XML::Parser object. The Parser can come in different styles, but when you have to deal with simple data, like the CB ticker, the Handlers way is the easiest (see also the Subs style, as it is really close to this one).

For this object, we define four handlers subs, each representing a different state in the parsing process.

The 'Start' handler is called whenever a new element (or tag, HTML-wise) is found. The sub given is called with the expat object, the name of the element, and a hash containing all the atrributes of this element.
The 'End' is called whenever an element is closed, and is called with the same parameters as the 'Start', minus the attributes.
The 'Char' handler is called when the parser finds something which is not mark-up (in our case, the text enclosed in the <message> tag).
Finally, the 'Default' handler is called, well, by default, when anything else matching the three other handlers is called.

Line 17

The line that does all the magic, parsing and calling all your subs for you at the right moment.

Lines 20-25: the Start handler

We only want to deal with the <message> elements (those containing what it is being said in the Chatterbox) so we'll happily skip every other element.

We got a hash with the attributes of the element, and we're going to use this hash to store the string that will contain the text to be displayed in the $atts{'_str'}

Lines 27-30: the End handler

Once we've reached the end of a message element, we format all the info we have gathered and prints them via the format_message sub.

Lines 32-35: the Char handler

This sub gets all the strings returned by the parser and appends it to the string to be finally displayed

Line 37: the Default handler

It does nothing, but it doesn't have to figure out what to do with this!

Lines 39-52

This subroutine mangles all the info we got from the XML file, with bad regexes and all, and prints the formatted text in a hopefully readable way. Please note that XML::Parser handled all of the decoding of the < and > entities that were included in the original XML file

Summary

We now have a complete and simple parser, ready to analyse, extract, report everything inside the Chatterbox XML ticker!

That's all for now, here are some links you may find useful:

Most of mirod's nodes (and especially his review of XML::Parser) http://perlmonks.thepen.com/mirod.html
davorg's Data Munging with Perl

subclassing vs. global variables

> This is nice, but I would rather know how to use XML::Parser by subclassing
> it. All of my attempts to do this ended up in very unclean, OOP-unfriendly
> code. I ended up with storing results in package-global variables rather
> than object attributes. This is both ugly and thread-unsafe.
>
> Is there some clean way how to subclass XML::Parser?

The problem is probably that XML::Parser is an object factory: it generates XML::Parser::Expat objects with each parse or parsefile call. The handlers then receive XML::Parser::Expat objects and not XML::Parser objects.

There is a way to store data in the XML::Parser object and to access it in the handlers though: use the 'Non-Expat-Options' argument when creating the XML::Parser:

#!/bin/perl -w
use strict;
use XML::Parser;

my $p= new XML::Parser(
       'Non-Expat-Options' => { my_option => "toto" },
       Handlers => { Start => \&start, }
                     );
$p->parse( '<a />');

sub start
 { my( $pe, $elt, %atts)= @_;
   print "my option: ", $pe->{'Non-Expat-Options'}->{my_option}, "\n";
 }

This is certainly ugly but it works!

Update: note that the data is still stored in the XML::Parser object though, as shown by this code:

#!/bin/perl -w
use strict;
use XML::Parser;

my $p= new XML::Parser(
       'Non-Expat-Options' => { my_option => "1" },
       Handlers => { Start => \&start, }
                     );
$p->parse( '<a />');
$p->parse( '<b />');

sub start
 { my( $pe, $elt, %atts)= @_;
   print "element: $elt - my option: ",
         $pe->{'Non-Expat-Opt?ions'}->{my_option}+?+, "\n";
   $p->parse( '<c />')
      unless( $pe->{'Non-Expat-Opt?ions'}->{my_option} > 3);
 }

Which outputs:

element: a - my option: 1
element: c - my option: 2
element: c - my option: 3
element: b - my option: 4

Comments

Why do you want to subclass it? It works much better as a "has-a" than an "is-a", unless you want to get *very* cozy from the base class implementation, which is a maze of twisty tiny packages all alike.

Just delegate the methods that you want to provide in your interface, and handle the rest. Make a hash with one of the elements being your "inherited" parser. I believe it's called the "wrapper" pattern, but I don't name my patterns — I just use them!

Randal L. Schwartz, Perl hacker

Comments

Well, but …. (there is allways a 'but') :-)

Suppose I do not subclass XML::Parser. But then, how do I pass parameters to XML::Parser handler methods and collect results of their run without using global variables of XML::Parser package? Only class that I get to handler methods is expat itself and there is no place for any aditional parameters/results of handler methods.

And if I subclass XML::Parser, only advantage that I gain is using my own package namespace for global variables instead of XML::Parser's namespace. This do not looks to me like a good example of object oriented programming style.

Possible silution is the one mirod suggested using Non-Expat-Options but it is just a little bit less ugly than these two.

There best solution will be forcing XML::Parser to use my custom subclass of XML::Parser::Expat instead of XML::Parser::Expat itself. Is there some way how to do that?

Comments

The way to do this, without relying on the fact that the $p is a hashref, is to pass a closure as the handlers, and have an object that you created saved in the closure. This is how PerlSAX is implemented.

Witness:

my $handler = bless {}, "MyHandler";
my $p = XML::Parser->new(Handlers => {
   Start => sub { $handler->handle_start(@_) }
});

package MyHandler;

sub handle_start {
  my ($handler, $p, $element, %attribs) = @_;
  ...
}

by Anonymous Monk