Regular Expressions and Matching

Perl's powerful ability to manipulate text comes in part from its inclusion of a computing concept known as regular expressions. A regular expression (often shortened to regex or regexp) is a pattern which describes characteristics of a string of text. A regular expression engine interprets a pattern and applies it to strings of text to identify those which match.

Perl's core documentation describes Perl regular expressions in copious detail; see perldoc perlretut, perldoc perlre, and perldoc perlreref for a tutorial, the full documentation, and a reference guide, respectively. Jeffrey Friedl's book Mastering Regular Expressions explains the theory and the mechanics of how regular expressions work. Even though those references may seem daunting, regular expressions are like Perl--you can do many things with only a little knowledge.

Literals

The simplest regexes are simple substring patterns:

    my $name = 'Chatfield';
    say "Found a hat!" if $name =~ /hat/;

If the range of allowed characters in these four groups isn't specific enough, you can specify your own character classes by enclosing them in square brackets:

    my $vowels    = qr/[aeiou]/;
    my $maybe_cat = qr/c${vowels}t/;

If the characters in your character set form a contiguous range, you can use the hyphen character (-) as a shortcut to express that range. Now it's possible to define the $letters_only regex:

    my $letters_only = qr/[a-zA-Z]/;

Move the hyphen character to the start or end of the class to include it in the class:

    my $interesting_punctuation = qr/[-!?]/;

... or escape it:

    my $line_characters = qr/[|=\-_]/;

Just as the word and digit class metacharacters (\w and \d) have negations, so too you can negate a character class. Use the caret (^) as the first element of the character class to mean "anything except these characters":

    my $not_a_vowel = qr/[^aeiou]/;

Capturing

It's often useful to match part of a string and use it later; perhaps you want to extract an address or an American telephone number from a string:

    my $area_code    = qr/\(\d{3}\)/;
    my $local_number = qr/\d{3}-?\d{4}/;
    my $phone_number = qr/$area_code\s?$local_number/;

Note the escaping of the parentheses within $area_code; this will become obvious in a moment.

Named Captures

Given a string, $contact_info, which contains contact information, you can apply the $phone_number regular expression and capture any matches into a variable with named captures:

    if ($contact_info =~ /(?<phone>$phone_number)/)
    {
        say "Found a number $+{phone}";
    }

The capturing construct can look like a big wad of punctuation, but it's fairly simple when you can recognize it as a single chunk:

    (?<capture name> ... )

The parentheses enclose the entire capture. The ?< name > construct provides a name for the capture buffer and must follow the left parenthesis. The rest of the construct within the parentheses is a regular expression. If and when the regex matches this fragment, Perl stores the captured portion of the string in the magic variable %+: a hash where the key is the name of the capture buffer and the value is the portion of the string which matched the buffer's regex.

Parentheses are special to Perl 5 regular expressions; by default they exhibit the same grouping behavior as parentheses do in regular Perl code. They also enclose one or more atoms to capture whatever portion of the matched string they match. To use literal parentheses in a regular expression, you must preface them with a backslash, just as in the $area_code variable.

Numbered Captures

Named captures are new in Perl 5.10, but captures have existed in Perl for many years. You may encounter numbered captures as well:

    if ($contact_info =~ /($phone_number)/)
    {
        say "Found a number $1";
    }

regex; $1 regex; $2

The parentheses enclose the fragment to capture, but there is no regex metacharacter giving the name of the capture. Instead, Perl stores the captured substring in a series of magic variables starting with $1 and continuing for as many capture groups are present in the regex. The first matching capture that Perl finds goes into $1, the second into $2, and so on. Capture counts start at the opening parenthesis of the capture; thus the first left parenthesis begins the capture into $1, the second into $2, and so on.

While the syntax for named captures is longer than for numbered captures, it provides additional clarity. You do not have to count the number of opening parentheses to figure out whether a particular capture is $4 or $5, and composing regexes from smaller regexes is much easier, as they're less sensitive to changes in position or the presence or absence of capturing in individual atoms.

Numbered captures are less frustrating when you evaluate a match in list context:

    if (my ($number) = $contact_info =~ /($phone_number)/)
    {
        say "Found a number $number";
    }

Perl will assign to the lvalues in order of the captures.

Grouping and Alternation

Previous examples have all applied quantifiers to simple atoms. They can also apply to more complex subpatterns as a whole:

    my $pork  = qr/pork/;
    my $beans = qr/beans/;

    like( 'pork and beans', qr/\A$pork?.*?$beans/,
         'maybe pork, definitely beans' );

If you expand the regex manually, the results may surprise you:

    like( 'pork and beans', qr/\Apork?.*?beans/,
         'maybe pork, definitely beans' );

This still matches, but consider a more specific pattern:

    my $pork  = qr/pork/;
    my $and   = qr/and/;
    my $beans = qr/beans/;

    like( 'pork and beans', qr/\A$pork? $and? $beans/,
        'maybe pork, maybe and, definitely beans' );

Some regexes need to match one thing or another. Use the alternation metacharacter (|) to do so:

    my $rice  = qr/rice/;
    my $beans = qr/beans/;

    like( 'rice',  qr/$rice|$beans/, 'Found some rice'  );
    like( 'beans', qr/$rice|$beans/, 'Found some beans' );

The alternation metacharacter indicates that either preceding fragment may match. Be careful about what you interpret as a regex fragment, however:

    like(   'rice',  qr/rice|beans/, 'Found some rice'  );
    like(   'beans', qr/rice|beans/, 'Found some beans' );
    unlike( 'ricb',  qr/rice|beans/, 'Found some weird hybrid' );

It's possible to interpret the pattern rice|beans as meaning ric, followed by either e or b, followed by eans--but alternations always include the entire fragment to the nearest regex delimiter, whether the start or end of the pattern, an enclosing parenthesis, another alternation character, or a square bracket.

To reduce confusion, use named fragments in variables ($rice|$beans) or grouping alternation candidates in non-capturing groups:

    my $starches = qr/(?:pasta|potatoes|rice)/;

The (?:) sequence groups a series of atoms but suppresses capturing behavior. In this case, it groups three alternatives.

Other Escape Sequences

Perl interprets several characters in regular expressions as metacharacters, which represent something different than their literal characters. Square brackets always denote a character class and parentheses group and optionally capture pattern fragments.

To match a literal instance of a metacharacter, escape it with a backslash (\). Thus \( refers to a single left parenthesis and \] refers to a single right square bracket. \. refers to a literal period character instead of the "match anything but an explicit newline character" atom.

Other useful metacharacters that often need escaping are the pipe character (|) and the dollar sign ($). Don't forget about the quantifiers either: *, +, and ? also qualify.

To avoid escaping everything (and worrying about forgetting to escape interpolated values), use the metacharacter disabling characters. The \Q metacharacter disables metacharacter processing until it reaches the \E sequence. This is especially useful when taking match text from a source you don't control when writing the program:

    my ($text, $literal_text) = @_;

    return $text =~ /\Q$literal_text\E/;

The $literal_text argument can contain anything--the string ** ALERT **, for example. With \Q and \E, Perl will not interpret the zero-or-more quantifier as a quantifier. Instead, it will parse the regex as \*\* ALERT \*\* and attempt to match literal asterisk characters.

Assertions

The regex anchors (\A and \Z) are a form of regex assertion, which requires that a condition is present but doesn't actually match a character in the string. That is, the regex qr/\A/ will always match, no matter what the string contains. The metacharacters \b and \B are also assertions.

Zero-width assertions match a pattern, not just a condition in the string. Most importantly, they do not consume the portion of the pattern that they match. For example, to find a cat on its own, you might use a word boundary assertion:

    my $just_a_cat = qr/cat\b/;

... but if you want to find a non-disastrous feline, you might use a zero-width negative look-ahead assertion:

    my $safe_feline = qr/cat(?!astrophe)/;

The construct (?!...) matches the phrase cat only if the phrase astrophe does not immediately follow.

The zero-width positive look-ahead assertion:

    my $disastrous_feline = qr/cat(?=astrophe)/;

... matches the phrase cat only if the phrase astrophe immediately follows. This may seem useless, as a normal regular expression can accomplish the same thing, but consider if you want to find all non-catastrophic words in the dictionary which start with cat. One possibility is:

    my $disastrous_feline = qr/cat(?!astrophe)/;

    while (<$words>)
    {
        chomp;
        next unless /\A(?<some_cat>$disastrous_feline.*)\Z/;
        say "Found a non-catastrophe '$+{some_cat}'";
    }

Because the assertion is zero-width, it consumes none of the source string. Thus the anchored .*\Z pattern fragment must be present; otherwise the capture would only capture the cat portion of the source string.

Zero-width look-behind assertions also exist. Unlike the look-ahead assertions, the patterns of these assertions must have fixed widths; you may not use quantifiers in these patterns.

To assert that your feline never occurs at the start of a line, you might use the zero-width negative look-behind assertion:

    my $middle_cat = qr/(?<!^)cat/;

... where the construct (? contains the fixed-width pattern. Otherwise you could express that the cat must always occur immediately after a space character with the zero-width positive look-behind assertion:

    my $space_cat = qr/(?<=\s)cat/;

    my $pet = 'CaMeLiA';

    like( $pet, qr/Camelia/,  'You have a nice butterfly there'       );
    like( $pet, qr/Camelia/i, 'Your butterfly has a broken shift key' );

    my $find_a_cat = qr/(?<feline>(?i)cat)/;

    my $find_a_rational = qr/(?<number>(?-i)Rat)/;

    my $attr_re = qr{
        ^                     # start of line

        # miscellany
        (?:
          [;\n\s]*            # blank spaces and spurious semicolons
          (?:/\*.*?\*/)?      # C comments
        )*

        # attribute marker
        ATTR

        # type
        \s+
        (   U?INTVAL
          | FLOATVAL
          | STRING\s+\*
          | PMC\s+\*
          | \w*
        )
    }x;

    # appease the Mitchell estate
    my $contents = slurp( $file );
    $contents    =~ s/Scarlett O'Hara/Mauve Midway/g;

    while ($contents =~ /\G(\w{3})(\w{3})(\w{4})/g)
    {
        push @numbers, "($1) $2-$3";
    }

    # appease the Mitchell estate
    my $contents = slurp( $file );
    $contents    =~ s{Scarlett( O'Hara)?}
                     { 'Mauve' . defined $1 ? ' Midway' : '' }ge;