TWiki is an open source project with 10+ years of history, built by a team of volunteers from around the world, and used by millions of people in over 100 countries. The community is focusing on building the best collaboration platform for the workplace. We invite you to get involved!
What is TWiki?
A leading open source enterprise wiki and web application platform used by 50,000 small businesses, many Fortune 500 companies, and millions of people. Learn more.
This is an advanced topic that teaches how to to parse nested tree structures such as programming languages and spreadsheet formulas using regular expressions in Perl. The article is generic enough to be used in other languages that support regular expressions.
Regular expressions are very powerful to parse and modify text. They are also very fast in processing large text over and over again. For example, the TWiki engine makes heavy use of regular expressions to turn TML (TWiki Markup Language) into HTML. Can regular expressions be used to parse nested structures, such as to parse a programming language or a nested spreadsheet formula?
Some people say that regular expressions (e.g. finite state automaton) cannot parse nested structures because they are not regular, by definition. They argue, you would need an LL parser such as ANTLR to generate an abstract syntax tree which can be further processed with a tree parser.
All that sounded too complicated when I was going to implement TWiki's SpreadSheetPlugin. Regular expressions are extremely fast, and I thought it might be faster than the LL parser and tree parser approach. So I took the challenge to parse spreadsheet formulas using regular expressions.
For illustration, let's take a spreadsheet formula that needs to be parsed and processed. The example formula has 7 function calls with 5 levels of nesting, some of which have more than one function per level:
There is a way to use regular expressions to parse, say, up to 5 levels, and with the limitation that you can have only one function call per level. Certainly we need a more flexible solution that does not limit us on the number of levels for nesting and number of function calls per level.
The solution is to parse twice, once to temporarily label the parentheses with level of nesting, then to resolve function calls recursively using matching parentheses identified by the labels. This can be done with regular expressions and recursive function calls.
First, let's determine the level of nesting, indicated by the numbers above the parentheses:
We use a regular expression to add the nesting level to the parentheses, together with an escape token (so that we can easily identify it later for processing and removal). Escape tokens with level are indicated by -esc-N; newlines are added for clarity:
Add an escape token with the level of nesting to each parenthesis of the formula. This is done using a regular expression that calls function _addNestingLevel(). We use modifiers geo, e.g. g for global (don't stop at the first match), e for execute (to call a function for each match), and o to compile the regular expression pattern only once (because the pattern is fixed).
Call _doFunc() to resolve the functions recursively.
The _addNestingLevel() function adds the escape tokens with the level of nesting to a parenthesis, such as -esc-5( and -esc-5):
The nesting level is passed as a reference so that we can modify it. The escape token is the null character, which is a valid character in a Perl string. For the opening parenthesis "(", the nesting is incremented before adding the escape token with the nesting level. For the closing parenthesis ")", it is decremented after the fact. This ensures that we have the same nesting level for matching parentheses.
The _doFunc function handles all spreadsheet functions:
The first regular expression s/\$([A-Z]+)$escToken([0-9]+)\((.*?)$escToken\2\)/_doFunc($1,$3)/geo finds all functions of format $NAME-esc-N( ... -esc-N) and calls the _doFunc function recursively to resolve the spreadsheet function. How can we identify the matching closing parenthesis of a function? The ([0-9]+) after $escToken captures the level. With \((.*?)$escToken\2\) we scan non-greedily over the text to find the first escape token of same level, indicated with \2. We use the same geo modifiers for the regular expression, the g modifier results in parsing the string from left to right to find and handle all spreadsheet functions of the same level.
Spreadsheet functions at a deeper level are resolved first because of the recursion. The combined use of recursion and g modifier results in an inside out / left to right handling of the spreadsheet functions, which is the desired behavior.
The second regular expression s/$escToken\-*[0-9]+([\(\)])/$1/go cleans up escape tokens that are left over. For example $EVAL( (2+7)/3 ) has parentheses that are not part of a function construct. Or, the formula might have unbalanced parentheses.
Finally we have a long if-then-else-if-then switch that handles all spreadsheet functions.
That is basically it! As you can see, with a few lines of code it is possible to parse complex nested structures by using just a few regular expressions and a function recursively.
This is a simplified representation of the actual implementation of the SpreadSheetPlugin. Some spreadsheet functions get special treatment. For example, the $IF( condition, then, else ) function needs to evaluate first the condition, then handle either the then or the else part based on the condition. This complicates the simple inside out / left to right parsing/handling described in this article. Interested folks are invited to download the SpreadSheetPlugin and study the code.
It would be interesting to do a reference implementation using the LL-parser/tree parser approach so that we could compare the performance to the pure regular expression/recursion approach discussed here. I have a hunch that the regular expression approach is an order of magnitude faster. Anyone challenging that? I would be happy to be proven wrong!
I hope this advanced regular expression topic is a learning experience for you as it certainly was for me. May be you can use this for one of your own projects?
Regular expressions, as implemented by common programming languages, are way more powerful than "regular expressions", as intended by computer science theorists in connection with finite state automata. There are very good reasons behind this power. Even something very basic, such as backreferences, are beyond the capabilities of regular expressions. I found this: http://dev.perl.org/perl6/doc/design/apo/A05.html, where Larry Wall says: "... generally having to do with what we call "regular expressions", which are only marginally related to real regular expressions".
-- Kushal Kumaran - 2011-09-19
Thanks for the pointer Kushal, I see there are dramatic changes/fixes to regexes coming with Perl 6.
-- Peter Thoeny - 2011-09-19
That tagging is a great idea. I've used recursion in some toy code of mine, which works well, but has less than wonderful performance, and it has the limitation that the inner parentheses must be processed away or the outer parentheses cannot be evaluated. I haven't tested this yet, but it does have potential.
That having been said, it looks to me like it'd be better to put the tag after the parentheses. That way, the parenthesis itself indicates the next character will be the first character of a tag. There's no way any non-parenthesis will be confused as a tag temporarily, triggering a back track. You then need to have a sequence to end the tag, but since you can't possibly be looking at a false positive, a single, non-digit character will do.
So, instead of
-- Ed Grimm - 2011-09-19
Thanks Ed for your insights. Not sure how much performance you gain by swapping the parenthesis with the escape token. The -esc- was just for visuals, in reality it is a null character. Does it make a difference in performance if you scan for .N( vs (N.? Your approach is better in a sense that you can take any non-digit char to terminate the sequence, vs. a null character to start the sequence.
-- Peter Thoeny - 2011-09-20