Tags:
caching2Add my vote for this tag database2Add my vote for this tag create new tag
view all tags

YetAnotherDBCacheContrib

Reusable code that treats TWiki forms data as if it were a database

ALERT! TWiki-4 only. If you want to use this extension with an earlier version of TWiki, please contact the author.

ALERT! This contrib was derived from an earlier version of TWiki:Plugins.DBCacheContrib and continued to be developed in parallel thereafter. Hopefully, at some point the capabilities will be merged.

Summary

This module supports structured queries over a database built on the fly from the forms in TWiki topics. It does not support any tags, as it is provided as a service for other plugins that want to treat a TWiki web as a simple database; for example, the TWiki:Plugins/FormQueryPlugin, which supports the display of query results.

The plugin encapsulates code that was formerly in the "engine room" of the FormQueryPlugin. It has been abstracted out in the belief that it will be useful to anyone who wants to do simple search operations from a plugin.

Features

  • Perform complex queries on the TWiki database
  • Cache TWiki topics for rapid queries

How the database gets built

You can think of the database as an array of all the topics in a web. Each array entry is a map (or hash in perl terms) that maps a set of field names to values.

Each topic in the web automatically gets a number of standard fields, generated by reading data and metadata (see TWikiMetaData) from the topic:

  • name - name of the topic
  • parent - name of parent topic
  • web - name of the topic web
  • _up - reference to the Map of the parent topic, if it exists
  • attachments - array of Maps, each of which contains:
    • _up - reference to the Map for the topic
    • name - attachment name
    • attr - e.g hidden
    • comment - attachment comment
    • path - client path used to upload attachment
    • size - size in Kb
    • user - who uploaded the attachment
    • version - e.g. 1.3
  • info - Map containing:
    • _up - reference to the Map for the topic
    • author - most recent author
    • date - date of last change
    • format - topic format version
    • version - topic version number
  • moved - Map containing:
    • _up - reference to the Map for the topic
    • by - who moved it
    • date - when they moved it
    • from - where they moved it from
    • to - where they moved it to
  • form - form name
  • form name - (e.g. if a Form named "MyForm" is attached, this will be MyForm); a Map containing:
    • _up - reference to the Map for the topic
    • a key for each field in the form (see TWikiForms) mapping to the value in the form data for that key
  • text - raw text of the topic

The sub-Maps created for info, form name, moved, and each row in attachments also have a reference back to the topic Map, called _up.

Other fields may be added by subclasses. Refer to the documentation for clients of DBCacheContrib for more details.

ALERT! Note that the text field contains the raw topic text plus all meta data that is not pulled out into standard fields, or fields generated by subclasses.

The cache

To achieve best performance the plugin caches the database read from the TWiki topics. It creates this cache in a file in the web, _DBCache, albeit the file name can be overridden in a descendant. If any topic changes in the web this cache is automatically rebuilt. The cache file can be deleted at any point with no ill effects.

Extending or customising

Extension or customisation is welcome, as long as all extensions are described and code provided back to the author.

The module is shipped with a perl build file, which should be used for installation and testing. Testing is done using CPAN:Test::Unit, and is invoked using the 'test' build target. Writing tests is a useful way of feeding back bugs as well. I can't encourage you enough to maintain and extend the tests!

Detailed Documentation

Clients use the DBCache by defining a subclass of the TWiki::Contrib::DBCache class. The following POD documentation describes the methods of this class and the various other classes provided by the plugin..

class DBCacheContrib

General purpose cache that treats TWiki topics as hashes. Useful for rapid read and search of the database. Only works on one web.

Typical usage:

  use TWiki::Contrib::DBCacheContrib;

  $db = new TWiki::Contrib::DBCacheContrib( $web ); # always done
  $db->load(); # may be always done, or only on demand when a tag is parsed that needs it

  # the DB is a hash of topics keyed on their name
  foreach my $topic ($db->getKeys()) {
     my $attachments = $db->get($topic)->get("attachments");
     # attachments is an array
     foreach my $val ($attachments->getValues()) {
       my $aname = $attachments->get("name");
       my $acomment = $attachments->get("comment");
       my $adate = $attachments->get("date");
       ...
     }
  }
As topics are loaded, the readTopicLine method gives subclasses an opportunity to apply special processing to indivual lines, for example to extract special syntax such as %ACTION lines, or embedded tables in the text. See FormQueryPlugin for an example of this.

new($dataDir, $web, $managed)

  • $dataDir - location of cache file
  • $web - name of web to create the object for.
  • $managed - Boolean flag indicating whether the application is responsible for managing the state of currency of the database.
Construct a new DBCache object. To increase performance, if it is always known to the application when topics are changed, added, or deleted, one can set the database to be managed. Managed databases are not monitored whether they need to be reloaded due to changes to the topics in the web.

invalidate($web, $topic)

  • $web - name of web to create the object for.
  • $topic - name of the topic to be reloaded
Should be called from afterSaveHandler when a plugin is managing the database. Ensures that the database will be reloaded.

readTopicLine($topic, $meta, $line, $fh) --> text

  • $topic - name of the topic being read
  • $meta - reference to the hash object for this topic
  • line - the line being read
  • $fh - the file handle of the file
  • return text to insert in place of line in the text field of the topic
Called when reading a topic that is being cached, this method is invoked on each line in the topic. It is designed to be overridden by subclasses; the default implementation does nothing. The sort of expected activities will be (for example) reading tables and adding them to the hash for the topic.

onReload($topics)

  • $topics - perl array of topic names that have just been loaded (or reloaded)
Designed to be overridden by subclasses. Called when one or more topics had to be read from disc rather than from the cache. Passed a list of topic names that have been read.

load()

Load the web into the database. Returns a string containing 3 numbers that give the number of topics read from the cache, the number read from file, and the number of previously cached topics that have been removed.

reload()

  • $topic - name of topic that needs to be reloaded

Invalidates the database. For a managed database, causes a reload of the database. This function should be called whenever a change to the web occurred that needs to be reflected in the cache.

write($archive)

  • $archive - the TWiki::Contrib::DBCacheContrib::Archive being written to
Writes this object to the archive. Archives are used only if Storable is not available. This method must be overridden by subclasses is serialisation of their data fields is required.

read($archive)

  • $archive - the TWiki::Contrib::DBCacheContrib::Archive being read from
Reads this object from the archive. Archives are used only if Storable is not available. This method must be overridden by subclasses is serialisation of their data fields is required.

class Search

Search operators work on the fields of a TWiki::Contrib::DBCacheContrib::Map. The fields are given by their name, and values by strings. Strings are surrounded by single quotes. Strings may be regular expressions and use 'perl' re syntax (see man perlre for help). An embedded search may also be applied to a field value. In addition, the left-hand and right-hand sides of a search expression may also by of the form $CALC( text ), where text is some text string surrounded by quotes which is evaluated by the SpreadSheetPlugin. The special operator $T interprets its single argument as a field expression. (The quotes need to be escaped if within a string.)

Warning single and double quotes are not allowed in values!

The following operators are available:

Operator LHS RHS Meaning
l = r field name or calc regular expression string or field name or calc Value exactly matches this regular expression. The expression must match the whole string.
l = r field name or calc regular expression string or field name or calc Field is not this RE. Inverse of =
l =~ r field name or calc regular expression string or field name or calc Value contains this regular expression i.e. the RE is found somewhere in the field value.
l < r field name or calc integer (string containing an integer e.g '4') field value is < integer
l > r field name or calc integer field value is > integer
l >= r field name or calc integer field value is >= integer
l <= r field name or calc integer field value is <= integer
l [? r ] field name search expression search applied to field value

A search string consisting of text somestring without any operator is interpreted as the string text=~'somestring'.

A search string consisting of a single =%CALC% expression is interpreted as a test of the result of the expression.

Searches may be combined by the following logical operators:

Operator LHS RHS Meaning
! none expr Boolean NOT
AND expr expr Boolean AND
OR expr expr Boolean OR
() N/A N/A Bracketed subexpression

A search object implements the "matches" method as its general contract with the rest of the world.

Example

Get a list of hidden attachments:
  $db = new TWiki::Contrib::DBCacheContrib::DBCache( $web ); # always done
  $db->load();
  my $search = create Search("attr='h'");

  foreach my $topic ($db->getKeys()) {
     my $attachments = $topic->get("attachments");
     foreach my $val ($attachments->getValues()) {
       if ($search->matches($val)) {
          print $val->get("name") . "\n";
       }
     }
  }

create($string)

  • $string - string containing an expression to parse
Construct a new search node by parsing the passed expression.

matches($object, $case) -> boolean

  • $object - object to test; must implement get
  • $case - boolean flag; the match is casesensitive if true
See if object matches the search. $object can be any object that provides the method "get" that returns a value given a string key.

toString() -> string

Generates a string representation of the object.

class FileTime

Object that handles a file/time tuple for use in Storable and TWiki::Contrib::DBCacheContrib::Archive.

new($file)

  • $file - filename
Construct from a file name

uptodate() -> boolean

Check the file time against what is seen on disc. Return 1 if consistent, 0 if inconsistent.

toString() -> string

Generates a string representation of the object.

write()

TWiki::Contrib::DBCacheContrib::Archive hook

read()

TWiki::Contrib::DBCacheContrib::Archive hook

class Array

Generic array object. This is required because perl arrays are not objects, and cannot be subclassed e.g. for serialisation. To avoid lots of horrid code to handle special cases of the different perl data structures, we use this array object instead.

new()

Create a new, empty array object

add($object)

  • $object any perl data type
Add an element to the end of the array

find($object) -> integer

  • $object datum of the same type as the content of the array
Uses "==" to find the given element in the array and return it's index

remove($index)

  • $index - integer index
Remove an entry at an index from the array.

get($key, $root) -> datum

  • $k - key
  • $root - what # refers to
Subfield syntax
  • get("9", $r) where $n is a number will get the 9th entry in the array
  • get("[9]", $r) will also get the 9th entry
  • get(".X", $r) will return the sum of the subfield X of each entry
  • get("[?search]", $r) will perform the given search over the entries in the array. Always returns an array result, even when there is only one result. For example: [?name='Sam'] will return an array of all the entries that have their subfield name set to Sam.
  • # means "reset to root". So get("#[3]", $r) will return the 4th entry of $r (assuming $r is an array!).
  • get("[*X]", $r) will get a new array made from subfield X of each entry in this array.

Where the result of a subfield expansion is another object (a Map or an Array) then further subfield expansions can be used. For example,

get("parent.UserTable[?SubTopic='ThisTopic'].UserName", $web);

See also TWiki::Contrib::DBCacheContrib::Map for syntax that applies to maps.

size() -> integer

Get the size of the array

sum($field) -> number

  • $field - name of a field in the class of objects stored by this array
Returns the sum of values of the given field in the objects stored in this array.

search($search, $case) -> search result

  • $search TWiki::Contrib::DBCacheContrib::Search object to use in the search
  • $case - boolean flag; the match is casesensitive if true
Search the array for matches with the given object. values. Return a TWiki::Contrib::DBCacheContrib::Array of matching entries.

getValues() -> perl array

Get a "perl" array of the values in the array, suitable for use with foreach

toString($limit, $level, $strung) -> string

  • $limit - recursion limit for expansion of elements
  • $level - currentl recursion level
Generates an HTML string representation of the object.

write($archive)

  • $archive - the TWiki::Contrib::DBCacheContrib::Archive being written to
Writes this object to the archive. Archives are used only if Storable is not available. This method must be overridden by subclasses is serialisation of their data fields is required.

read($archive)

  • $archive - the TWiki::Contrib::DBCacheContrib::Archive being read from
Reads this object from the archive. Archives are used only if Storable is not available. This method must be overridden by subclasses is serialisation of their data fields is required.

class Map

Generic map object for mapping names to things. A name is defined as name = \w+ | \w+ "." name The . indicates a field reference in a sub-map. Objects in the map are either strings, or other objects that must support toString.

new($string)

  • $string - optional attribute string in standard TWiki syntax
Create a new, empty array object. Optionally parse a standard attribute string containing name=value pairs. The value may be a word or a quoted string (no escapes!)

fastget($k) -> datum

  • $k - key
Get the value for a key, but without any subfield field expansion

get($k, $root) -> datum

  • $k - key
  • $root what # refers to
Get the value corresponding to key $k; return undef if not set.

Subfield syntax

  • get("X",$r) will get the subfield named X.
  • get("X.Y",$r) will get the subfield Y of the subfield named X.
  • # means "reset to root". So get("#.Y", $r) will return the subfield =Y of $r (assuming $r is a map!), as will get("#[Y]".

Where the result of a subfield expansion is another object (a Map or an Array) then further subfield expansions can be used. For example,

get("UserTable[0].Surname", $web);

See also TWiki::Contrib::DBCacheContrib::Array for syntax that applies to arrays.

set($k, $v)

  • $k - key
  • $v - value
Set the given key, value pair in the map.

size() -> integer

Get the size of the map

remove($index) -> old value

  • $index - integer index
Remove an entry at an index from the array. Return the old value.

getKeys() -> perl array

Get a "perl" array of the keys in the map, suitable for use with foreach

getValues() -> perl array

Get a "perl" array of the values in the Map, suitable for use with foreach

search($search, $case) -> search result

  • $search - TWiki::Contrib::DBCacheContrib::Search object to use in the search
  • $case - boolean flag; the match is casesensitive if true
Search the map for keys that match with the given object. values. Return a TWiki::Contrib::DBCacheContrib::Array of matching keys.

toString($limit, $level, $strung) -> string

  • $limit - recursion limit for expansion of elements
  • $level - currentl recursion level
Generates an HTML string representation of the object.

write($archive)

  • $archive - the TWiki::Contrib::DBCacheContrib::Archive being written to
Writes this object to the archive. Archives are used only if Storable is not available. This method must be overridden by subclasses is serialisation of their data fields is required.

read($archive)

  • $archive - the TWiki::Contrib::DBCacheContrib::Archive being read from
Reads this object from the archive. Archives are used only if Storable is not available. This method must be overridden by subclasses is serialisation of their data fields is required.

class Archive

Simple file archive storer and restorer. Handles serialising objects using their "write" and "read" methods. Serialisable objects must have a no-parameters constructor.

This module is only used if Storable isn't available. Storable is much faster, because it is implemented in C.

new($file, $rw)

  • $file - archive file path
  • $rw - mode "r" or "w"
Create a new archive, using filename $file and mode $rw which must be "r" or "w". The archive will remain in existence (and the file remain open) until "close" is called. An exclusive lock is taken for write as long as the file is open. Throws an exception if the archive cannot be opened.

close()

Close this archive. MUST be called to close the file.

writeByte($b)

  • $b - byte to write
Write a byte to the archive

writeString($s)

  • $s - string to write
Write a string to the archive

writeInt($i)

  • $i integer to write
Write a 32-bit integer to the archive

writeObject()

Write an object to the archive. An object must implement read($archive) and write($archive), or may be undef or a string. No other types are supported.

readByte() -> byte

Read a byte from the archive

readString() -> string

Read a UTF8 string from the archive

readInt() -> integer

Read a 32-bit integer from the archive

readObject() -> object

Read an object from the archive

Settings

  • Name of the perl package
    • Set STUB = TWiki::Contrib::DBCacheContrib

Installation Instructions

  • Download the ZIP file from the Plugin web (see below). Contents:
File: Description:
data/TWiki/DBCacheContrib.txt  
lib/TWiki/Contrib/DBCacheContrib.pm  
lib/TWiki/Contrib/DBCacheContrib/Archive.pm  
lib/TWiki/Contrib/DBCacheContrib/Array.pm  
lib/TWiki/Contrib/DBCacheContrib/Map.pm  
lib/TWiki/Contrib/DBCacheContrib/FileTime.pm  
lib/TWiki/Contrib/DBCacheContrib/Search.pm  
lib/TWiki/Contrib/DBCacheContrib/Search.yp  
lib/TWiki/Contrib/DBCacheContrib/build.pl  
lib/TWiki/Contrib/DBCacheContrib/DEPENDENCIES  
test/unit/DBCacheContrib/ArchiveTest.pm  
test/unit/DBCacheContrib/ArrayTest.pm  
test/unit/DBCacheContrib/DBCacheContribSuite.pm  
test/unit/DBCacheContrib/DBCacheTest.pm  
test/unit/DBCacheContrib/FileTimeTest.pm  
test/unit/DBCacheContrib/MapTest.pm  
test/unit/DBCacheContrib/SearchTest.pm  
test/unit/DBCacheContrib/testDB.dat  

  • Unzip YetAnotherDBCacheContrib.zip in your twiki installation directory.
  • Optionally, run YetAnotherDBCacheContrib_installer to automatically check and install other TWiki modules that this module depends on. You can also do this step manually.
  • Alternatively, manually make sure the dependencies listed in the table below are resolved.

Contrib Info

Author: TWiki:Main/CrawfordCurrie developed the original version of this contrib. Changes to functionality by TWiki:Main/ThomasWeigert.
Copyright ©: This code is based on an original development of Motorola Inc. and is protected by the following copyrights:
© 2002-2006 Motorola Inc. All Rights Reserved.
Portions © 2004. Crawford Currie http://www.c-dot.co.uk
License: As required for the publication of all extensions to TWiki, this software is published under the terms of the GNU General Public License. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details, published at http://www.gnu.org/copyleft/gpl.html
Version: 01 Sep 2006
Change History:  
04 Dec 2006 Support "managed cache", where the application is responsible for detecting changes to the cached topics.
01 Sep 2006 Expand encoding in form fields when creating cache.
23 Jun 2006 Fixed updating of parents when reloading from cache. Replaced date operators by the ability to leverage the SpreadSheetPlugin. Replaced parser with LALR-1 table driven parser. Store web information in cache. Fix summing of subfields, expansion of fields, [? search ] syntax, array notation. Empty search matches the whole data base (consistent with %SEARCH%).
9303 don't die on broken symlinks
8682 Item1580 one-char fix that makes the difference
8110 Item663 formatting and text fixes
7552 Item997 test update
7274 Item719 onReload() is not a static method.
7262 Item719 MichaelDaum's patch (almost) to correct parameters to onReload
7260 Item727 made it clean the form name using normaliseWebTopicName
6353 Item380 do as the man says; make all $/ local
5720 Updated tests
5719 Fix for correct handling of parent relations
5229 Small improvement to the way it handles errors from Storable and Archive
5223 Documentation fixes, adding gifs.
5048 Cairo readiness
5036 Split from SharedCode
5031 Moving to new name
5030 About to rename
5019 Improved topic data model, cleaned up tests
5008 Added extended access syntax, [?], [*] etc.
5006 Doc fixes
5005 Poddified documentation
5003 Initial version
8 Jul 2004 Initial version, split out from FormQueryPlugin
Dependencies:
NameVersionDescription
StorableRecommended; accelerates cache handling. Available from CPAN
Parse::YappRequired; parsing of queries. Available from CPAN
TWiki::Plugins::SpreadSheetPluginRequired. Download and install from the TWiki:Plugins/SpreadSheetPlugin code library.
Perl Version: 5.0
Plugin Home: http://TWiki.org/cgi-bin/view/Plugins/YetAnotherDBCacheContrib
Feedback: http://TWiki.org/cgi-bin/view/Plugins/YetAnotherDBCacheContribDev
Demo URL:  

-- TWiki:Main/ThomasWeigert - 01 Sep 2006

Topic attachments
I Attachment History Action Size Date Who Comment
Unknown file formatmd5 YetAnotherDBCacheContrib.md5 r5 r4 r3 r2 r1 manage 0.2 K 2007-01-15 - 11:29 UnknownUser  
Compressed Zip archivetgz YetAnotherDBCacheContrib.tgz r5 r4 r3 r2 r1 manage 39.4 K 2007-01-15 - 11:29 UnknownUser  
Compressed Zip archivezip YetAnotherDBCacheContrib.zip r5 r4 r3 r2 r1 manage 52.7 K 2007-01-15 - 11:29 UnknownUser  
Unknown file formatext YetAnotherDBCacheContrib_installer r1 manage 30.6 K 2007-01-15 - 11:29 UnknownUser  
Edit | Attach | Watch | Print version | History: r4 < r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r4 - 2009-07-12 - ThomasWeigert
 
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.