A Data Storage Framework for TWiki

Many times in Codev people have requested:

• An easier time installing TWiki - usually problems centre around RCS
• Being able to change the storage backend e.g. to use a database or a different version control system
• Having a more object orientated TWiki

The data storage framework aims to help in the above by:

• Giving 100% Perl code for reading and writing RCS files - RcsLite
• Both RCS and RcsLite will be made available through the same interface - other storage back ends can be added later
• The interface being used for RCS and RcsLite is a simple Perl OO one - a good place for some OO because:
  • Store.pm has got rather complicated; mainly due to me, JohnTalintyre
  • My judgement is that loss of speed due to the way Perl OO works won't be an issue. There aren't too many sub calls, which are slightly slower under OO; there is losts of reading values by lookup (%vars) but still I would think not enough to matter

Questions/votes

Refering to version numbers

TWiki uses the form x.y to refer to a version number with x=1. In RcsLite.pm and =Rcs.pm x=1 is built in and versions refered to as just y. Why not do so in the user interface as well i.e. list would be r3, r2, r1 rather than r1.3, r1.2, r1.1.

Thoughts?

-- JohnTalintyre - 09 Mar 2002

Rational ClearCase uses only major version numbers (e.g. 1, 2, 3, ... n), so it'd be transparent there.

-- PeterNixon - 10 Mar 2002

Putting in cvs

I'm ready to put my work to date in CVS. It will mean a new directory Data with 3 modules (alternatively this could all go in a directory called Store; I've kept separate so that Store.pm remains procedurally based and Data OO based. Additionally there are quite a few changes to Store.pm put variable changes in TWiki.cfg and TWiki.pm. The RCS option should work as before, but needs more testing; if it isn't good enough for the next release Store, and the TWiki files can be regressed. RcsLite should be treated as ready for initial testing, but certainly not prime time. You swith between them by setting the variable dataImpl in TWiki.cfg.

Unless I hear objections I'll be doing a commit in the next few days.

-- JohnTalintyre - 10 Mar 2002

TWikiAlphaRelease is currently in a stable state, at any time we can create a TWikiBetaRelease which can be used for production. Therefore my questions:

• Is there any noticable performance hit when using the external RCS? (Comes into play when search hits 100 or more topics)
• How stable is the data framework code when using the external RCS?

Regarding module structure, how about placing it below TWiki::Store where it logically belongs? For example TWiki::Store::Rev, TWiki::Store::RevRCS and TWiki::Store::RevRcsLite ("Rev" for revision storage). The variable names in TWiki.cfg should be matched accordingly.

-- PeterThoeny - 12 Mar 2002

Peter, as always, you bring up good points.

• I've still got to test for performance on external RCS. But, I'd be very suprised if there is a hit.
• The data framework code for external RCS is largely extracted from Store.pm. But, I've made enough changes that general testing is needed. Note that I've swithced to trapping RCS errors using return values rather than stderr. So the data for the next release is really key to commiting to CVS.

I ensisage this framework doing all storage related calls in the future e.g. including searching and TWiki lock capability. It seems logical to me that all data IO should be configured together. As such Rcs.pm and RcsLite.pm would be slightly misleading names. I'm warry of using much Perl OO, but by inheritance or composition would could use these modules to produce a generic storage capability.

-- JohnTalintyre - 13 Mar 2002

Some more information and thoughts:

Timing for edit of TWikiVariables topic (266 MHz PC, Win2k), ignoring page compilation stage

• current: 0.59s
• Rcs.pm: 0.58s
• RcsLite.pm: 1.56s

Module structure. How about:

• Store::Topic.pm - "interface" RcsLite and Rcs support and other topic/attachment level functions
• Store::RcsLite.pm
• Store::Rcs.pm
• Store::RcsFile.pm - used by both of above

could later add the following for searching.

• Store::Search.pm
• Store::LsGrep.pm

could also have:

• Store::Admin.pm - various admin functions e.g. creating a new Web [15 Mar 2002 JohnTalintyre]

for configuration purposes all the above would share the same hash. Note that note all sets of implementation would be compatible (in the future). [15 Mar 2002 JohnTalintyre]

-- JohnTalintyre - 13 Mar 2002

When Andersens built the IndexServerSearchForMsIisAddOn we also added the Search perl modules such as lib/IISearch.pm.

Taking this into account you might want to make space in your naming by factoring out common search components into Store::Search and have implementations in a place such as Store::Unix::GrepSearch and Store::Windows::IndexServerSearch.

-- MartinCleaver - 14 Mar 2002

My approach would allow any number of implementations for Search, with module (class) name specified in TWiki.cfg. Some more details added above.

-- JohnTalintyre - 15 Mar 2002

John, on timing, could you run a test that checks the Store backend with lots of accesses on view, for example a WebChanges or WebIndex?

On module structure, looks good. One point to consider is how much performance loss we have with many OO modules. May be reduce the number of modules to a bare minimum if it helps on the performance?

-- PeterThoeny - 16 Mar 2002

At present I've haven't altered the way view runs, although I had intended to get it to call via DataFramework later - I'll try this and post the times.

I've been very conscious of not going heavily down the OO route (the I would say in Java or C#). So at present you call RcsLite which might call back to RcsFile (which it inherits from). So the main affect on performance is the cost of an indirect functional call, plus variables held in a cache. Viewing would at most have minimal calls into this system. The timing for edit (above) shows that the OO calls don't at present make a noticable different. RcsLite is slower, but I've yet to even try and speed it up.

-- JohnTalintyre - 18 Mar 2002

I've checked some files into CVS (no affect yet because I've not checked in any existing files). Unfortunately, CVS is dying when I try to commit TWiki/Store/Rcs.pm with error:

cvs -z3 -d:ext:talintj@cvs.TWiki.sourceforge.net:/cvsroot/twiki commit lib/TWiki/Store/Rcs.pm
cvs: hash.c:312: findnode: Assertion `key != ((void *)0)' failed.
cvs [server aborted]: received abort signal
cvs commit: saving log message in /tmp/cvs61095630.1

Anyone have any ideas on this?

-- JohnTalintyre - 19 Mar 2002

You might want to try without -z3 just in case. This Google search didn't find anything useful...

-- RichardDonkin - 19 Mar 2002

I changed the name from Rcs.pm to RcsWrap.pm to get around the problem. Now this is in CVS I'd be grateful if people could give it a try - RcsLite should work, but don't use it in production, in particular I'm not sure if attachment histories will always work yet.

testenv is likely now a bit broken and documentation out of date - I'll take a look at these next. If this proves to have stability problems that can't be quickly solved then I'll withdraw it from the next release.

Note that unit tests for this appear in twiki/tools/test/rcslitetest.

-- JohnTalintyre - 19 Mar 2002

I'm still getting cvs server: nothing known about twiki/lib/TWiki/Store/Rcs.pm on doing cvs update - is there some problem on my side, or is there a half-checked in Rcs.pm file?

I just put in some changes to testenv to check RCS versions - these are in a couple of places. And of course the variable changes will have broken things...

Do we just report Alpha bugs as normal, or log problems here?

-- RichardDonkin - 19 Mar 2002

I did a remove on Rcs.pm and instead used RcsWrap.pm I think we've generally gone for alpha bugs in normal way in Codev.

-- JohnTalintyre - 20 Mar 2002

I'm still getting that error, tried again just now. I don't have Rcs.pm anywhere in my tree, and there aren't any references to it in the Store or Store/CVS subdirectories.

-- RichardDonkin - 20 Mar 2002

Unfortunately machine moves mean that my work access to CVS is down at present. Also my home machine is a bit ill! So not sure when I'll be able to check this. Sounds as if CVS got more than a little confused when I tried putting in Rcs.pm.

-- JohnTalintyre - 20 Mar 2002

I am testing out the new code on my Alpha installation and found that TWiki tries to unsuccessfully load TWiki::Store::RcsWrap.pm when you view a topic that has an embedded search:
%SEARCH{"test"}%

Strangely enough edit, preview and rdiff work, but not view. Looks like the current directory differs from the initial bin at the time RcsWrap is loaded. If, for testing, I put a use TWiki::Store::RcsWrap just after use TWiki in view it does work.

-- PeterThoeny - 22 Mar 2002

I'll try and duplicate the error and see if the path techniques used in the Plugin code can avoid this problem. I'm also interested to see where the call is that needs this - I thought I'd so far avoid calls during view and search.

-- JohnTalintyre - 23 Mar 2002

Fix for this in Store.pm checked into CVS. Problem was that the Search.pm module did a directory change and at the end of the view script a call into Store.pm caused an eval call, that then failed due to relatively library reference ("../lib"). Note that this was read the top line of meta data in the topic, to check if the parent topic itself had a parent.

-- JohnTalintyre - 23 Mar 2002

Verified and it works now, great!

-- PeterThoeny - 22 Mar 2002

Interesting signature sequence here (check the last sig vs the last but one...) - probably SignatureUsesWrongDate (fixed in CVS).

I've been testing my new statistics script against the CVS code, using the old RcsWrap mode, and I get this (using Cygwin Perl 5.6.1-2):

TWiki: Create Usage Statistics
* Executed by a guest or a cron job scheduler
* Statistics for Mar 2002
* Reporting on TWiki.Know web
  - view: 45, save: 2, upload: 1
  - top view:  15 WebHome
  - top contributor:   3 Main.RichardDonkin
Revision 1.4 is already locked by system.
rcs: /twiki/data/Know/WebStatistics.txt,v: revision 1.4 still locked by system
ci: /twiki/data/Know/WebStatistics.txt,v: no lock set by Administrator
  - Topic WebStatistics updated

The rlog output looks like this:

$ rlog WebStatistics.txt
RCS file: WebStatistics.txt,v
Working file: WebStatistics.txt
head: 1.4
branch:
locks: strict
        system: 1.4
access list:
symbolic names:
keyword substitution: kv

I ran the script as Administrator, rather than the Windows 'system' user, but it seems to me that these error messages should not appear...

Also, is there some way to get all topics (using RcsWrap.pm) to be created with the -ko flag, to avoid keyword expansion as discussed in ExpandsRcsKeywordsInText? This would remove the need to add -ko to the RCS commands, though of course that would still be a good idea.

Some error messages that may be of interest, using latest CVS code. This was on editing Main.WebHome as RichardDonkin. Not sure why it is trying to create the RichardDonkin file, but in any case the saveFile and _saveFile routines need to avoid continuing when they do the or warn part. (Have just committed changes to Store.pm and RcsFile.pm to do this, and provide a bit more info through $!).

[Sun Mar 24 10:27:08 2002] preview: Can't create file RichardDonkin
[Sun Mar 24 10:27:08 2002] preview: 1016965628
[Sun Mar 24 10:27:08 2002] preview: print() on closed filehandle TWiki::Store::RcsFile::FILE at ../lib/TWiki/Store/RcsFile.pm line 454.
[Sun Mar 24 10:27:20 2002] save: Can't create file RichardDonkin
[Sun Mar 24 10:27:20 2002] save: 1016965640
[Sun Mar 24 10:27:20 2002] save: print() on closed filehandle TWiki::Store::RcsFile::FILE at ../lib/TWiki/Store/RcsFile.pm line 454.

I've also fixed a couple of minor bugs with taint mode etc, see twiki-dev for details.

-- RichardDonkin - 24 Mar 2002

Above error messages are now fixed - was a problem in setLock in RcsFile.pm. Fix is in CVS.

I had a look at this problem, which I get on editing a topic - sometimes seems not to create revision in RCS, though other times it works OK (all using RcsWrap):

24 Mar 2002 - 11:41 Rcs: /usr/bin/rlog -x,v -h /twiki/data/Test/WebHome.txt,v():

RCS file: /twiki/data/Test/WebHome.txt,v
Working file: WebHome.txt
head: 1.6
branch:
locks: strict
        system: 1.6
access list:
symbolic names:
keyword substitution: kv
total revisions: 6
=============================================================================

24 Mar 2002 - 11:41 Rcs: /usr/bin/rlog -x,v -r1.6 /twiki/data/Test/WebHome.txt,v(): /usr/bin/rlog -x,v -r1.6 /twiki/data/Test/WebHome.txt,v
24 Mar 2002 - 11:41 Store::save date = 2002.01.23.11.00.00
24 Mar 2002 - 11:41 Rcs: /usr/bin/ci -x,v -q -l -m"none" -t-none -w"RichardDonkin" /twiki/data/Test/WebHome.txt():
24 Mar 2002 - 11:41 Rcs: /usr/bin/rcs -x,v -q -i -t-none -kb /twiki/pub/Test/WebHome/TWiki::Store::RcsWrap=HASH(0xa3769cc)( Error: 2):

This seems to be trying to set binary mode on the file, but no attachments are involved so I'm not sure why it's doing this. The setBinary code seemed OK but I didn't look very hard.

-- RichardDonkin - 24 Mar 2002

Richard, thanks for the fixes You say above that a new revision isn't always created. Is this simply the the $editLockTime setting? Changes within this time by the same user don't result in a new revision.

-- JohnTalintyre - 24 Mar 2002

No, I was careful to only test on topics not modified for at least a day, and it still happened. I just re-tested this - in some cases it upped the revision number, and in others it didn't. The Apache log and debug.txt for one where it didn't were:

[Sun Mar 24 21:07:32 2002] save: Use of uninitialized value in pattern match (m//)
at ../lib/TWiki/Store/RcsFile.pm line 370.
Syntax error: "(" unexpected

...........

24 Mar 2002 - 21:07 Rcs: /usr/bin/ci -x,v -q -l -m"none" -t-none -w"RichardDonki
n" /twiki/data/Test/TestTopic6.txt():
24 Mar 2002 - 21:07 Rcs: /usr/bin/rcs -x,v -q -i -t-none -kb /twiki/pub/Test/Tes
tTopic6/TWiki::Store::RcsWrap=HASH(0xa36069c)( Error: 2):

-- RichardDonkin - 24 Mar 2002

I've also found that running the new statistics script (attached to StatisticsUsingLessResources and will soon be in CVS) under ActivePerl generates some warnings from RcsWrap - curiously, switching to Cygwin Perl in the same environment removes these errors and adds the ones shown above, although both Perls are 5.6.1! ActivePerl is build 631, while Cygwin is 5.6.1-2 package from cygwin.com.

* Reporting on TWiki.TWiki web
  - view: 160, save: 1, upload: 0
  - top view:  92 TWikiTutorial
  - top contributor:   1 Main.RichardDonkin
[Mon Mar 25 00:07:40 2002] statistics: Use of uninitialized value in concatenati
on (.) or string at ../lib/TWiki/Store/RcsWrap.pm line 94.
[Mon Mar 25 00:07:40 2002] statistics: Use of uninitialized value in pattern mat
ch (m//) at ../lib/TWiki/Store/RcsWrap.pm line 262.
[Mon Mar 25 00:07:40 2002] statistics: Use of uninitialized value in concatenati
on (.) or string at ../lib/TWiki/Store/RcsWrap.pm line 295.
[Mon Mar 25 00:07:40 2002] statistics: Use of uninitialized value in concatenati
on (.) or string at ../lib/TWiki/Store/RcsWrap.pm line 94.
[Mon Mar 25 00:07:40 2002] statistics: Use of uninitialized value in pattern mat
ch (m//) at ../lib/TWiki/Store/RcsWrap.pm line 369.
  - Topic WebStatistics updated

-- RichardDonkin - 24 Mar 2002

Grr, I am getting the ci: /data/www/twiki/data/TWiki/RCS/WebStatistics.txt,v: no lock set by root error that was mentioned (and fixed??) above. I run httpd as user apache. I have had this problem using both modperl Apache::Registry and Apache::PerlRun. I am running htppd in non-forking mode to track down a bug other than this one. The twiki is the current cvs code as of 8/4/2002 on top of the last TWiki beta, with it setup to put ,v files in the RCS subdirectory. The file in question is owned by apache and locked by apache:

% ps -ef | grep httpd
apache   32038 31010  0 16:09 pts/6    00:00:01 /tools/apache-1.3.24/bin/httpd -X

% ls -l WebStatistics.txt,v 
-r--r--r--    1 apache   apache       1288 Jul 30 18:15 WebStatistics.txt,v

% rlog  WebStatistics.txt,v 
RCS file: WebStatistics.txt,v
Working file: WebStatistics.txt
head: 1.2
branch:
locks: strict
        apache: 1.2
access list:
symbolic names:
keyword substitution: kv
total revisions: 2;     selected revisions: 2
description:
none
----------------------------
revision 1.2    locked by: apache;
date: 2001/08/08 05:53:40;  author: PeterThoeny;  state: Exp;  lines: +3 -2
none
----------------------------
revision 1.1
date: 2000/08/18 06:09:19;  author: PeterThoeny;  state: Exp;
none
=============================================================================

Anybody got any idea why TWiki is trying to lock things as the root user? This is driving me batty.

-- JohnRouillard - 04 Aug 2002