RcsSync Add-On

This is the result from my internship I did for my last year at the KHK ( http://www.khk.be ) in Belgium. The first project i started was TWiki:Plugins.MercurialContrib but this was quite a hassle to get it up and running with lots of configuration. This is another way of synchronizing multiple TWiki servers to make a high available TWiki with much less configuration to be done

For this project i used the TWiki way, using bin scripts to communicate between servers. It supports HTTP and HTTPS since I used LWP from perl. It inherites a large part of the RcsWrap code and should therefore be easy maintainable. Also TWiki::Merge is used for merging changes (if needed!).

I also would like to thank my mentor TWiki:Main.WoutMertens who has supported the development for this during and after my internship

Feel free to comment and make changes, when I started working on this i hadn't very much experience with Perl and you will notice this sometimes in the code

Tests have been run only with two servers at the moment but it should work with more.

The synchronization is started after each save. Lease files are synchronized as well.

Synchronization process

1. Get local checksum of the repository before save
2. Save local topic to file
3. Get checksum and lock topic on all sites in TWikiMirrors
4. Compare for each mirror's checksum against the local one. If it is different we pull the remote repository and do a merge for the changes (repeat this for each host)
5. Push the updated files to the mirrors
6. Return the topic with or without Merge conflicts

Merge

When two repositories have a different checksum, their latest version, at least, isn't equal. At the moment we need to rebuild the whole repository in chronological order.

The first loop is that of the equal revisions. Here we compare the content of both revisions and save one of the two. Each time this is done we set the current revision as the youngest ancestor. If they are found not to be equal we move one to compare the remaining revisions in a different way. For looping the remaining revisions, those who are not the same we save the oldest revision and remove it from the remaining revisions. The last step is a 3 way merge of the ancestor and the yougest version of each repository ( local & remote ).

To look if revisions are equal a checkout is needed at this moment. This is because I don't know a way to compare them without. I was thinking about saving the checksum from the .txt as a tag for that revision. Please comment

Todo

• Check if a host in the TWikiMirror list is the same as the server where the script is running on and delete it from the array. This way TWikiMirrors can be distributed also.
• Create some way of comparing revisions without a checkout. See Merge.
• Test with 3 or more servers (should be quite ok)
• Clean up code
• Get only the revisions the local repository doesn't have in 1 connection
• Create a script to start synchronizing a web or whole site with mirrors (Now they are only synchronized if a topic is saved.)

Usage

• Just like normal usage but look out for Merge conflicts announces in the top of the page

Add-On Installation Instructions

Note: You do not need to install anything on the browser to use this add-on. The following instructions are for the administrator who installs the add-on on the server where TWiki is running.

• Download the ZIP file from the Add-on Home (see below)
• Unzip RcsSyncAddOn.zip in your twiki installation directory. Content:

  File:	Description:
  data/TWiki/RcsSyncAddOn.txt	Add-on topic
  data/TWiki/RcsSyncAddOn.txt,v	Add-on topic repository
  data/Main/TWikiMirrors.txt	Add-on mirror list topic
  data/Main/TWikiMirrors.txt,v	Add-on mirror list repository
  bin/sync	Add-on script
  lib/TWiki/UI/Sync.pm	Add-on UI
  lib/TWiki/Store/RcsSync.pm	Add-on Store

• Edit LocalSite.cfg and add or change $TWiki::cfg{StoreImpl} = 'RcsSync';
• Edit data/Main/TWikiMirrors.txt and use a newline for each mirror. Start with MIRROR: before the url. At the moment you can't put your local url in here. See Todo

Add-On Info

• Set SHORTDESCRIPTION = Synchronize multiple TWiki sites

Add-on Author:	TWiki:Main.TimJanssens
Copyright:	© 2006, TWiki:Main.TimJanssens
License:	GPL (GNU General Public License)
Add-on Version:	28 Jun 2007 (V0.600)
Change History:
28 Jun 2007:	Initial version
TWiki Dependency:	$TWiki::Store::RcsWrap, $TWiki::Merge
CPAN Dependencies:	LWP, RCS
Other Dependencies:	none
Perl Version:	5.8.8
Add-on Home:	http://TWiki.org/cgi-bin/view/Plugins/RcsSyncAddOn
Feedback:	http://TWiki.org/cgi-bin/view/Plugins/RcsSyncAddOnDev
Appraisal:	http://TWiki.org/cgi-bin/view/Plugins/RcsSyncAddOnAppraisal

Related Topic: TWikiAddOns

-- TWiki:Main/TimJanssens - 28 Jun 2007