Mercurial Contrib Package

Mercurial is a distributed version control system. This Add-on uses it to replace the default RCS-based backend store in order to create a "Highly Available" TWiki installation without changing user experience.

Mercurial is like CVS in that it allows multiple locations to share the same data, and synchronize changes.

Unlike CVS however, each Mercurial installation is stand-alone. Changes can be pushed or pulled from/to other repositories, with a merging mechanism available should two repositories have conflicting changes.

This means that two TWiki sites can go a long time with no network between them, and when the network is restored, all changes between them can be synchronized and merged in one go. This makes TWiki:Codev.ReadWriteOfflineWiki possible.

Comparison to default RCS backend

Advantages:

• Synchronize multiple TWiki sites for a highly available, loadsharing TWiki experience (even multiplatform!)
• TWiki:Codev.ReadWriteOfflineWiki is now possible
• Backend store uses less space than default RCS

Disadvantages:

• Need to have Mercurial installed
• History lookups currently take longer, code hasn't been optimized yet

Settings

• One line description:
  • Set SHORTDESCRIPTION = Mercurial, a distributed version control system for TWiki

Installation Instructions

Note: You do not need to install anything on the browser to use this add-on. The following instructions are for the administrator who installs the add-on on the server where TWiki is running.

The below instructions are beta quality. They haven't been thoroughly tested. Please test and comment.

• Download the ZIP file from the Add-on Home (see below)
• Unzip MercurialContrib.zip in your twiki installation directory. Content:

  File:	Description:
  lib/TWiki/Storage/Mercurial.pm	Storage Add-on

• Install Mercurial from http://www.selenic.com/mercurial. Now you can choose between a fresh TWiki install or converting your current install. For a fresh install, see further down.
• Convert your TWiki installation. This can be done while TWiki is "live".
  For the conversion from RCS to Mercurial we used cvs20hg. You can find cvs20hg and others here. We had it working on Solaris after a little patching. It will install cleanly on Linux.
  We rely on the fact that TWiki by default looks a lot like a CVS repository. A CVS repository has no RCS directories and all files are stored simply as their "file,v" history file. Furthermore, cvs20hg happily ignores files that aren't "*,v" files.
  1. Open a shell as root. The instructions below are written for bash, not tcsh! For safety, type "bash".
  2. Set HGDIR to a directory next to the twiki directory, and TWIKIDIR to the path of your TWiki root:
     • export HGDIR=/path/to/twiki-hg TWIKIDIR=/path/to/twiki
  3. Create the Mercurial repository:
     • mkdir $HGDIR; hg init $HGDIR
  4. If our RCS files are in the RCS directories then we need to move them out. This operation doesn't impact TWiki:
     • find $TWIKIDIR -type d -name RCS -prune | while read r; do mv -i "$r/*" "$r/.."; rmdir "$r"; done
  5. Then to be sure the RCS history file contains the latest version of the file. Sometimes this isn't the case. Run this as the web user
     • find $TWIKIDIR -name \*,v | while read v; do t=`dirname "$v"`/`basename "$v" ,v`; rcs -l -M -q "$t"; ci -q -l -mnone "$t"; done
  6. Create the CVS root directory in the parent dir of TWiki, to fake that TWiki is a CVS repository. The config file is needed to convince cvs20hg. If you use a different tool you might need to touch more files, but they will probably all be empty.
     • mkdir $TWIKIDIR/../CVSROOT; touch $TWIKIDIR/CVSROOT/../config
  7. Now we run cvs20hg. This will take a while but is the fastest way we could come up with. Note that it can run while TWiki is active, you can run it multiple times and it will add the new changes. Just do not interrupt it, this seems to break the repository you're making
     • cvs20hg $TWIKIDIR/.. `basename $TWIKIDIR` $HGDIR
  8. When the conversion is done you should copy all the files that weren't in RCS
     • Create directory structure: cd $TWIKIDIR; find . -type d | while read d; do mkdir -p "$HGDIR/$d"; done
     • Copy files: find . -type f \! -name \*,v | while read f; do if [ ! -r "$f,v" ]; then cp -fp "$f" "$HGDIR/$f"; fi; done
  9. Tell the new TWiki to use Mercurial: add these to the $HGDIR/lib/LocalSite.cfg. Change the BinDir location to where Mercurial is installed.

     $cfg{StoreImpl} = 'Mercurial';
     $TWiki::cfg{Hg}{BinDir} = '/opt/local/bin';
     $TWiki::cfg{Hg}{initCmd} = "$TWiki::cfg{Hg}{BinDir}/hg add %FILENAME|F%";
     $TWiki::cfg{Hg}{tmpBinaryCmd}  = "$TWiki::cfg{Hg}{BinDir}/hg add %FILENAME|F%";
     $TWiki::cfg{Hg}{ciCmd} = "$TWiki::cfg{Hg}{BinDir}/hg commit -m%COMMENT|U% -u%USERNAME|S% %FILENAME|F%";
     $TWiki::cfg{Hg}{ciDateCmd} = "$TWiki::cfg{Hg}{BinDir}/hg commit -m%COMMENT|U% -u%USERNAME|S% -d%DATE% %FILENAME|F%";
     $TWiki::cfg{Hg}{ciDateCmd} = "$TWiki::cfg{Hg}{BinDir}/hg commit -m%COMMENT|U% -u%USERNAME|S% -d%DATE% %FILENAME|F%";
     $TWiki::cfg{Hg}{histCmd} = "$TWiki::cfg{Hg}{BinDir}/hg log --follow %FILENAME|F%";
     $TWiki::cfg{Hg}{infoCmd} = "$TWiki::cfg{Hg}{BinDir}/hg log --template 'date:{date|hgdate}user: {author}summary: {desc}' -l1 -r%REVISION|N% %FILENAME|F%";
     $TWiki::cfg{Hg}{rlogDateCmd} = "$TWiki::cfg{Hg}{BinDir}/hg --template '{node|short}' --follow -d%DATE|D% %FILENAME|F%";
     $TWiki::cfg{Hg}{diffCmd} = "$TWiki::cfg{Hg}{BinDir}/hg diff -w -B -r%REVISION1|N% -r%REVISION2|N% %FILENAME|F%";
     $TWiki::cfg{Hg}{numRevisions} = "$TWiki::cfg{Hg}{BinDir}/hg log --follow --template '{node|short}:{desc},' %FILENAME|F%";
     $TWiki::cfg{Hg}{moveCmd} = "$TWiki::cfg{Hg}{BinDir}/hg mv %SOURCE% %DEST%";
     $TWiki::cfg{Hg}{copyCmd} = "$TWiki::cfg{Hg}{BinDir}/hg cp %SOURCE% %DEST%";
     $TWiki::cfg{Hg}{filePermission} = 0644;
          

  10. Your new TWiki installation is now at $HGDIR. To activate your new Mercurial installation, we now have to take TWiki down.
      • The easy way is to simply move the TWiki installation directory away: mv $TWIKIDIR $TWIKIDIR.old
  11. Synchronize one last time: (this hasn't been tested, it could be that cvs20hg doesn't like renaming the twiki "cvs module")
      • cvs20hg $TWIKIDIR.old/.. `basename $TWIKIDIR`.old $HGDIR
  12. Make the Mercurial TWiki "live":
      • mv $HGDIR $TWIKIDIR
  13. Test if the installation was successful: Try creating, editing, attaching.
• To create a new installation instead:
  1. Unpack the TWiki tarball where you want it
  2. Run hg init in the TWiki root
  3. Delete all RCS files (you'll lose the history): find . -name \*,v | xargs rm
  4. Add all files to Mercurial: hg add .
  5. Commit everything: hg commit -m"Upstream version"
  6. Edit lib/LocalSite.cfg to add the same settings as under "converting an installation" above.
  7. Tweak TWiki as needed to make it work, follow the regular install instructions
  8. Commit changes: hg commit -m"Initial configuration"
  9. Use TWiki

Special thanks go to TWiki:Main.WoutMertens for sharing his knowledge and contributing to this Add-on.

Synchronizing repositories

To set up automatic synchronization between TWiki repositories, you need to add handlers to your Mercurial configuration. At commit, a script should fire that starts a push to all known sites, in the background.

These handlers are the hooks in Mercurial, you can read more information here.

We have a perl script, hg_push.pl that does a push through ssh.

hg_push.pl Installation Instructions

1. Save hg_push.pl to tools/hg_push.pl under your TWiki installation
2. Make sure the web user can execute hg_push.pl
3. Create an ssh key to allow non interactive login for the web user: ssh-keygen -t dsa -f hg_push_key -N ""
4. You now have 2 files, hg_push_key and hg_push_key.pub.
5. hg_push_key needs to be readable only by the web user, and outside of the space that the web server shows. For example, put it under the .ssh/ subdirectory of the home directory of the web user on each server. .ssh/ needs to have 700 permissions.
6. hg_push_key.pub needs to be appended to .ssh/authorized_keys (create if it doesn't exist yet), again for the web user on each server.
7. Change the host array in hg_push.pl to your hosts
8. Change the $cmd variable in hg_push.pl to suit your enviroment:
   my $cmd = "/opt/local/bin/hg push --ssh \"/opt/local/bin/ossh -i /path/to/hg_push_key\" ssh://UserName@".$host."/TWiki";
   • /opt/local/bin/hg is the location of the Mercurial hg binary
   • /opt/local/bin/ossh is the location of your ssh client
   • Use the path for the ssh key (ie. /home/webuser/.ssh/hg_push_key) after the -i option
   • Change the username in ssh://UserName@ to your web user
9. Create a hgrc file in your .hg directory in the parent of your TWiki and add the following hooks for each host:

   [hooks] 
   changegroup = hg update 
   update = hg commit -m"Push Update"
   commit = /path/to/perl /path/to/TWiki/tools/mercurialcontrib_push.pl 

10. Test it.

Mercurial hooks explained

• changegroup is called after an entire push command was completed.
• commit and update are called after their equivalent hg command

Hg update will stop if there are merges that need to be done. See Known issues

Known issues

• If 2 TWiki sites change the same file at roughly the same time, a merge conflict will arise. We are working on letting this be handled by TWiki using the same framework it uses to handle this situation locally.

• Due to the way Mercurial stores file renames, you currently can't read revisions of a file before it was renamed. To clarify, if a topic A has 2 revisions, and gets renamed to topic B, you currently won't be able to read those 2 revisions where the file was called A. We're not quite sure if we should fix this in this code or in Mercurial itself.

• No optimization/caching has been performed yet.

Future work

• There are some technical details regarding working directories, but it should be possible to only synchronize a subweb if desired.

• It would be interesting to see if using Mercurial will make it easier to upgrade twiki. For example, one could clone the active repository, apply the patch to go from the current baseline version to the next, solve all merge problems, and then extract a changeset to be merged into the "live" TWiki.

• It is likely that this Add-on will work using git instead of Mercurial with little modification, for people who prefer git. It's not something we're pursuing, however.

Contrib Info

Author:	TWiki:Main/TimJanssens
Copyright ©:	2006, TWiki:Main.TimJanssens
License:	GPL (GNU General Public License)
Dependencies:	none
Version:	0.5b
Change History:
27 Apr 2007:	Initial version
Home:	http://TWiki.org/cgi-bin/view/Plugins/MercurialContrib
Feedback:	http://TWiki.org/cgi-bin/view/Plugins/MercurialContribDev
Appraisal:	http://TWiki.org/cgi-bin/view/Plugins/MercurialContribAppraisal

Related Topics: TWikiPreferences

-- TWiki:Main/TimJanssens - 29 Apr 2007