Replication Using Unison
Unison is a file synchronization tool for Unix and Windows. It allows two replicas of a collection of files and directories to be stored on different hosts (or different disks on the same host), modified separately, and then brought up to date by propagating the changes in each replica to the other. (Unison User Manual and Reference Guide)
Why this is useful for TWiki
The goal is to enable multiple TWiki sites each containing the same content, with users of each site making updates on their local site that are replicated to all other sites.
Examples of how this is useful include:
- Laptop users who are frequently disconnected when on planes or some client sites can still contribute to a shared logical TWiki
- Large companies can run multiple replica TWikis to provide better performance and resilience
- Each 'local TWiki site' is an automatic up-to-date backup of all other sites, simplifying disaster recovery at any one site
Useful discussion pages: ReadWriteOfflineWiki
(lots of links), ReplicationTechnologiesForTWiki
Unison - What it does, how to use it
Unison for Windows using PuTTY
- Instructions for syncing via Unison and PuTTY
Other SSH programs for Windows
Unison for TWiki replication
Unison for Backup
- 22 Jul 2005
These are my exploratory working notes, prior to any bundling as a Contrib.
I might never get around to that so feel free to explore Unison yourself and release something. If you want me to continue and document this properly it would probably take me a week - I would need such an initiative funding. Ask your boss.
Issues to be resolved in using Unison
- Invocation method
- Shh vs. socket method - unison supports both
- Unison recommends use of the cygwin ssh tool which rather defeats the object
- Will putty’s shell or GnuWin32’s ssh work?
- Clashing names – UNIX would allow topics FooBar and Foobar to coexist.
- Are backups stored at present? Shouldn’t these be in C:/Documents And Settings/Martin Cleaver/.unison/backups? This directory is not present
- The backup dir does not seem to use hierarchy of files? * The
backup pref does create a hierarchy of files under
~/.unison/backups - I tested this. However, it does mix files from different replicas without having different replica names to distinguish them, so replicas that have the same directory names could have problems. Generally, the backups features are not that great, as this is mainly a sync tool, but they do seem to work OK. --RD
• TWiki data location – will vary according to the user name yet the remote profile will not know what that is.
- Logfile location
- It would be good to be able to reveal unison’s status logs from within TWiki
- Robustness and userfriendlyness
- Can unison be run as a console service?
- Recovery after errors?
- Wildcard questions
- What is the purpose of Harmony, the followon project from the author? Would this be a more suitable tool for us?
- A: Probably not, it's about XML data synchronisation and is still in deep-research phase. --RD
Client-Server vs. Peer to peer
These instructions show client server interaction, with the virtual PC session acting as a long-running server. You could use two PCs or a Windows and a UNIX box.
When unison is run ssh nothing needs to be pre-running on a server side; the initiator runs the server process and the two act as peers.
The instructions I show are for the GUI version, but there is a command line batch version too
I am not clear whether Unison has to have a SSH server for peer-to-peer windows to windows synchronization.
Of particular concern is the issue of naming. I connect our PCs at home in many different ways: WiFi
, Bluetooth, infrared. The IP addresses continually change. And will be different over each transport.
Merging TML files via Unison
From the manual:
5.8 Merging Conflicting Versions
Unison can invoke external programs to merge conflicting versions of a file. The preference merge controls how this program is invoked.
The merge preference may be given once or several times in a preference file (it can also be given on the command line, of course, but this can be a bit awkward because of the spaces and special characters involved). Each instance of the preference looks like this:
merge = ->
The <PATHSPEC;> here has exactly the same format as for the ignore preference (see Section 5.11 [Path specification]). For example, using “Name *.txt” as the <PATHSPEC;> tells Unison that this command should be used whenever a file with extension .txt needs to be merged.
The <MERGECMD> part of the preference specifies what external command should be invoked to merge files at paths matching the <PATHSPEC;>. Within this string, several special substrings are recognized; these will be substituted with appropriate values before invoking a sub-shell to execute the command.
- CURRENT1 is replaced by the name of (a temporary copy of) the local variant of the file.
- CURRENT2 is replaced by the name of a temporary file, into which the contents of the remote variant of the file have been transferred by Unison prior to performing the merge.
- CURRENTARCH is replaced by the name of the backed up copy of the original version of the file (i.e., its state at the end of the last successful run of Unison), if one exists. If no archive exists, then an error is signalled.
- CURRENTARCHOPT is replaced by the name of the backed up copy of the original version of the file (i.e., its state at the end of the last successful run of Unison), if one exists, or the empty string if no archive exists. (Storage of backup copies is controlled by the backup flag.)
- NEW is replaced by the name of a temporary file that Unison expects to be written by the merge program when it finishes, giving the desired new contents of the file.
- PATH is replaced by the path (relative to the roots of the replicas) of the file being merged.
A large number of external merging programs are available. For example, on Unix systems setting the merge preference to
merge = Name *.txt -> diff3 CURRENT1 CURRENTARCH CURRENT2 -m > NEW
will tell Unison to use the external diff3 program for merging. Alternatively, users of emacs may find the following settings convenient:
merge = Name *.txt -> emacs -q --eval ’(ediff-merge-files-with-ancestor
"CURRENT1" "CURRENT2" "CURRENTARCH" nil "NEW")’
I therefore suspect that we could use a command similar to
Merge = Name *.txt -> perl -I..\lib -MTWiki::Merge -e "use Data::Dumper; print Dumper TWiki::Merge::fileMerge(“CURRENT1”, “CURRENT2”, “NEW”);
Where fileMerge is like simpleMerge except that it takes files rather than strings as input.
- Calling external progs: Is there a unified framework for knowing how to merge Word, Excel, etc?
- At worst case we have to upload the conflicting files and let users sort them out. Even if we resolve it for the users we have to ensure both the remote and local files get uploaded. Note we would need anything dumped into the pub directory to be attached.
- Multiple syncs through a network of machines
- Should we be merging the topics or merging the deltas stored in the history? i.e. Sync topics vs. history (,v)
Reports from the trenches
I've been using unison for about a week to keep my windows machine synched with a linux server. Only the server is running twiki, the windows workstation is just used (at this point) to look at and manage the differences between the running twiki and svn dakar.
The general process I'm using is: 1) ssh to server, 2) svn up, 3) unison sync to local machine, 4) tortise-svn diff/merge as necessary, 5) unison sync to server.
Only significant problem so far is that windows doesn't support symlinks so when I run
on the server to install the plugins unison keeps wanting to update them, and then fails (which is annoying because there too many to easily add to the ignore list).
- 04 Aug 2005
Matt, glad it worked for you! Which hosting provider are you using?
- 04 Aug 2005
I use unison in a quite simple way to do backup for my various TWiki installations and it just works fine. Im using unix-to-unix sync via ssh.
- 07 Aug 2005