Tags:
create new tag
, view all tags

Thesis

There's been a fair amount of chat - mainly on TWikiIRC - regarding replacement of the RCS module in TWiki with something more functional/standard/future proof. Subversion (SVN) has been discussed several times as a possible platform technology. This topic is a place to discuss this further.

Subversion is now a stable technology. It is even used as the CM tool for TWiki, replacing CVS. It's fast, easy and intuitive to use, and attractive GUI clients (both Windows and cross-platform) are starting to emerge. Additionally and powerfully it supports the WebDAV protocol, in theory allowing other versioning WebDAV clients (e.g. Microsoft Office, Open Office) to operate harmoniously on the same data.

Replacing RCS with subversion could be as simple as "rip out RCS and plug in subversion" process, but doing it that way would waste much of the power of subversion. It's not even a good starting point, as all the existing RCS baggage in StoreDotPm would just layer on top of, and conflict with, the (better) SVN solutions. For example, SVN provides meta-data storage using a much cleaner paradigm than the TWiki+RCS. The TWiki locking model is orthogonal to the SVN/WebDAV locking model. SVN data is simply a versioned file hierarchy, where TWiki uses typed files in a typed directory structure.

We also want to make sure that TWiki's use of a subversion repository is consistent with other WebDAV clients. There should be no problem with doing this as long as TWiki meta-data is moved into SVN properties. The existence of those properties would allow TWiki to differentiate between "topics" (.txt files) and "attachments" (everything else).

So, what are the negatives? Well, you need a subversion server. Fortunately these are pretty trivial to set up. There are actually two flavours of subversion server, the "subversion" flavour and the "svn_dav" (webdav apache module) flavour. For full integration with other applications the svn_dav solution is required. This module only works with Apache 2, which given TWiki's problems with Apache 2 can be seen as a negative. And what about support with other webservers (e.g. IIS)? Subversion properties are stored in XML away from the actual files, so we lose the ability to have topic text and meta-data in a single (text) file. (I consider this to be completely overrated; we already split meta-data - specifically the revision history - into a separate meta-data file i.e. the ,v file in RCS). Some people have encountered problems installing subversion, because of its dependencies (such as that on XML). It just hasn't been around long enough yet for prebuilt binaries to be widely available.

From the perspective of the TWiki code, it would be more than StoreDotPm that would have to change. Some code outside STore uses the fact that TWiki topics are organised into files in directories to perform operations such as searches - for example, SearchDotPm. The complexity of fixing this code so it acknowledges and works with a consistent interface to the data store is probably this biggest challenge laid down by this request.

Contributors (not necessarily to this text directly, but to the discussions that led to it. Apologies if I missed you.)

CrawfordCurrie
SvenDowideit
MS

(see also TWikiWithSubversion for another conversation about this topic)

Discussion

As long as there is a standard "API" through which the TWiki core and the plugins connect to the store, the only problem I see with this is that (if SVN is to be the "default" store), some system administrators will probably have an issue with running an extra daemon for the store, the idea being the less servers you have running the less risk you run of security problems.

It would be cool if you could have a pluggable store system so that you could have several kinds of stores running at once ... I can't think of a particularly compelling reason why one might want this, but perhaps for security/data-sharing reasons one might want to have one web set so that it is stored in the traditional way on a shared filesystem, and another web set up on a private SVN.

-- ChristopherRued - 08 Aug 2004

The reason to have pluggable stores is that not all software is available everywhere. RCS probably is available everywhere, but .. bleah. SVN should be available everywhere, but might not be. It seems likely that people will want SQL based stores. SQL wouldn't bring much to TWiki since TWiki already implements most of what SQL offers (searching, ...). The point is, if a storage API existed, anyone with an itch could create their own store. It doesn't need to be over-designed. The interface would probably consist of a Store object for wrapping the API, and a Node object for wrapping objects. For that matter, it would probably look like a slimmed down version of Perl's DBI.

Of course, I'm just wanking at this point since I've got no code to show.

-- RobertDeForest - 26 Oct 2004

Not only code inside twiki. Also at least one plugin I know (XpTrackerPlugin) use the fact that topics are in directories.

One posible "solution" to this is to always have the "latest revision" of a topic checked out in /data, and twiki reading from there (as is today).

-- RafaelAlvarez - 27 Oct 2004

And ActionTrackerPlugin, and DBCacheContrib and everything that depends on it. However if provided with a robust, featureful interface to the Store via TWiki, I'd be delighted to rewrite these plugins as appropriate.

LocationLocationLocation goes a long way towards a Store API. ReleaseLocksOnSave goes further.

-- CrawfordCurrie - 27 Oct 2004

OK, as of today I'd say the DevelopBranch has collected all the filesystem sensitive code into the store module and its children, so it should be possible to implement the Store API in subversion without breaking the rest of the system. Of course plugins that don't use the API will not work, but that's another step away.

I'm changing this topic to a ChangeProposal so we can track what happens with this. It's probably our first change proposal!

-- CrawfordCurrie - 11 Jan 2005

This comment is slightly off topic but involves subversion.

May I suggest that when implementing Subversion (or any backend store for that matter) that provision be made to allow the arbitary retrieval of a particular version of the page or 'tag' if the back-end does not support repository versions?

This would allow some form of url parameter to be added to Twiki urls indicating which version/tag of the page you want. Under subversion, this would allow us to view the state of the entire site at a particular version/tag, if we appended the version/tag id to each url.

The reason I ask for this is that it would lead quite nicely to being able to 'digitally sign' a particular page, saying the web (or at least some pages) of this version are definitive.

A digitally signed page (by whatever means) would then store the repository version/tagin the topic meta data.

Such a page, when rendered, would have all links on the page (not in the left bar or heading/footing) include the version number in them during rendering - allowing you to navigate around the 'signed' page set.

When a Topic is shown using a 'version' id, some visible indicator would be shown on the page somewhere that you are not necessarily viewing the 'latest' page.

Once a topic is edited, it loses its signed status - unless re-signed - once again, rendering with links to 'latest'.

I think it has already been discussed simply having the repository for the topics and Twiki runs externally, checking out and updating the entire repository. Would allow multiple twikis all referencing the same dataset.

I am excited by this whole idea.

-- LyallPearce - 03 Feb 2005

I think that using subversion would give us several benefits besaids the better meta-data handling:

  1. We can use either the current search implementation or a Plucene based one to quickly search the site.
  2. Use svn stat to determine the files that changed
  3. Fast diff for changes
  4. Multiple sites can be stored in the same SVN repository, thus easing the backup of twiki-farms

(more to be added)

(removed some comments because they where done after 20 hours of sleep deprivation) -- RafaelAlvarez - 04 Feb 2005

Using a server with a SVN repository it is clear that while the SVN is active the entire file system seems frozen and it is not exactly fast.

I think SVN performance as I know it would not work at all for TWiki. It would be too dead slow and now allow concurrent access.

-- KennethLavrsen - 30 Jun 2006

I dispute "it is clear", but then I also dispute "a database backend will be better" too: until we try it, we'll never know.

-- SvenDowideit - 01 Jul 2006

Yet another variant: how about using Git as store? Or at leat as the RCS replacement? Big advantage of Git: distributed - allows me to edit wiki while disconnected on airplane.

-- AndyGlew - 12 Sep 2006

See GitAsStore

-- AndyGlew - 12 Sep 2006

Andy - thats no different from svk - a wrapper around svn that uses svn as its backend - would be nice if those damned versioning system developers would get together and make one unified API big grin

-- SvenDowideit - 12 Sep 2006

I think the real problem with a unified API for version control has been that the very concept of a version control system has been making major changes. Git is the first major example of a content based VC system; Monotone was an earlier example. they implement a superset of what old style systems like RCS, CVS, SVN, and SVK do.

-- AndyGlew - 05 Oct 2006

Oh, ad I want to emphasize the advantaes of using Git or the like over using SVN: modern VC systems like Git are decentralized. They allow people to make private copies of the world, and check things into these private copies. They also then allow the diverged VC repositories to be merged.

SVN is an old style centralized VC system. SVK adds a distributed veneer to SVN, but not 100% successfully.

If you go to a distrubuted VC sstem you are that much closer to having a ReadWriteOfflineWiki

-- AndyGlew - 05 Oct 2006

Edit | Attach | Watch | Print version | History: r19 < r18 < r17 < r16 < r15 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r19 - 2007-01-16 - TobyCabot
 
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2017 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.