We (at PassEdge) are moving to CVS for our source code system, and there are some nice features (like the 'concurrent' in CVS) which could make TWiki a little better.
I think there might be some Wiki variants out there that use CVS on the back end, but I am not quite the PERL guru to tackle such a job.
- 17 Feb 2000
What user interface do you envision when you are saving a topic that has been edited and saved by somebody else? I am not familiar with CVS, somebody could help out explaining the main features?
TWiki could be made more modular at the back end, so that the version control tool could be set to RCS
- 18 Feb 2000
If figure using the VCS::CVS CPAN
module it should be easy to quickly hack and replace RCS
functionality with CVS. The tricky
thing is getting things like concurrent edits working.
For that I figure just forcing the last person to commit an edit to reedit the conflicting bits if any.
I can envision that using CVS concurrentence would make things like email submissions and maybe AutoSave
features easier to design.
- 20 Feb 2000
The user interface to merge concurrent edits must be carfully designed.
How about showing the conflict using different colors in preview? The [ Save ] button would change to [ Merge Conflicts ] , which in turn shows the merge screen. The tricky part is how to present the merge screen. Any ideas?
- 18 Feb 2000
I'm working on the CVS conversion at the moment since there seems to be some need. Here are some notes of my design thoughts so far:
- It's going to be rather invasive.
- Seperate working directories on a per User basis, leaving the main data/$WebName still there. This allows and gives:
- Modify least amount of code as possible
- Easiest way to handle multilpe checkouts
- Still have to give some thought on how to handle the race condition occuring when multiple commits occur at the same time.
- preview defaults to running 'cvs update' before display.
- 'out-dating' or deleting a revision is a little trick in cvs, particularily with multiple checkouts. Going to initially ignore commands related to this:
- delRev - delete last revision
- repRev - replace revision save
- When TWiki itself gets put into CVS, some fiddling with .cvsignore for the data directory, plus a transparent way to create it is probably worth while. Otherwise you get ,v,v files.
- Fiddling with the CVSROOT/module file on a per Web basis to easy checkout of data/Web/file into UserNameWorkSpace.
- Any further serious work will require some core code rewrites.
- Consider the use for XML data files for adding things like: <info> $Id: TWikiWithCVS.txt,v 1.29 2002/08/08 09:44:00 MartinCleaver Exp nobody $ $Author: MartinCleaver $ $Date: 2002/08/08 09:44:00 $ $Revision: 1.29 $ </info> <payload> .... </payload> to remove the need for 'rlog'
- 15 Apr 2000
Tromp tromp -- I'm assuming the edit is cold after ten minutes, Nicholas.
The original Wiki
gets huge numbers of edits on small numbers of topics all the time, once in a while you get tromped on and have to reload & re-edit, but for the volume and the number of topics it's not as bad as you might think. I'll bet a lot of our percieved need for CVS is based on the lock warning in TWiki.
I still like the idea you expressed above (force the users to commit) but I bet we don't need CVS for that. See BetterFileLocking
for other views on this.
- 15 Apr 2000
This is always going to be an issue in revision control systems for multiple users. However, I don't see how improved locking will help, since it's simply control in the time dimension. I would prefer to force the current author or a following author to deal with the clash rather than possible have to rewrite a 30 mintue change to the document.
So I think a better underlying revision control mechanism is the way to go. Anyway I'm working on it at the moment. We'll see what I come up with. 8)
- 15 Apr 2000
I was thinking we could just put together a conflict format and anything that CVS kicks back a conflict to gets resolved by the last submitter who generated the conflicts. Maybe highlight the conflicts in yellow and make a little marker that it needs to be resolved.
Then maybe give the submitter the option to disregard their last changes, or resolve the conflicts and then resubmitt. This would be the point that I would maybe put an absolute file lock on the page..for maybe 30 minutes...
Others would get a message saying: ->Conflicts being resolved.
The conflicts should never get too big unless there are revs of a manual or something like that...or spelling, which is no biggy.
CVS has the ability to put watchers on files...maybe that would be a good way to track and warn people who go to edit a page that others are doing the same thing, then they can natural regulate what they edit, rethink or just add on like we do here. We would not really be generating any conflicts just tacking on new comments as we go...this type of conversation is supported by Slashcode anyway...
- 16 Apr 2000
I applaude your efforts to get TWiki running with CVS. I still have to familiarize myself with CVS, the Open Source Development with CVS
book I ordered at Barnes&Noble arrived today (BTW, off topic, I stopped ordering from Amazon last December because of the one click order patent dispute)
I would like to give the user the choice of back end storage. TWiki should be modularized to have a back end plug in of your choice:
- RCS and flat file
- CVS and flat file
This was proposed some time ago in ModularizationAndCommonTwiky
The TWiki core should have some generic functionality, may be a switch that distinguishes between a back end that can / cannot handle simultaneous topic updates.
- 19 Apr 2000
I believe that Twiki should use the cvs pardigm
to eliminate locks. The notable CVS characteristics are:
No locks ever
User attempting to commit resolves all conflicts BEFORE commiting
Merges are done by applying the diffs to bring the users version up to date, while still keeping thier changes.
Version that the user is editing is recorded so the correct patches can be applied.
To implement this record the VERSION of the page the user is editing (in a hidden field). When saving (and maybe previewing), if a newer version has happened:
- You warn them , and optionally show the diff(s) from what they originally saw.
- Merge the changes by applying the patches
- Let the user check the applied changes, and resolve any conflicts.
- Once ready user hits save again to start cycle again.
(cvs) has a very good model for implementing good concurrency support, even if you do not support cvs as the back end. Note that CVS uses RCS
for its file storage, cvs just provides a higher level abstraction that supports the extra features, I believe early implementations were just shell scripts driving rcs.
- 24 Apr 2000
Just to note, I've been working on a CVS version of twiki. Although I haven't done much with it in the last week and a half. I do have a working Proof of Concept version which until we get cvs for development I can give out to people to have a look see. (Definitely not production readily.)
Anyway here are some notes:
- Multiple edit windows on the same file will cause problems
- Not existant Web not dealt with yet
- Category data not handled yet
- No repRev handling
- TWikiAttachment not handled well
- Updates: need to parse information returned.
- No -W for cvs commit, implies the need to store author information some other way
- No checking for conflicts on final save. Need to parse commit messages with process pipes in order to determine if problems.
- Some multi-thread logic will be need to sort pushing the final update into the main read location
- Old UserWorkSpace revisions not released on 'save', should delete as might cause issues when re-editing with a later version.
- edit checkouts a copy of current revision at time of edit into UserWorkSpace
- preview saves to working copy and updates working copy from repository. Conflicts will be viewed at this stage.
- save needs to update main version after commit in some atomic manner.
- Which the revision system is more complex, there is an increase need to process the information returned form the cvs process.
- Too much magic (number) type stuff happens in the current code all over the place. Next step would be to design in some nice application logic layering.
Although I haven't had time to write any more code, I've been thinking a little about the whole exercise. I have a feeling in the long run it might be better to hack something like diff3 into a perl module and do it all in native perl modules. Rather than the system calls that are currently used.
On another note, I thin before the CVS version becomes production ready, at least the file IO part needs to be modulised with a fixed API. Maybe a perl object module or something. See ModularizationAndCommonTwiky
for more detail.
- 24 Apr 2000
Just to note I've decided that it's quite tricky to use CVS instead of RCS
internally. Basically this is due to the way TWiki current uses revision replacements and the username option on the rcs command line. The 'web server user' is seperate from the twiki user and cvs provides no way of providing that information on the command line in the same manner rcs does.
Of course we could store the information inplace in the file along with other revision information as per discussions elsewhere in TWiki.Codev. (Particularily TWikiXML
However after some consideration, including the thoughts at LoadTesting
, I've decided that its working coming up with a native perl version control mechanism. Rather than calling rcs all the time. At this stage it's work in progress.
One thing I'm hunting for at the moment is the RCS
spec. The best you may be able to get is
man 5 rcsfile
- 11 May 2000
With respect to the LoadTesting
discussion, CVS could also be used to speedup TWiki. CVS is a lot heavier than RCS
, so trying to simply replace RCS
with CVS without changing the number of file operations will only make TWiki slower. Instead, it seems way more useful to me to provide a CVS specific framework.
- When a user logs in to TWiki, a cvs update is performed on his private directory, giving him the latest copies of all pages.
- He can now directly edit all these files, and potentially copy them to another location for offline editing.
- Only after he is done editing and happy with all the new or modified documents, he performs a "sync", which does the following cvs steps: cvs update (get new changes from others), merge (resolve possible conflicts in the documentation), cvs commit. This instantly gives him the latest pages, plus submits his changes to the repository.
Except for allowing offline editing, this also lowers the server load quite a lot.
- 22 May 2000
Actually it doesn't and probably wont if the cvs server is remote. Consider that views need to know they are in sync with the master cvs version. Since the main web version is basically a read-only checkout. So every read (currently with the pre-alpha version) involves at least calls to 3-4.
Also current the write mechanism involves an cvs update when previewing and just before saving. The other very trick problem is dealing with the race condition involved in committing. Effectively it's not really a problem since you can notice the conflict after it;s been committed and re-edit, but it's not a nice way to do things.
At the moment I'm working on reducing any system call to rcs on viewing, then I might do some testing to see what sort of difference it makes. I can then get an idea of how much extra work cvs might cause.
- 22 May 2000
First of, to handle the RCS
viewing, etc., this may be an interesting link for you:
Next, I will argue that your assessment of CVS overhead is wrong
- To view a file, the only thing needed is loading it from disk (from a read-only cvs checkout). We can assume that these files always exist, and are always up-to-date, so no additional checkouts or updates are needed (see the next points). We can pretty safely assume that the file contains all data needed to display it, especially if the XML data in it also contains $Id: TWikiWithCVS.txt,v 1.30 2002/08/08 11:30:21 PeterKlausner Exp nobody $ or other identifiers that provide automatic updated revision data.
- When editing files through the normal web interface, a checkout of the file is done to a private workspace for the editor. The previous displays that edited version, without tying to update it. Only when the user presses Save, we perform an update and subsequent commit. If the update or commit fails, the user moves back into the editor, but now with the merging file displayed. You can update the read-only copy of the fiin two ways:
- Create a file similar to the .lock file, e.g. WebHome.update. The next time a view is done on this page, the view function first needs to perform on update on the file. As this update is only done when the file has actually changed, it can't possibly affect the system load a lot.
- Instantly initiate the update of the read-only version from the edit tool, right after the commit of the new version. This is the least preferred approach though as 1) someone might view the file before or (worse) during the update. 2) Committing files that were edited offline can't work this way.
- If someone wants a copy for offline editing, he can simply checkout all twiki pages, and use them on his system. Either by using a web-based twiki which is setup knowing that it doesn't have a CVS connection, and simply edits the text files without trying to store them in CVS. Later, when that user syncs his pages with those on the server (and merges and commits all his changes), a rather simple CVS trigger script can be used to create the .update files described above, and ensure that the web view is constantly in sync. (Trigger scripts are executed every time someone attempts to commit a file. They are used for automatic file verification, e.g. making sure that all functions are documented before a commit is allowed. Such a trigger script could easily be used to create the .update file instead though.)
- A simple cvs update locking mechanism may need to be added (e.g. taking a lock on the .update file), so that only one view can update some page at the same time. I'm not certain whether CVS protects against racing conditions of two simultanious updates on the same file.
Overall this approach allows both online and offline TWiki usage, and lowers the system load compared to the current implementation. Adding a selection mechanism to defined which document tree you are editing, so that different versions of the pages can be maintained (e.g. for different product versions) should be quite easy as well.
- 24 May 2000
Interesting idea. I hadn't thought of using the RCS
auto-fill tags to save the need to query the cvs repository for the version details. The issue arises of encoding the author information, and replacing versions.
Rcs is used in a way which is difficult to repeat with cvs. It's probably possible with cvs admin, but gets messy very quickly.
Currently with my proof of concept version of the cvs code I do updates at every preview. I think that it is certainly worth making that a server option and
As for cvs commit locking it occurs and has to be dealt with. Essential involves parsing the information returned from cvs.
The real painful thing about doing this cvs version is having to parse all this 'command line' returned information. Which might not be stable from version to version and OS to OS. Leaves me wishing for an API. As much as I hate to copy current functionality, to create something easily extensible I might have too.
The thing about using cvs is that the cvs repository might not be on the same machine as the web server. This is one of the assumptions I'm used to develop the cvs code I have at the moment.
The great thing about that is, since http is stateless, it doesn't matter which web server you use to access and edit the information. As long as that web server has cvs access and uses the same logic to update the data.
In the case of server-one editing and updating the repository it course can updates its view checked out version. So what happens if server-one edits and updates the cvs repository? How will server-two know about it? One solution is probably to use watch tiggers and things like that. The design/logic and code for that quick becomes complicated though.
I'm not interested in Offline editing at the moment. That requires some thought into what sort of update logic we want to use.
Actually at the moment I've removed two system calls per view and probably can remove the last, by directly parsing the rcs file. In fact Kelvin is writing a rcs parser So cvs wont be a better performer, just add further functionality.
NOTE: That rcs.interactive page is interesting, but the code it links too no longer exists.
- 24 May 2000
- 15 Jul 2000 (restored from backup)
- we use CVS at work already, i'd like to have everything in one repository
- which was why at ACUS (Unisys) i did a port to Microsoft Visual Source Safe
- i reckon that we could use pserver access to do multi-server / distributed TWiki
- I don't think its a huge change (based on the VSS port only taking a few days work)
- Tagging, branching and stuff sounds cool
- 06 Jan 2002
Common repository: It should
be possible to integrate an RCS
repository into CVS. After all, RCS
is CVS' core engine.
Though I'm somewhat at a loss why you'd want to integrate them.
The only purpose that I can find for this kind of integration is the possibility to attach common tags to a TWiki and its associated software... but rolling back a TWiki to a previous tagged revision isn't going to be very useful. You'll have the tags but no real way of making use of them.
_Comment by AndyGlew: actually, as I posted today (Apr 17), I am already checking the twiki database, both source code and data files (topic pages) into CVS. I have "rolled back" the entire twiki database twice in the past week, prior to making things public. I have replicated an earlier version of the twiki installation, and I have branched at least once. I'm doing that from the filesystem, sure; maybe one day I will provide a twiki interface. But just having the twiki filesin CVS gives me the ability to do everything
that CVS does, from the filesystem._
Pserver for distributed TWiki: Not very useful. It just replaces the http server with the pserver, with all existing advantages and disadvantages. I don't see why we should change a running system here.
Not a huge change: Agreed.
Tagging, branching and stuff: this is already in RCS
No need here.
In summary: I don't see a compelling case for such a port.
Do you have a use case that RCS
- 07 Jan 2002
Merged above two posts by Sven and Joachim from TwikiWithCVS (topic with almost identical spelling)
- 08 Jan 2002
Thanks Peter - mabe we should make the Topic names non case sensitive (that way TwikiWithCVS is the same page as TwikiwithCvs etc) ala FileSystemNameClash
Tagging a version is not only for rolling back.....
most of the work that I do with TWiki moves towards using TWiki for process and information management. eg Requirements / Bug tracking. When you sign off on a set of requirements (one requirement per TwikiTopic
) you need to tag those requirements (and their version) so that you can see what contract deviations you have..
also, PServer support for the backend does not mean that you exchange the pserver for the http server. thats only the first step. for distributed wiki you have multiple cvs repositories, either peer to peer, or with a master server. that way you get offline TWiki for free using cvsup... and so on..
at the moment i have a TWiki at home, a TWiki on my notebook, and a TWiki at work. it would be more sensible if I had only one TWiki with multiple repositories at each location (i don;t always take my notebook home, and i'm not always online), eachof which oportunisticaly syncs & merges.
even without these reasons, doing so would increase the flexibility of the TWiki, further decoupling the Data from the UserInterface
- 08 Jan 2002
Hmm... I'm too unfamiliar with that contracting stuff to see what the requirements for the versioning system are. Could you elaborate on this?
Multiple CVS repositories: Most emphatically no
! CVS does not
give you that!
It would be easy enough to set up multiple CVS servers, but they won't exchange data. At least not automatically - you'd still have to organize the data exchange - and that's exactly what would be required for the current httpd/RCS setup.
Improving flexibility: Yes, adding another backend would make TWiki more flexible in this respect. But that's something that can be achieved with other means (see GetRidOfRcs
for just one example).
- 09 Jan 2002
Requirements management is one of the key tools in software development. its not jus for contract work - all projects (including software) need a way of tracking what the end user wants, what gets delivered, and what differences there are from what was origninally asked for. When you hit a milestone, or a reporting point, you want to keep a tag of the state of the project at that point. No different from tagging the source in the repository when you pass a version to testing..
stop being pedantic. a CVS pserver gives you remote access to the raw files, and provides versioning. this means that you can mirror that repository elsewhere. therfore you have the ablility to have multiple repositories.
- 09 Jan 2002
Sven, I'm not being pedantic, and I beg you to avoid ad hominem arguments.
I just want to avoid doing stuff that sounds great, but doesn't offer a real improvement when looking closer at it.
Requirements management is important - agreed.
But I don't see how this relates to the RCS
-vs-CVS issue. RCS
can do this type of thing easily. CVS' tagging facilities are mapped one-to-one to RCS
' tagging facilities, so CVS buys you nothing here.
I also simply don't see how pserver would help with replication.
I have taken a look at the CVS pages.
It's as I expected: interesting but not currently useful for us.
Multiple CVS repositories are recommended for getting around "flaky networks", and even then you get into trouble if there's a merge conflict (since the local repository doesn't have the master copy until the next update). TWiki already has this, BTW - you can always request the raw ,v file from TWiki via HTTP.
The CVS developer pages say that they are thinking about something more interesting (true automatic updates between CVS repositories), and this might indeed prove useful... but it doesn't seem to have gone beyond the proof-of-concept stage yet.
Another problem I'd like to see is the question whether we should rely on CVS' replication facilities even if they were already available. One of TWiki's goals is that TWiki should work for multiple versioning tools, at least in the long term. If we use multi-site capabilities of one tool, this creates a new dependency, at least until multi-site capabilities have become a common feature of versioning tools.
Summary of my current thinking on the issue:
- I still don't see how a CVS backend helps with release tagging. I'd be open to a serious argument here.
- CVS' replication facilities don't give TWiki a serious advantage, and create new dependencies at this time. We might want to wait for the versioning tools to catch up, or we might roll our own replication facilities. Either way, this it not a case for integrating CVS right now.
- CVS integration might still be useful, if only to clean up the interface of Store.pm. It would also be helpful for political reasons (simply because CVS administrators will oppose a tool that doesn't use CVS).
I wouldn't consider this worth the effort, but YMMV.
- 10 Jan 2002
Using CVS for TWiki replication is non-trival. The whole IO sub-system, searching particularly, would need to be re-written. Given this, there are probably much better ways than using CVS/pserver - like XML
-RPC or similar.
I think the most important idea when looking at the clustering of TWiki is that reads-are-frequent and writes-are-rare.
- 10 Jan 2002
I think these are good reasons to implement CVS for TWiki:
- Concurrency: Could be done like this:
- When editing, first do a
cvs update in a temporary edit.$USER folder and present the user with the text
- When committing, you need to run
cvs update first. CVS will not allow you to commit a version that is not updated to the latest.
- If there are update conflicts, the user can be presented with the raw text and be asked to sort it out or the conflicts could be handled one-by-one.
- Repeat conflict resolving until able to commit
- This way, users can always edit, and tools like the CommentPlugin or mail gateways will be far easier to implement.
- One CVS repository and one copy for each cluster node
- Editing is only possible if the link to the repository is up
- Alternatively, edits are possible and automatically merged on next connect, and conflicts are not committed but notified to the user on his next visit
- General updates can be run at regular intervals, or a mail notification per topic change could be set up
- Small step:
- CVS is a wrapper around RCS, providing all the locking and robustness necessary to do its job.
- It's only a small step to implement this versus rolling your own.
- Standard tool: It's the Unix way to use tools that do one thing very good instead of reinventing the wheel the whole time. Using CVS instead of rolling your own does that.
- 07 Aug 2002
I strongly agree
My train of thought:
- TWiki is good at collaboration
- It is good for distributed teams
- Distributed often means: nomadic
- Nomadic often means: offline or behind firewalls or just plain slow
- To really compete with Lotus Notes or GrooveNetworks, you need a ReadWriteOfflineWiki and a DistributedTWiki. This means:
- conflict detection
- conflict resolution
- Clearly, this smells like CVS
How could that work?
Like Wout's concurrency + clustering scenario:
- The TWiki server should work against the locally checked out *.txt files.
- The actual check-in should by tried, when the lock expires
- The access to the repository is left to cvs - locally, pserver, ssh, you shouldn't care.
- The .changes files could be parsed to run selective updates with a tight schedule. (A simple global update means reading the complete repository, even if there is no single change!)
On a TWiki server with local repository, some behaviour changes:
- The version number would not increase before the lock expires. I guess TWiki's odd RCS locking is meant to fix this, right?
- Conflicting changes could net get committed; you would have to re-edit the page, fix CVS' merge conflict markers and save again, like Wout described.
A "remote" TWiki server additionally
- Cannot complete check-ins when offline; this could be handled like a conflict
- Operations which need access to previous revision work only, while the repository is online.
- 07 Aug 2002
Incidently, this strategy of holding back changes until the lock expires is what we used in TWiki's MailInAddOn
outlines the current architecture. Once the new release names are agreed, I will produce an architecture diagram showing swappable modules for items such as the backend. This process should in-turn highlight any inter-dependencies such as search's need to examine the filesystem.
You might also want to look at the discussion at CpanPerlModulesRequirement
if you have not already done so.
- 08 Aug 2002
Oops - this page reaches Netscape's textarea size limit.
(masked the $Id: tags for no good use : )
Continued at ReadWriteOfflineWiki
- 08 Aug 2002
Since I've done a lot of work on installation of TWiki (WindowsInstallCookbook
and answering Support questions), I am a bit concerned about how easy CVS might be to set up. I do use CVS client tools but have not done CVS admin - last time I looked at this it was not at all trivial. It would be preferable to use pure Perl modules if at all possible, to simplify installation - see RcsLite
for an example.
CVS would presumably be an option for more advanced users, but it would be good if ReadWriteOfflineWiki
was available even to new users of TWiki, as it is a very useful thing to have! In fact, any distributed installation of TWiki must really solve the installation problem, as I believe MichaelSparks
has done at Inktomi.
As for Netscape limitations - you might try Opera or Mozilla, as they don't seem to have this problem.
- 08 Aug 2002
This is a very long topic with good points I'm sure. Can someone please summarize the status at this point, two years later?
- 04 Jan 2003
As a wierd sort of aside, I use CVS for everything, and have been CVS'ing my TWiki installation.
Yes - I have not replaced TWiki's RCS
, but instead have been checking the RCS
into CVS (which you can do via cvs import -I!).
It's wierd, but it actually works. Or, at least, I have not run into any problems yet.
And it provides a neat benefit: TWiki renames pages, and moves them from
web to web, by renaming the RCS
Which basically breaks the version control system.
But when I CVS the RCS
,v files, I can check out any version of the wiki
database that has been checked into CVS.
Now, CVS merging isn't always the right thing. Conflicts get reported
- which is good. Worse, CVS merge may do the wrong thing, and create
a ,v file that is inconsistent. Must specify -kb.
So, I wouldn't use this to check a wiki out to multiple sites in general,
although I have been using modifications to templates, etc.,
in a separate copy of my main twiki.
Also, after a rename or move, "cvs update" will by default check out the original
file. If you know CVS, yiou know how to handle this.
This isn't as good as a proper object mapping database,
but it goes a long way towards handling CVS renaing.
(I started doing this because I was CVSing changes to the twiki perkl
scripts; applying it to the data files just came for free.)
For earlier folk: I have released a patch to cvs that allows it to handle
multiple cvs repositories... --cvsadm.
- 17 Apr 2003
Andy, please, could you explain WHY
you would like to store RCS
,v files in CVS? Is there any reason? I cannot see it. I would like (it will be wonderful!) to use CVS (which uses RCS
internally) instead of pure RCS
- 23 Apr 2003
(0) This is an example of why some sort of mail notification, between than TWiki's, would be a good idea.
I don't always go and read old pages.
(1) It's not a question of wanting to store TWiki's RCS
,v files in CVS. It just fell out for free, by accident.
Realistically, I would prefer to have TWiki use CVS directly. But, then you need to address
CVS's inability to rename files, etc. I suspect that, rather than renaming the RCS
the way TWiki does now, we would want to use the cvs remove/cvs add protocol.
Now, the CVS'ing of the RCS
,v files is interesting in another way:
it allows you to version control the meta data, such as tags, in the RCS
TWiki doesn't use such meta data much if at all,
but people who use CVS a lot may understand the desire to be able to do
things like versioning a floating tag.
- 23 Apr 2003