Tags:
development1Add my vote for this tag create new tag
, view all tags

Page for Ideas and Requirements for the WebDAVPlugin .

This page is contributed to ideas for reimplementing the TWiki WebDAVPlugin. First of all, I come forward with a proposal to take the Catacomb WebDAV Server (http://catacomb.tigris.org). This because of two reasons:

  • Catacomb is a apache module too, like the WebDAVPlugin.
  • I'm confided with catacomb-development so if on this side would be any changes necessary, it would be easier.

On the first look there will be the disadvantage that catacomb uses a mysql database and that would increase the twiki requirements. But I've done some accurate performance testing on mod_dav_fs, and with many files it's simply too slow. On small wiki that would mostly be ok, but on enterprise-level collaboration there would be a great amount of files, accessed by many people. Then, at latest, mod_dav_fs will be no options any more.

WebDAVPlugin should become a apache 2.X module. I think a lot of companys would (or could) not use the plugin if it is apache 1.3 only.

The next point is that we should improve usability of use and installation. I've not tested the TWiki WebDAVPlugin so far, are there any disadvantages?

-- MarkusLitz - 14 Dec 2006

Thank you Markus for helping out here. A solid WebDAV integration fits well into the TWikiMission.

From 10,000 feet above ground I'd say we need this:

  • A seamless attachment integration, e.g. files attached via browser and via DAV should have the same result and should be interchangable.
    • Open a dialog to ask/change comment when dropping a file to an attachment folder?
    • Same handling of illegal chars in file names
    • Creating an attachment folder via DAV creates an empty topic in TWiki (with standard signature)
  • File attachment table should have a link to the DAV URL so that you can edit documents without a download/modify/save/upload cycle
  • Expose two DAV root directories, one for topics, and one for attachments
    • Both root dirs would list all top level webs
  • Transparent authentication and authorization:
    • Use same auth as TWiki
    • Ask for auth only once
    • Use access control of TWiki

Here are some ideas:

1. In DAV, expose topics with TWiki specific extensions such as WebHome.twiki instead of WebHome.txt. This allows one to associate files with a thin client to edit TWiki topics.

2. In the DAV space, merge topic space and attachment space. Example:

  • root - no files
    • Main - contains WebHome.txt, WebNotify.txt, FooBar.txt, ...
      • FooBar - contains attachments of FooBar topic
      • ...
    • Sandbox - ...

-- PeterThoeny - 14 Dec 2006

I know Peter already mentioned this, but I want to reiterate that Security is a very important requirement.

  • If a user doesn't have read or write access to a topic (using topic variables), that should propagate to the DAV interface too.
  • Moreover, a user shouldn't be able to circumvent a web's access list by starting at the Main web and navigating up and then down into a secure web.

-- PankajPant - 14 Dec 2006

Ok, it will be a big effort to satisfy all this points. But I confident that this plugin will be a great benefit to TWiki. How far is the current status of the WebDAVPlugin? Which of the point are implemented and which not? I'll start testing the WebDAVPlugin the next days and then think about porting to Apache 2.2.x.

-- MarkusLitz - 21 Dec 2006

Wow, this sounds exiting! I agree with the points Peter and Pankaj mentioned, but to me an implementation without the strict security would already be very useful, as I assume it 'd be for many people with small TWiki's behind a corp firewall. By releasing such a version, already you can have people test the plugin and give you feedback.

This would make a good case for TWiki versus Sharepoint

-- JosMaccabiani - 21 Dec 2006

Sounds promising. Did you make some progress on porting this plugin to Apache 2.x?

Cheers,

-- CarloSchulz - 16 Mar 2007

I like the idea of using the existing catacomb module but am concerned about the issues with using MySQL:

  • Using MySQL implies major change to TWiki and its users
  • Simply loading files into MySQL from files for backwards compatibility would duplicate data resulting in data sync headaches.

MarkusLitz, I think it would be relevant to know where that mod_dav_fs performance bottleneck was in terms of scale. For example, maybe it would be easier to optimize TWiki and mod_dav_fs than it would be to replace the current RCS implementation with MySQL.

-- IanTegebo - 07 May 2007

I don't want to save all in mysql, but maybe webdav metadata and right management. Otherwise maybe the development of a new module would be easier. A kind of mod_dav_twiki which uses the twiki internal right management.

-- MarkusLitz - 11 May 2007

That's what twiki_dav is.

This isn't rocket science. All it takes is someone to write a port to Apache 2. It's not going to be me unless I get paid to do it, and I suspect most other people are in a similar position.

-- CrawfordCurrie - 11 May 2007

How much do you think it would cost us for you to port it to Apache 2?

-- TimJacob - 12 Dec 2007

Is there a bounty somewhere to port this to Apache2? My company would definitely be interested.

-- JamesPeverill - 22 May 2008

I think support of hierarchical webs is important.

We are willing to help fund a port to Apache 2.

-- RickMach - 06 Jun 2008

I will try to work out an estimate for this work, to give you an idea of cost.

-- CrawfordCurrie - 07 Jun 2008

Crawford, Any progress on the estimate?

-- RickMach - 17 Jul 2008

Rick, sorry, I have been busy with other things. Working on it now.

-- CrawfordCurrie - 30 Jul 2008

Background

In 2004 I implemented the WebDAVPlugin module to extend HTTP to support the DAV (Distributed Authoring and Versioning) protocol with TWiki content. DAV is a protocol extension to HTTP. It defines a set of requests that can be sent to a server to manage a remote filesystem. On the server side, DAV is installed as an Apache module called twiki_dav, which intercepts DAV requests and handles them. twiki_dav module was a c-code program based on the original mod_dav written for Apache 1.3, coupled with a TWiki plugin and external maintenance scripts.

Features of the module included:

  • Support for TWiki ACLs in limiting access to TWiki topics and attachments
  • Presentation of pub and data areas of TWiki topics for access via DAV
  • Simplistic locking mechanism, for locking during updates
  • No requirement to run mod_perl
Limitations were:
  • No support for Apache 2 - tightly coupled to Apache 1.3
  • No support for locking
  • No support for Delta-V (the versioning API) or DASL (the proposed searching API)
  • No subwebs
  • No integration of pub and data views of topics
  • Hard to test
  • Tricky to install on "untested" platforms (OSX, Windows, VMS etc)

Possible approaches

Apache::WebDAV

WebDAV support has moved on quite a bit since I wrote the original TWikiDAV module. Most interesting is the creation of the Apache::WebDAV module in CPAN. This is a Perl module that performs the same functions as mod_dav, but does it in Perl. The module is layered on top of Filesys::Virtual, which suggests an interesting possibility. If the TWiki store had a Filesys::Virtual layer pinned on top of it, it could be used. This idea presents some interesting possibilities:
  • Not only would the TWiki store be available as a DAV store, but it would also be available to other users of Filesys::Virtual (and would make it all a lot easier to unit test).
  • Because it is a pure perl module, it should be installable in any Apache that supports mod_perl (as long as all the many dependencies can be resolved!)
On the negative side:
  • Apache::WebDAV is Perl, and performance may not be great (though it should be good enough)
  • Requires mod_perl, incompatible with Speedy CGI and FastCGI
  • Written to use mod_perl 1, and must be ported to mod_perl 2
  • It has not seen any maintenance since 2006, which suggests that either the 0.01 version is perfect (unlikely) or that the usage has not been heavy enough to debug it. Indeed, that is not surprising given the easy availability of the mod_dav module for Apache 2.
  • Still no Delta-V or DASL (this is no great loss unless someone has a specific requirement for this)
  • May be limited to Apache login (tbd)

mod_dav provider

Apache 2 now has the mod_dav module as standard. This is a high performance C-code module that dynamically links with Apache. The module has an internal provider API to support different filesystems - for example, the Catacomb module mentioned above is a provider for MySQL database stores. A provider module could be implemented to realise this API for TWiki, by creating a virtual file system of topics and attachments that looks like a traditional filesystem. I think some of the code from the Apache 1.3 twiki_dav could be adapted to do this, but it also requires a lot of new code as well. The advantages of this approach are:
  • Evolution from Apache 1.3 twiki_dav
  • Much better performance
  • Lower risk of bugs in the DAV module
  • Support for properties, though the value of this in a TWiki context is questionable.
  • Does not require mod_perl
The disadvantages are:
  • Requires installation in the Apache server, even when mod_perl is available. This means that it is unlikely to be installable on hosted boxes.
  • The apache module code is difficult to work with (atrocious documentation).
  • The existing twiki_dav code can inspire and inform the work, but unfortunately very little of the code can be reused, due to the extensive refactoring of the Apache APIs.
  • Still no Delta-V or DASL.
  • Same porting issue as existed with twiki_dav

Estimates

Apache::WebDAV

Step Estimate (hours)
Port Apache::WebDAV to mod_perl 2 8
Create Filesys::Virtual::TWiki 12
Testing 12
Total 32

mod_dav provider

Step Estimate (hours)
Analyse API changes & create detailed plan 8
Port permissions DB 8
Write mod_dav_twiki provider 16
Create build and install environment 4
Testing 20
Total 56

In addition I have spent 5 hours researching this, which has to be added to the estimate.

Conclusions

I have performed an extensive code review of the latest mod_dav, the old Apache 1.3 twiki_dav code, and the Apache::WebDAV module, including experimentally bringing up servers for all three modules.

At the end of the day, it depends on the requirements which is the best approach. I feel that for most uses of DAV with TWiki, then the most appropriate approach is to derive a new module specifically for TWiki from Apache::WebDAV. The code is relatively simple, and the use of perl means it is straightforward to integrate with TWiki. However there is a risk inherent in this module that it is more likely to have undiscovered bugs than the official mod_dav module.

The second approach, of writing a new provider for mod_dav would deliver a higher performance solution. However it would be significantly harder to write, and would involve porting some of the existing TWiki code to perl, which cannot be a good thing.

Estimated cost for the Apache WebDAV would be £2220 (approx $4440 US or ¤2815 EU) and for mod_dav_twiki £3660 (approx $7320 US or ¤4700 EU).

Please note that while I would make every possible effort to remain within the relevant estimates, this cannot be guaranteed in a piece of work like this where there remain a number of unknowns.

-- CrawfordCurrie - 31 Jul 2008

Let's pass the hat. smile Hm, I'm wondering how such fund raising could work? If there are, let's say 100 people interested in getting such a feature, each would have to spent just 50 EUR. Where do I send my 50 EUR? wink

-- FranzJosefGigler - 31 Jul 2008

TimJacob and JamesPeverill are you still interested in helping to fund this? I'm working through the process to get us to fund some of this work. Can we work out a cost sharing arrangement?

-- RickMach - 07 Aug 2008

In considering the options, it seems the all Perl route might be better in line for integrating with TWiki and easier to extend in the future. However, I don't like have a dependency on a CPAN module that is .01 and doesn't seem to be in development. Would it be better to fork it, improve it and keep all the code as part of the plugin directly? I think it might be.

-- RickMach - 07 Aug 2008

I don't like it either. The CPAN module is written for Apache 1.3 and needs to be ported to Apache 2 anyway. When I looked at what that would involve, it became clear that the choice is between supporting the CPAN module, and creating a TWiki-specific fork. I would prefer the latter for a couple of reasons:

  1. it would allow us to forget Apache 1.3 completely (if we updated the CPAN module we'd have to continue to support it)
  2. it makes it easier to embed TWiki specifics into the module.

-- CrawfordCurrie - 08 Aug 2008

Agreed on creating a TWiki-specific integration.

-- PeterThoeny - 16 Aug 2008

Crawford, I cannot find enough support in my organization to fund the entire effort but we could provide $2k US toward the effort which is a significant percentage.

Anyone else able to help fund this effort?

-- RickMach - 06 Sep 2008

Who Offer £Roughly Percent
FranzJosefGigler 50 euros 40.1991965
RickMach 2000 USD 1115.94688
Shortfall   1063.8539235

-- CrawfordCurrie - 08 Sep 2008

Edit | Attach | Watch | Print version | History: r25 < r24 < r23 < r22 < r21 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r25 - 2008-09-08 - CrawfordCurrie
 
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2017 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.