File System Tool to TWiki Plugin Adapter

Sometimes I have or want tools that operate on the TWiki filesystem. E.g. "find all WikiWords in this web" is a fairly simple UNIX command line.

Sometimes I want to run such a tool against my twiki/data/Web files - because I am working on the raw filesystem, because my twiki server is not up, or because I have a directory full of twiki format *.txt files that is not actually on the server.

But, sometimes I want to run almost exactly the same tool from the twiki web interface. E.g. "do this operation for all files in the web" is almost exactly equivalent to

   prepare command line using a form
   system("tool twikiurl/web/*.txt arguments from form");
   capture stdout and stderr 
   filter it

Conversely, sometimes I want to take an arbitrary twiki plugin, and just run it as a Unix-like filesystem tool.

Windows users: I do not mean to leave you out. Many Windows tools operate in a similar manner.

Q: is there some sort of generic adapter to take a Unix-like filesystem tool, and create a wiki plugin, or vice versa?

-- AndyGlew - 16 Mar 2006

Possibly TWikiShellContrib to the rescue?

-- PeterThoeny - 16 Mar 2006

I don't think TWikiShellContrib does what I mean - or if it does, the documents are written in a way that I don't understand. TWikiShellContrib seems to be a shell for common twiki admin actions. Not a browser interface (or did I miss that). It seems to be a shell that gives access to existing twiki tools. I'm wondering about tools/stuff that allows me to take a new script, and wrap a web page around it. And, vice versa - take any script whose argunments are set up by a form, and make it usable as a command line script.

But maybe I'm missing something in the discusion of TWikiShellContrib. I'll study it some more.

The sort of thing I'm looking for: you, Peter, pointed me to scripts that detect dangling links. Now, is there an easy way to plug that script in so that it becomes a wiki page? One way would be to have a generic tool that took an arbitrary shell command, ran it, and wikified the output. But that would be a horrible security hole, unless it was run in a suitably narrow chroot box.

-- AndyGlew - 18 Mar 2006

you might find ProgramsPlugin helpful ( although i'm a little scared of it )

-- WillNorris - 19 Mar 2006

Brainstorming: How about a new RunScriptPlugin that allows one to run a predetermined set of commands?

Example:

%RUNSCRIPT{
  search "City: San Francisco" format="   * $topic: $summary" | grep "sushi" > $result
  set $total = cat $result | wc
  print "Restaurants found:"
  print "$result"
  print "Total: $total"
}%

The scripts need to be aware of TWiki's permissions. Additional commands can be added by the administrator, after a security check.

-- PeterThoeny - 19 Mar 2006

TWikiShellContrib exposes functionality at an API-to-commandline mapped level.

Once these APIs are exposed they can equally be exposed via another framework to the web, or to SOAP.

This would allow a plugin to be used from both the command line, and web, and via XML-RPC.

Such a presentation neutral exposure would encourage a service oriented architecture componentisation.

The only plugin I know that can be accessed this way is my TWikiReleaseTrackerPlugin: you can access it via the web or via twikishell.

-- MartinCleaver - 19 Mar 2006

this is somewhat incorrect.

all plugins that use the rest interface are automatically accessible from the command line and web cgi.

I use this functionality quite a bit!

-- SvenDowideit - 19 Mar 2006

Great - I had not realised that.

We certainly need the REST methods better publicised. Can we get editorial coverage in some journals about it?

We need to demonstrate both the purpose and the the mechanisms for accessing

Additionally there needs to be a twikishell discovery mechanism for it.

M.

-- MartinCleaver - 19 Mar 2006

Um, can someone point me to the REST interface? (Alright, already, I'm searching for it now...)

-- AndyGlew - 20 Mar 2006

By the way: I'm not religious about a tool starting up as a UNIX command line tool and then being adapted to TWiki. I agree that there are major security issues with this, since the UNIX filesystem tool is not compelled to use TWiki's access control mechanisms.

I'm equally happy with a mechanism that takes a TWiki tool - a plugin or the like - and makes it accessible as a UNIX filesystem CLI tool. This may be easier... e.g. I can imagine taking a TWikui form which provides the parameters and arguments for a plugin, and automatically creating a command line parser that recognizes those arguments, via some standard form to getops mapping.

For that matter, I'm not even religious about UNIX filesystem access. It's just that there are many things that I can do in the filesystem that I cannot do via TWiki. But I can just as well imagine tools that use wget to get the raw TWiki source, and which similarly automate the putting... and such tools might have the advantage that, if the are accessed via wget/wput, the twiki tools that provide get/put access may impose TWiki access control.

I just want to be able to access all of the TWiki page contents via such a mechanism, including the METADATA.

-- AndyGlew - 20 Mar 2006

I think the TWikiShellContrib addresses your second paragraph, though it can be hard to tell :-(.

Another advantage of this approach is that if you have webs that are using a different store implementation (e.g. have data in a DB, or even on another node) you will still be able to use the TWiki tools to manipulate it.

-- CrawfordCurrie - 23 Mar 2006

I'll admit that I don't follow all of this discussion, but personally, I think there is value in using tools external to TWiki for some of the things discussed here, for security reasons if nothing else. I personally use Fileman, a browser-based ftp-type program which also allows use of unix command.

-- LynnwoodBrown - 23 Mar 2006

The scripts I attached to http://twiki.org/cgi-bin/view/Plugins/DanglingLinksToolNeeded, that use Perl inheritance tricks so that I can use TWiki::Render to parse twiki in a command line app, are an example of what I've had to do to get this sort of dual interface.

Actually, I have not yet made these CLI scripts work as lugins. My dream is to have that just work easily, instead of requiring a lot of coding.

-- AndyGlew - 24 Mar 2006

Can somebody give me an example of a fairly comprehensive UNIX command line tool using TWikiShellContrib?

E.g. something simple, like using using TWikiShellContrib to move all topics whose names begin with Foo and which have not been accessed in 2 years to a DefunctTopics web.

And maybe something fancier, like using it to locate all of the twiki links in a page - the sort of stuff DanglingLinksToolNeeded does.

I swear, I just cannot understand what TWikiShellContrib is good for from its documentation. (I can hope that there are such examples in the source once unpacked, but I am still not sure that it is worth spending the time to find out, since the documenttaion makes it look so not-applicable to what I want to do.)

-- AndyGlew - 24 Mar 2006

I've moved some documentation from the dev topic into TWikiShell - this should form an outline for discussion.

In short, its an extensible mechanism for exposing functionality to the shell. It exposes very little by default.

Included in what it does not do is any sort of fucntionality. Its a framework for making functionality available, not the supplier of that functionality.

A logical first CommandSet to build one that exposes the methods in TWikiFunc.

I do know that Rafael has updated the TWikiShell SVN package more recently that he has published the ZIP/topic on TWiki.org; really I think we would do well to have the odd automated reminder...

-- MartinCleaver - 24 Mar 2006

I'm just plain confused: "twikishell is a generic command line utility for TWiki. Think of it as bash for twiki"

Why do I want a bash for twiki? Why cannot the various twiki functionalities be run directly from bash? Or any other UNIX shell for that matter?

This applies to interactive use, but also, especially, to batch/scripting use.

Perhaps what I would do is write good old bash or Perl scripts, that invoke TWikiShell commands. Something like:

   #!/bin/sh
   find .../twiki/data/ -type d -print \
      | xargs -1 twikish TreeViewPlugin
   find .../twiki/data/ -type d -print \
      | xargs -1 twikish DanglingLinksPlugin

Is this what twikish is good for?

-- AndyGlew - 24 Mar 2006

Yes. I've updated the TWikiShell description again. Keep the questions coming!

-- MartinCleaver - 24 Mar 2006

AndyGlew, can you check the new documentation of TWikiShell? Then we can work together (Martin, you and me) to create any CommandSet that you need.

-- RafaelAlvarez - 06 Apr 2006