TWiki, A Transformation Story

-- ThomasWeigert - 08 Feb 2004

Abstract. This note discusses the transformation of TWiki from a web-based whiteboard application to a content management framework comprising a number of integrated applications.

This topic contains philosophy and a rational reconstruction of TWiki background from what I observe, which may or may not correspond to reality. I realize that there is a long history of wikis and a complicated geneology. Just looking at a list of available wiki engines is mind-boggling. In this little essay I shall pretend that TWiki is all there is, to focus on the message I am trying to tease out. Please forgive me for this myopic view, but sometimes tunnel vision actually furthers understanding.

Let me apologize also for the topic title. Reading it now I realize that it implies that twiki is not mature. Quite the opposite. What I want to talk about here is how twiki has grown (in my fictional rational reconstruction of history) and where it could develop further. I guess you could view the "not yet grown up" positively as still having the ability to evolve.

The Roots

TWiki had its origins in Wiki - a simple tool for allowing groups of people to brainstorm, collaborate and share information through the web. Key design goals were to allow the free construction of content and the avoidance of the "one web-master syndrome". For more detailed discussion of TWiki background please see the presentation by PeterThoeny. (1)

As such, TWiki provided a single application: a web-based white board where users could create content and present it nicely using a simple mark-up language (see TWikiShorthand). The focus of this application was to allow users to produce well formatted pages, layed out using simple rules, with text flowing from top to bottom. A number of TWikiShorthands makes formatting text easy.

Various additions surfaced to make formatting easier or give additional power, such as RicherSyntaxPlugin, SmiliesPlugin, RenderListPlugin, to just name a few.

However, the main structure of TWiki was the white board. There is an edit button that allows the user to make the white board writeable and more or less sophisticated ways of scribbling on the board. Others then could add their own scribbles or delete or overwrite what has been written.

More sophisticated white boards

Users and designers quickly realized that due to the flexible framework based on perl it was easy to produce more sophisticated content. Tables and searches across a web became vehicles to generate content from a number of topics, and again, many additions made producing such content easy. TWiki.org has many examples of searches extracting information from webs, and plugins such as TablePlugin or SpreadSheetPlugin aid in the presentation of this information. A rather complicated tool to produce content based on searches is the FormQueryPlugin. However, these enhancements were basically still creating a fancier unstructured white board.

In another way the white boards became more sophisticated was by the addition of customizable "edges". Templates allowed the change of the page headings and page bottoms with generic content, such as logos, menu bars, action bars, copyright notices, or other information.

Addition of structure

It did not take long that the need to add more structure to one's content surfaced. The first such application was the addition of category tables that eventually grew into today's forms (see TWikiForms). A web form presents information in a structured manner, in form of a table of two columns, with properties of the topic in the first column, and the values of that property in the second.

With the addition of forms TWiki stepped on a slippery slope. Forms contained content which was not any more freely editable an the white board (text area). A whole machinery had to be built around this concept: forms are defined in a table, added or changed with the form chooser button and its associated script, and rendered at a place that was predetermined by the template designer (usually at the bottom of the screen). In an important sense, TWiki had become two applications:

• An unstructured white board, and
• a highly structured form.

The branch in the road

The form was the first application in addition to the white board, but others quickly followed. An attachment application allowed to associate files with a topic. A number of corporate users built defect or bug tracking systems. I have seen internal request tracking systems, phone lists, action trackers, todo list trackers, project tracking systems, and so on.

What all these applications had in common was that the generation and presentation of data was not via the white board, but through highly structured means, usually based on forms. Typically, some machinery had to be added to twiki supporting each application, as this had to be done for the form application also. For example, there is an attachement management subsystem involving several dialogs, forms, and scripts. Similarly for these other applications.

The other addition to twiki to support these applications was the so-called "meta data". I use scare quotes, as ordinarily, meta data refers to information about the data, such as when a topic was created, who edited it, and similar. The "meta data" that is most prevalent in twiki, however, is really the information stored to support most of these other applications. For example, there is a line of meta data for each attachment which references the actual file, and holds some information such as whether the file should be hidden, a comment made during the attachement, who made the attachement, and so on. For the forms, there is a line stating which template the form is based on, and then there is a line for each field in the form giving its value. As the applications proliferated, so did they various types of meta data users added.

Examples of true "meta data" are very few: the "parent" information for a topic, the "topic info" which really just duplicates information in the RCS file associated with a topic, and the "topic moved" information, indicating that a topic had been renamed.

At the same time, of course, the white board application grew as well: Highly customizable rendering for tables was a great visual enhancement, as were the many better ways of formatting lists. The search mechanism became ever nicer integrated with the table mechanism to render computed information in an attractive fashion, and clever tricks were worked out to filter or extract information from searches. Note, however, that even for tables we find a specialized application: SimpleTableEntryUsingForms, which allows creation and editing of tables through structured forms, and stores table data in meta data.

The two extremes of twiki

Looking at the twiki installation that have been implemented, in particular, at twiki installations at corporations, we can identify the following two extreme points:

1. The freely editable white board at which people exchange unstructured information (or where structure emerges in the form of lists, bullets, or tables)
2. Highly structured applications where users can enter information only in a tightly controlled manner.

Examining the various plugins, add-ons, and patches that have been proposed, one can usually spot quickly whether they support more the view of extreme (1) or extreme (2).

Many plugins, of course, make the whiteboard application more powerful. Examples are TablePlugin, SpreadSheetPlugin, ExplicitNumberingPlugin, FindElsewherePlugin, RicherSyntaxPlugin, SmiliesPlugin, SpacedWikiWordPlugin, SuggestLinksPlugin, or UpdateInfoPlugin.

On the other hand, there have been extensions that supports to make twiki more structured and contolled. Among those are

• Extensions to enhance forms to be the vehicle of presenting applications by adding mandatory fields or imposing work flow (WorkFlowAddOn) on the user interaction.
• Extensions that make input more structured and give control over where on a page user input will appear are CommentPlugin, EditTablePlugin, SectionalEditPlugin, HiddenTextPlugin, FeedbackPlugin, or RecursiveRenderPlugin.
• Extensions that present content in structured ways are, for example, QuickCalendarPlugin, ChartPlugin, or HeadlinesPlugin.

Many or most of the structured applications, however, have been developed within organizations, and are often not publically available. Examples that are posted at twiki.org are XpTrackerPlugin, SlideShowPlugin, or PollPlugin.

Most other extensions either provide additional infrastructure for twiki or support additional data integration.

In our own twiki installation, the above two extremes are usually separated into different webs. There are webs where people collaborate, and there are webs where issues or projects are tracked. This separation is due to implementation constraints more than actual need. Twiki customization is easiest done at the web level, by changing templates and adding web-specific scripts. Consequentially, when applications change the default layout of a rendered page, or omit aspects of the standard twiki installation (such as presenting a topic without the whiteboard application, or without the ability to add forms), these applications demand their own web.

Towards Synergy

The lesson in above is that twiki has developed beyond the whiteboard and is, instead, a potpurri of applications. Twiki should recognize this fact and support it in its infrastructure. There is no reason why the whiteboard application should be any more privileged than any other application. There is no reason why the only three default applications are the whiteboard, the form, and the attachment.

Wikis began with the idea that users should be able to post and discuss content on the web without interference of a web master. Recognizing above, TWiki should support, in a manner true to this heritage, that users can assemble a web page from a number of applications without requiring access to the hosting computer system.

Consider the following simple example: On twiki.org, some topics now consist of a text presenting some issue, and a text entry box presented by the CommentPlugin, intended to collect reader input to the issue raised. Such topic layout would be a natural candidate for the described separation into applications. We have the combination of text with the "comment application".

To make the assembly of topics from applications feasible, the twiki infrastructure should separate a twiki into three aspects:

1. The skin should allow the modification of appearance of a topic only, independent of the rendered content. Skins should allow headers, footers, menus, and side bars to be defined (or undefined).
2. The content area of a topic should be arrangeable into separate applications, taking over part of the content area.
3. Each application then manages its own further specification and user interaction, if any.

We have experimented with supporting this separation within the context of twiki, attempting to modify twiki as little as possible. In this, we relied primarily on the ideas captured in SectionalEditPlugin combined with RecursiveRenderPlugin. The basic insight is to treat the text area the user is presented when clicking the standard twiki edit link as the "application layout area". In this area, one arranges twiki applications by placing appropriate TWikiVariables in a table describing the page layout.

• If a portion of the topic should hold a whiteboard application, it is placed within an editable section in a table cell.
• If a portion of the topic should display content only, the text rendering this content is placed in a table cell.
• If a portion of the topic should hold some other application, a variable calling up that application is placed in a table cell, possibly augmented by some text.

The only modifications required for twiki were the generation of variables that would allow rendering of the form application and the attachment application through a variable, and a change to the standard templates to avoid placement of the form and attachment applications by default.

Admittedly this is somewhat clunky in that the edit button at the bottom of the page is hijacked as invoking application layout but such little blemishes are easily rectified. However, this little experiment demonstrates how easy it is to reinterpret a twiki as a user-driven web application delivery platform and content management framework. In this manner twiki goes well beyond Zope in the ease in which it allows the user to assemble a web page from various applications. It also goes well beyond the standard twiki in that the person interacting with the generated page interacts with each application separately without having to worry that edits in the whiteboard application will interfere with other applications. It further goes beyond the standard twiki in the customizability of a page without requiring changes at the template level.

A final wish list

In order to better support the vision of twiki as described above, a few changes would be helpful. These are listed below:

• The skin package needs to support laying out all aspects of the page that should be customizable by the user. This should include style sheets to change the rendering of fonts, etc., at the skin level. We found FlexibleSkin the best suited among the skin packages for this task. FlexibleSkins provides stylesheet support, definitions for an area around the topic (left bar, top bar, right bar, and bottom bar), as well as headers and footers. The various areas are defined in topics.
• TWikiVariables need to be made available for adding the form and attachment applications, and these need to be deleted from the standard templates.
• A "application layout" capability should be produced which lets the user conveniently define the topic as composed from various applications. While this can be done in a table in the text area, a more error proof definition mechanism is preferable.
• It might be beneficial to focus topic changes, revision control, etc., on the white board application rather than on the topic as a whole. The user might not be interested how a whole page changed but only in how the content of the whiteboard application changed?
• Some (or most) of the applications comprising a topic may need the capability to store data that is being rendered. While the SectionalEditPlugin allows the topic data to be segmented into the corresponding applications, a more natural place might be meta data. Indeed, many applications already use meta data or auxilliary topics for data storage. (2)

In conclusion, this note argued that twiki has evolved to be able to function as a content management framework that could be programmed through the web by the end user, in the heritage of the twiki mission. A full support of these capabilities would require only minor changes to twiki.

Notes:

(1) Comment by MichaelSparks: TWiki's roots are in JosWiki. The Jos project was a collaborative attempt to build a Java based operating system. They decidied that they needed a wiki, so they wrote one, based on Ward Cunningham's wiki. Jos itself hasn't gone very far, but the wiki engine has. Jos did ditch their own wiki after a while in favour of PeterThoeny's Wiki fork, but have since ditched TWiki in favour of a simpler collaborative Wiki. (There's a lesson there to be learnt if someone cares to write about that.)

(2) One of the few added applications that left their data on the white board was the ActionTrackerPlugin, based on the philosophy that actions are part of the meeting minutes that generate the actions (which are captured in the topic). Note, however, that there are several variations of this application in use that resulted from convering the action tracker to be based on data stored elsewhere.

Please add feedback, comments, or discussions at TWikiWhatWillYouBeWhenYouGrowUpDiscussion.