create new tag
, view all tags

Over the last few months I've been working on the editor applet (see PowerEditPluginDev) - and it's made me think laterally. The new editor applet has a display window that displays the formatted text while you are editing. To do this, it translates the TML on the fly into XHTML and displays that. It has a reverse transformation into TML as well, though I'm not sure yet where it might be used in the applet context.

Anyway, my thinking runs something like this; "to make a WYSIWYG editor work on stored text (TML), you have to strip off metadata, download to client, jump through hoops to translate into XHTML, then jump back through hoops to translate that XHTML back into TML and finally upload to server". Hardly thin client. So what might work better? Well, a smarter store engine would help enormously.


Imagine a storage engine able to serve stored text in a variety of formats, providing bidirectional transformations on it. The base store would store topics in format 'X'. I don't care what 'X' is at this point. It would publish interfaces for a variety of other text formats as well.

To view stored text in a browser, I would ask the storage engine to "serve me the text in XHTML". The initial processing of X to generate XHTML would be done by the store engine.

To edit the content in a standard textarea, I would ask the storage engine to "serve me the text in TML" - exactly as it is done today - and when finished I would hand the edited TML back to the server. The server would then munge the TML back into X before storing it.

If I want the text as XHTML1.0, to edit in an applet or other HTML WYSIWYG editor, I ask the storage engine to serve that instead.

Any other formats that the storage engine supports could be imported or exported. Such as MIME. Or MIF. Or XML+DTD (for the brave only). Or LaTeχ (god forbid). Or (importantly) other wiki meta-languages as used in wiki clones such as KWiki or Tiki.

So how realistic is this?

Some formats are much richer than others (XHTML1.0 is rich in formatting but poor in meta-data, TML is rich in meta-data but less so in formatting). So how can a 1:1 transformation be achieved? One way is through use of a lowest common denominator. If X contains formatting TML can't handle, then those portions of the topic could be served as HTML to a TML client. Or X could be constrained to be the lowest common denominator, and unsupported aspects of HTML formatting stripped off on store.

Using the latter approach, TML <-> X is really not that hard; I managed TML <-> XHTML fairly easily. Of course, the devil is in the details. I only handled the functionality of getRenderedVersion, and didn't attempt to handle meta-data such as embedded variables that are specific to TWiki. I also ignored other topic meta-data such as forms. Nascent interfaces to these meta-data already exist in TWiki (though they are currently a bit kludgy). These interfaces could be implemented by the storage engine as well, to round out the offering.

At this point I'll give X a name; "WikiCanonicalForm". TML gets relegated to it's rightful place - a quick and easy editing format - rather than master of the universe. So do all other Wiki ML variants.

The actual impact of this approach on the current TWiki implementation is not as great as you might imagine. An engine is still required that handles "non-store" functions, such as expanding variables, applying templates, attachments etc. etc. (though further lateral thinking can take you in the direction of considering attachments as just being alternative page formats that are stored "raw". So you no longer have attachments, instead you have read-only pages in the same storage area as pages in format "X", but only serveable in one format).

Practicalities, or how to get there from here?

TWiki rendering

TWiki works on TML + %META, and it can continue to use exactly that format. The store engine would map 'X' to TML.

Converting existing databases

To load an existing TWiki DB into 'X', simply apply the TML->X transformation to all the topics in it.


Current TWiki search would continue to operate, via the X->TML transformation. I think the only wrinkle is grep. The store engine should provide search facilities as well - perhaps cacheing search results.


Most plugins essentially transform meta-data embedded in TML. As long as the plugin doesn't pull tricks like backdoor reading of topics, they should continue to work via X->TML->X. Smarter plugins - such as ActionTrackerPlugin - can interact directly with the store engine.

Configuration Management

With X in your hand you can add meta-data to your hearts content without breaking TWiki. You can store differences, branches, the works, if you want to. Plugins can elect to operate on TML, or HTML, or even raw X if appropriate. The store engine can implement a full CM system, or it can layer on top of CVS, RCS, ClearCase, whatever.

Points for consideration

  • Language neutrality: The store engine must publish an interface that is accessible from many languages (perl, python, java, C, ruby - basically anything that you can write CGI scripts in).
  • Speed: The store engine has to be very very fast.
  • What goes in, what stays out?. e.g. are variables a common enough concept that X has to store them as metadata? Or should that be left to the wiki engine?
  • What's the minimum to demonstrate the concept?: Probably the ability to serve TML and XHTML.

-- CrawfordCurrie - 04 Feb 2004


Note: Almost everything in wikis differs - not all use CamelCase, the majority don't have variables, many have the plugin abilities, many don't, many have namespaces called lots of things, many don't, some have wiki trails, relationships between pages, others don't.

At present, your "X" format requirements are met by TWiki Markup - (depends on whether you append raw=1 on the end wink ). An alternative is to store everything as XML - Openwiki - http://www.openwiki.com/ already does this - unfortunately it's ASP based, and it's XML format seems tailored to their syntax rather than sufficiently general.

I need a decent Wiki:WikiInterchangeFormat due to the need to import/export content from arbitrary wikis. Furthermore, it prevents user lockin by any particular wiki engine. Such a format should IMO aim for machine readability, not human readability, and be similar to XML for obvious reasons, but differ from XML in the same way perl hashes differ from C structs. Why? Because any DTD will in all likelyhood be wrong - you want to capture Wiki semantics from multiple wiki engines or multiple applications in such a structure, not the rendering. (ie capture the parse tree - this can be achieved through the use of a) a rendering module b) using it to generate both formats from either format - ie reflexively)

That said, playing devil's advocate briefly, a good Wiki:WikiInterchangeFormat goes against some apparent local viewpoints that TWiki.org should compete with other wiki engines ("gain competitive advantage") rather than work with other wiki engines.

After all, another point worth noting is that TWiki is very unusual in that it allows users to enter arbitrary HTML into topics. Most other wikis consider this verboten. That alone makes one wiki sending content to TWiki in WIF, editted locally, passing through a second WIF exchange and back, and then passed back to the original could result in unexpected behaviour.

If you consider that strange consider two wiki front ends sharing the same storage. (Which is an interesting exercise in it's own right considering that many use berkley style dbs for page storage)

Incidentally your X appears to confuse, AFAICT, two things: (in case my response seems odd)

  • A serialisation format
  • A storage system
In case it's unclear, by Wiki:WikiInterchangeFormat I'm referring to a serialisation format for transferring topic text, not a physical store. (Consider that many relational DBs will happily spit out XML as an interchange format)

-- MS - 04 Feb 2004


Glad to see you doing some lateral thinking. I think something along the lines you mention could be very useful.

Just as an aside, I'll mention that, IIUC, AbiWord does something similar. Their import filters take text (and formatting) from other word processors' files and import it into AbiWord's run time piece table (rather than making some intermediate translation into the AbiWord native file format). Similarly, their export filters do the reverse (i.e., export directly from the piece table to the "foreign" file format, with no intermediate translation to AbiWord file format). Thus, in some sense, AbiWord can treat any "foreign" file format as native. I'm not sure what bearing this has here exactly, but maybe it validates some of your thinking in terms of other people trying related things.

-- RandyKramer - 04 Feb 2004

Michael said: " At present, your "X" format requirements are met by TWiki Markup " - yes, the format is already proven by my TML<->XHTML transformation in the editor; though the full TWiki translation to XHTML strips meta-data and decorates the raw format, making the reverse transformation impossible. As I said, I don't care what X is - or how it is stored - at this point. I only coined "X" to leverage existing familiarity with the idea of "text on disc". I'm really only interested in a storage engine that can serve multiple formats of the same data.

It has been remarked (on Meatball) that while there are a few non-intersecting capabilities between wiki implementations, these are mostly "bloaty" features. I agree - they are mostly enabled by what we call meta-data, and in terms of the data we have already (semi-unconciously) kept them separate from the rest of TWiki syntax by using %VARIABLES (I consider %META to be a %VARIABLE here).

  • This semi-structured use of %VARIABLES might well make TML a good candidate for "X".

TWiki already acknowledges the idea of serving multiple formats; all I really want to do is replace "?raw=on" with "?format=(xhtml|mif|twiki|kwiki...)" (and support "bin/save/Topic?format=" as well, of course).

This concept started out in the editor applet, which is currently a TWiki plugin. I don't really want it to be a TWiki plugin - I'd like it to be a storage engine plugin. That way it could be used with any application that implements the storage engine interface. In this way, plugins are suddenly shareable among wiki implementations, and even other applications. Then let the best skin win.

-- CrawfordCurrie - 05 Feb 2004

A %VARIABLE type approach would in all likelyhood scare off most of the community, since it takes the view that TWiki is right for TWiki and is therefore good for everyone else. Even the TWikiMission statement states it targets corporate environments rather than normal wiki environments.

http://atox.sourceforge.net/atox.html is an interesting approach to the issue - the idea behind that parser is to effectively act like XSLT for plain text. The "X" it means here is specifically XML, but it's the similarities to what's needed here jumped out at me.

-- MS - 06 Feb 2004

Due to WikiMania, there has been the beginings of a dialog into WikiInterchangeFormat - I'm hoping that this would help the edit UI (web and rich client), and the cross wiki search system, as well as giving us a goot import/export format.

(I'm leaning towards an rdf based format, thus building on existig formats and tools (and because there were a number of semantic web presentations))

-- SvenDowideit - 05 Aug 2005

Edit | Attach | Watch | Print version | History: r10 < r9 < r8 < r7 < r6 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r10 - 2006-05-31 - RandyKramer
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2018 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.