create new tag
, view all tags

Idea: Drop TML?

This topic was refactored out of ReplaceKupuWithTinyMCE, where the discussion covered the point that the main limiting factor on WYSIWYG in TWiki is the use of TML as the store format.

I wish I had more knowledge of the different technologies involved in the discussion to make a more technical contribution, but I haven’t. So,what I will do is post some of my thoughts on the subject which, perhaps, aren’t my prerogative only. Here they are.

  • From a pure technical point of view, how hard would it be to change TWiki’s store format from tml to HTML? (or other language, for that matter, but I’ll refer to HTML here.) Trivial
  • What are the advantages, disadvantages and implications, from a developer’s point of view in having pages stored directly in HTML? Would it make future development easier/more difficult? For core developers, easier. Having to fight with TML is a constant problem. For plugin and contrib developers, it's less clear, as many generate TML.
  • What are the advantages, disadvantages and implications, from the user’s point of view? Basically reduced complexity.
  • Would TWiki gain a real competitive advantage from switching to HTML? No, but it would lose a competitive disadvantge in some people's eyes.
  • Would this open new avenues for developers and ease the adoption of new technologies? Probably not
  • Would it boost development or just brake the core code irremediably? The code is not affected
  • I understand we would immediately gain a usability improvement in having a better WYSIWYG experience. Would the %SEARCH% engine need to be re-written? What else? Most templates would need to be recoded, and some plugins that generate TML.
  • Is tml still suitable to cater for the needs of a “modern” TWiki, or has it become a dead weight that is bogging TWiki’s development down? This is the key question and I'm going to leave it open for others to answer
  • I assume one of the main reasons for not switching to HTML is, and will always be, the protection of the existing TWiki pages’ content. I should also assume that all the pages written in tml could be translated into HTML. The tml -> HTML translator Crawford’s working on does exactly that, doesn’t it? (although with a different aim in mind) Yes. An "upgrade on the fly" would be provided.

Sorry for sounding rather naive. It’s that I have the impression that developers are spending their brain power in finding workarounds to shortcomings other than doing what they perhaps would like to do the most: development! This can be very frustrating at times.

  • Damn right!

On a more constructive note, I've just installed TinyMCE myself in the hope to help crush some bugs.

(Crawford et al.: thanks for pushing the WYSIWYG forward. )

-- EmanueleCupido - 29 Aug 2007

I embedded some comments above, but before you read them, understand one thing; TML operates at three levels

  1. TWiki variables
  2. Inline formatting, such as * for bold and _ for underline, and importantly, wikiwords.
  3. Block formatting, such as lists, tables, blockquotes and verbatim
The proposal is only to remove (3) and maybe (2). (1) would always remain; it is key to TWiki. The store format will not be HTML, it will be rich HTML.

I'll also point out that changing the store format to HTML does not mean the end of TML. It would be perfectly reasonable for an editor to convert rich HTML to TML, allow the user to edit the TML, and then save back as rich HTML.

-- CrawfordCurrie - 30 Aug 2007

IF we change the format from TML to "TWIKI HTML" (HTML with variables), wouldn't that actually enhance the performance of the view script?

One (problem?/hump in the road?) that I see is that plugins that work on tables MAY be more complicated to code.

-- RafaelAlvarez - 30 Aug 2007

Plugins that currently emmit TML can use the existing rendering functions in the TWiki::Func module before returning or using that text, anyway. So the short term fix is not that difficult (we can do that in the core, during the adoption phase).

-- RafaelAlvarez - 30 Aug 2007

It's actually easier for plugins to use the CGI functions to generate HTML (e.g. CGI::table), because they don't have to worry about line breaks in the middle of table data. The one problem is the plugins that "decorate" tables, such as the EditTablePlugin and TablePlugin.

  • It would be a trivial change for TablePlugin to make it style any html table. -- ArthurClemens - 30 Aug 2007

My gut tells me we really want to avoid going all the way to presenting the topic as a DOM tree to the plugin author, but that would be a very real option.

-- CrawfordCurrie - 30 Aug 2007

Hmmm…am I missing something here? From Crawford’s comments to my questions it looks like ditching tml is the obvious thing to do. Were I not an engineer myself I would immediately say “let’s go for it”. But nothing is straightforward as it appears to be. Sadly! The next question(s) that springs to mind is: how many man-hours (in addition to the 200 already spent by Crawfard and the nnn spent by other developers) will be necessary to overcome tml shortcomings and have a working WYSIWYG? Let’s not forget that a working WYSIWYG has been recognised as TWiki’s priority #1 to success! Also, I suspect that TML limitations will continue to create troubles in the future too, when the time to implement the latest you-name-it-in-a-browser technology will come (how about those spreadsheet-in-a-browser things? Are they just toys or they actually serve a purpose?)

The fact that a tml editor will always be available should make anyone feel at ease. After all tml is the perfect tool when you want to write down quick notes and create simple pages. I assume we can also have tlm tables alongside HTML tables. Am I wrong? What else?

Would it make sense at this stage to try to quantify the effort (in man-hours?) necessary to move TWiki from tml to TWiki HTML (twHTML? – marketing people like fancy acronyms :-))? Just rough figures, with some simple breakdown of the tasks (I know…I know, no one has got time to spare after silly exercises.)

Is it only me or others too are feeling uncomfortable in having to stand on dodgy foundations (i.e. tml)?

(Crawfard: to prove I can be constructive too, I posted a comment to “WYSIWYG nitty gritty”, just in case you missed it. I am curious to know the answer.)

-- EmanueleCupido - 30 Aug 2007

Wikis always advertized to be more easy than HTML, wouldn't this kill this argument, and what about Creole? TWiki could support some enhanced XML dialect, too, couldn't it? I always dreamed of a Wiki fully supporting DITA, but I guess I wouldn't like to use it. Have you ever played around with SemanticWikis yet? IMHO they are a PITA. What about KISS?! What about the TopicObjectModel? ... Don't drop TML, just make it more easy to detect for WYSIWYG editors so they can leave their dirty fingers of it. wink

-- FranzJosefGigler - 30 Aug 2007

While I'm well aware of the programing problems related to supporting TML, I'd still like to note why I like having TML support:

  • I've never seen an WYSIWYG editor that worked 100% and have always had to resort to editing the raw code at some point.
  • Editing raw HTML code is often a drag. That was the original appeal of TML and I believe it still holds.
  • I often compose content for my wiki outside of the browser. It's nice being able to do this in any plain text file (such as Palm memos). It would be such a drag to have to do that with full html. There's a world of difference being able to use 3-spaces+asteric for bulletted instead of having to define (and close) nested UL and LI tags.
  • Reading content in TML is often workable and nearly always much easier than reading the same content in full html.

-- LynnwoodBrown - 30 Aug 2007

I agree with all points raised by LynnwoodBrown. In more than one occasion I managed to screw up a page so badly that it couldn’t even be edited in TWiki. A quick look in the page.txt file saved me. All that would have been much harder if the code were HTML. Just to elaborate on two of your point:

  • composition of new topics in tml: I suppose it could still be possible to use tml for this purpose. The code would then get translated into HTML when saving the topic.
  • editing of existing topics stored in HTML: as long as the HTML is tml-compliant (this is the case if the HTML were generated by saving a page written in tml) it should already be possible to have a reliable-ish HTML->tml and back that would allow support for tml editing. (some with more knowledge than me: please comment)
Perhaps there is a way to have simultaneous full tml and full WYSIWYG support ???

-- EmanueleCupido - 30 Aug 2007

I think the main issue for the upgrading is that a lot of TWiki Application - if not nearly all - relies on TML.

I have plenty of formatted searches that both search for content in TML type tables and generate tables in the output.

And a huge number of plugins uses TML and looks for TML.

There are no upgrade scripts or on-the-fly code that can ever figure out how to convert all that. Sometimes a search takes a variable in its search pattern or its formatted output.

I understand the background. And I have thought about it myself. But I do not see this as very feasible.

The consequences are huge. Both for programmers on TWiki and for our customers.

Personally I believe the right path is a Wysiwyg coded for TWiki for direct conversion between Wysiwyg and TML without the HTML in between. That is for sure not 4.2.0 material. There we are stuck with TinyMCE.

But for 5.0 I doubt we will get what we want with any of the standard editors available.

-- KennethLavrsen - 30 Aug 2007

My point of view:

  • Many TWiki apps depend on TML. We would break a lot of apps by switching to HTML
  • HTML editors behave differently, depending on HTML editor technology, browser, OS, and versions thereof. There is no standard. One editor generates a <IMG ...> tag, another <img ... />, another <img ...></img>, yet another <img ...> and at the very end of the page </img>.
  • Meaning, version control of topics gets messed up if one user edits a topic with editor A, and another with editor B.
  • Not all bowsers have an HTML editor. Meaning, users with browsers on odd platforms would be forced to edit raw HTML. Yuck!

I am all with Kenneth: In the long run, avoid TML/HTML/TML roundtrip. Flex is very powerful, and an Flex based editor runs the same on all platforms, guaranteed.

-- PeterThoeny - 31 Aug 2007

Like Lynnwood, I'd like to keep TML around for a while. The foundation of wysiwyg editing in browsers is still too weak to rely on it 100% a project of this size. I think I don't have to elaborate on all the possible points where wysiwyging could fall apart. There are enuf to let us feel uneasy.

However, we all know that we need wysiwyg, as solid as can, at least as good as our competitors.

While, I don't think that the main block in the road is our TML, it seems obvious to separate wysiwyged content from tml'ed. Sales people and other "technophobians" (sorry for that wink - but let me make this point) simply love wysiwyg, while the IT guys are happy to type in their docu easily using TML. Therefore I'd always suggest to separate those two groups and give them a web of their own each where they don't harm each other. They don't talk to each other in real life anyway wink . No, seriously. We should get away with any HTML-TML roundtrip, because it simply seems to be impossible to do 100% correct.

So what I'd suggest is the following:

  • Edit is WYSIWYG by default, as long as you did not configure it otherwise per web or topic
  • There is a way to switch to TML editting, with a big warning sign.
  • Once a topic got editted using a TML editor, it is tainted.
  • There is a way to switch it back to WYSIWYG, with an even bigger warning sign.

The goal of this is

  • to let WYSIWYG store in TWikiHTML into the topic textarea and
  • to be able to have pure TML areas around.

Now on backwards compatibility of TWikiApplications. We are not talking about removing the TML renderer from TWiki, are we? So they will still be able to produce TML output. The much more serious issue for them is how to deal with input, i.e. non-TML-input they read from topic texts. They do so right now by means of some $pattern() in a FormattedSearch. To be honest, this is no solid foundation of your mission critical TWikiApplication. I'd suggest that you rewrite your applications making use of TWikis biggest strength - forms. This is the best you can do now, as long as we don't have an implementation of TOM usable for TWikiApplications.

This discussion is about technologies targetting TWiki-5.0. So remember a DotOh release is a anything-can-change release. Give TWiki a chance to change on regular intervals.

Still, we need a viable WYSIWYG solutions to catch up. With that in mind, I'd suggest to iron out the issues with TinyMCE that we have now for TWiki-4.3, not TWiki-4.2, and continue this thread for TWiki-5.0. TWiki-4.2 should have been released already. There's no reason 4.3 can't follow up quickly. This will give TinyMCE a chance to be as good as can with the current approach.

-- MichaelDaum - 31 Aug 2007

Great discussion! However there has been quite a lot of point-missing going on.

If we change the store format to HTML (or some other canonical form, such as XML) it does not mean an end to TML. Think of it this way; instead of a TML -> HTML -> TML round trip for WYSIWYG editors, you have a HTML -> TML -> HTML round trip just for TML editors.

The advantage of a canonical store form is that you can map this form to and from many different dialects; including WikiCreole, or even XML (for use in e.g. Open Office). But to achieve this the canonical form has to be rich enough to represent content from any source without information loss. TML just can't do that; it is a "lowest common denominator" format.

So, the only real cons (those without trivial solutions) to going to a canonical store form have actually all been mentioned above:

  1. Some plugins, TWiki Applications, and scripts that write topics without going through TWiki, would break

On the pros side we have:

  1. Much easier to provide high quality WYSIWYG
  2. Much easier to provide editors for other dialects, such as WikiMedia and WikiCreole
  3. Eliminate loss of formatting information when cut-pasting existing HTML

-- CrawfordCurrie - 31 Aug 2007

Dojo, or javascript dom apis like the one in extjs (see extjs.com) enable current page dom editing without using the dom api of the browser and will alwas produce the same XHTML code. That way you won't break the veersionning and you won't have to deal with all the different fancy ways the browser will write bold, italic, img, etc...

dojo and the extjs dom editing library just synchronize the dom of the navigator with their own version of the dom.

I'm a very regular user of JOT and believe me, the JOT wysiwyg editor is just working very well. But in WYSIWYG : no macros, no variables, no complex component (except tables). If you want to write jotscript apps, you need to rurn back to text editing or xml editing, and any page that contains one single line of jotscript is no more editable using the WYSIWYG.

I can share my jot account with crawford or peter, colas or any person I know in case you want to try and have no more an account.

-- MichelBuffa - 31 Aug 2007

I would emit a word of warning: TML is actually needed for a lot more things than just editing: most %SEARCHes and formatted searches that are the bread an butter of TWiki apps rely a lot of TML and its line-orientation.

Thus you cannot use the raw (X)HTML saved by the WYSIWYG editors as it will break all the structure of TWiki (as the saved HTML will depend on the editor and/or the browser version and/or the client OS), making %SEARCH impossible.

So Crawford suggestion make sense. But the store format should be also designed to allow for a %SEARCH as powerful as we have today. Actually, it could be a kind of "XTML": just like XHTML can be seen as an HTML-related syntax, but made specifically for machine parsing while still being human-readbale. In our case, the goal of XTML would be to be easily parsed by WYSIWYG editors, but also by (new versions of) %SEARCH, %INCLUDE, and all the %-... - the tscript? - that make TWiki powerful.

Note that this is a long-term goal. A shorter goalk could be to go the Jotspot way: only wysiwyg edit pages with no %-vars, perhaps allowing only %INCLUDEs to be able for WYSIWYG users to include "active pages" prepared by TWiki experts?

-- ColasNahaboo - 01 Sep 2007

That is an excellent point. As I said above, the stored form can easily be canonicalised to be independent of the browser/editor, but moving to a structured store format still has major issues for searching. If they were forced to write simple searches as XQuerys, or using SQL, I imagine people would soon run away!

w.r.t "going the jotspot way" - we already have WYSIWYG_EXCLUDE, which gives us a lot of control over what can and what cannot be in pages, and still have them WYSIWYG-able. But the one case which is not adequately catered for is the one case I have found where you need a text mode edit - and that is when you want to enter TML directly in the topic. But that is off at a tangent to the discussion subject here.

At the end of the day I don't think DropTML is going to happen, because if we changed the store format in TWiki, it would have such a knock-on effect that what you ended up with would no longer be TWiki.

At the root of TWiki's popularity has been its apparent simplicity. However beneath this facade, TWiki is a complex mass of compromises, and every compromise has a history of other compromises behind it, peppered with the occasional naive or shortsighted design decision. As I have often experienced, the need to maintain compatibility soon bites so hard that you realise the best thing to do would be to give in, and start again. So DropTML has been an interesting thought experiment, but it is not, IMHO, a realistic proposition for TWiki. Especially since (IMHO) we are so close to really good WYSIWYG support without this change.

I'd love it if someone could prove me wrong, of course.

-- CrawfordCurrie - 01 Sep 2007

The idea of only enabling Wysiwyg for pages without %VARIABLES does not hold in practical.

In our TWiki it is about maybe 10% of the pages you could edit then. There is always some neat little feature on a topic. And it makes sense it is like this. The main reason for using TWiki instead of Word and saving a word files on a web based file share is exactly the possibilities to do intelligent stuff in TWiki. So when my team write their weekly status in TWiki it is because the schedule has a nice calendar popup - there is a section that shows the milestones from the other projects (SEARCH), there are named sections that can be included in a rolled up hotline and in an autogenerated project management plan etc etc.

What I still think is an area not fully explored is the ability to sectional edit in Wysiwyg with only clean text areas being accessible in Wysiwyg and once you want to design the twiki app part of the topic you go in normal classic edit mode. Then we could also make technology that matches what you want to do. When you edit the text you may get a Javascript/Ajax based thing. When you edit a table you could get a Java applet that would allow dragging columnwidths, when you edit a drawing a TWikiDraw alike applet is presented etc.

-- KennethLavrsen - 01 Sep 2007

Good point, Kenneth, I'd also prefer to separate twiki apps from wysiwyg areas using a clean cut.

-- MichaelDaum - 01 Sep 2007

Kenneth, any chance you could explain in some more detail how your "weekly status" application is set up? It might give me - and others - ideas of how to convince our engineers to use a wiki rather than word or excel file.

-- JohnFitzpatrick - 05 Sep 2007

Edit | Attach | Watch | Print version | History: r18 < r17 < r16 < r15 < r14 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r18 - 2007-09-05 - JohnFitzpatrick
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2018 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.