What You See Is Not What You Get

Or; Why TWiki can't do Wysiwyg properly

There has been a fair amount of recent discussion on Wysiwyg, and demands for Wysiwyg editing in TWiki. Since I'm probably the most experienced at trying to get Wysiwyg going in TWiki I thought it would be useful to summarise where we are at, and where I think we need to go, with respect to Wysiwyg.

History

Wikis, and TWiki is no exception, started out in a world where the only browser support for editing was the textarea. A textarea is just a plain-text edit box, and is really very, very crude. It allows full control over text formatting, but only as long as all you ever want is 10 point Courier with no line breaks.

To address this problem, the inventor of wikis came up with the concept of a simple markup language that could be used in the textarea to apply formatting to text. Markup languages work on the principle of "decorating" text with extra instructions that describe how to format the text. Markup languages have a long and venerable history; most word processors started out life as markup processors. Indeed, HTML is a markup language (Hyper Text Markup Language).

However markup languages have had to put features ahead of usability. This is no good for wikis, where the ease of use is the key selling point, so wikis were given a new markup language. This language is much, much simpler than - say - HTML, but is accordingly much easier to learn, and faster to type. There has never been any standard for markup in wikis, so every wiki implementation has its own unique markup. Wikis work by converting this simple markup language into the more complex language (HTML) that browsers can understand.

TWiki is no exception in this respect, except that it takes markup a step further. Not only can a TWiki user enter text using simple markup, they can also use TWiki variables that, when expanded, generate new TWiki markup. Or bits of markup. Or even modify markup already in the content. TWiki topics can be programs that generate new content when they are run.

Since the invention of wikis and markup, browsers have moved ahead. These advances have stuttered rather than sung, as browsers were still being seen as a "window" onto static text rather than a means to generate new content. With the advent of Web 2.0 this is changing, and the last few years have seen major improvements in browser support for more sophisticated editing.

Nowadays most browsers support some sort of Wysiwyg editing, usually through some sort of plugin-in editor that sits on top of textareas, and presents their contents as Wysiwyg editable HTML instead of simple plain text. The arrival of these editor plugins has been seen as a way to improve the editing experience for wiki users.

TWiki and Wysiwyg

To date there have been various plugins and contribs that have aimed to improve the editing experience (see EditingSummaryNeeded). None is totally successful, and none provides the full Wysiwyg editing experience. Why not? Well, here's the key issue.

TWiki Variables and Wysiwyg

As mentioned above, TWiki has an engine that converts from the simple markup language (TML) used in the plain text editor to something the editor plugins can understand (HTML). Since HTML is what browsers eat for breakfast, it would seem on the face of it to be straightforward to integrate an HTML editor. However this conversion is what is referred to as a "lossy conversion", because while it is possible to fully translate TML to HTML, the reverse translation is not possible. For example, consider this TWiki topic:

   * Bull
<ul><li>Bull</li></ul>

both produce identical HTML.

Even worse, TWiki variables turn TWiki topics into programs:

   * Set A = Rabbits
   * Set B = Hares
%A% make excellent hats, and %B% make excellent stews.

This is represented in HTML as

Rabbits make excellent hats, and Hares make excellent stews.

Clearly there is no way to get back to the original topic content from this output. These are very simple examples; but it's very easy to write more complex TML that is a total nightmare.

So, for a plugin Wysiwyg editor to support editing TWiki topics that contain TWiki variables, it has to have some way of reconstructing the original topic from the HTML it edits. In fact, two things are needed:

1. Some way to reverse the TML (without TWiki variables) -> HTML transformation
2. Some way to reverse the action of TWiki variables, or, at worst, protect them

What has been tried

There have been a range of attempts at improving the editing experience, ranging from simple enhancements to the textarea, through integrations of external editors, through to attempts at full Wysiwyg. Since this topic is about Wysiwyg we'll focus on that end of the spectrum.

All the attempts at Wysiwyg editing in TWiki to date have been based on integrating third party Wysiwyg editors, written in Javascript, with TWiki using the WysiwygPlugin. This plugin supports bidirectional translation between TML and HTML. The development of this plugin was sponsored by ILOG, and was a huge effort, involving many, many hours of careful coding and testing. It is reasonably successful, as evidenced by the number of sites using Wysiwyg on a daily basis. However it is also deeply flawed; to work, it has to disable Wysiwyg editing of topics containing complex TWiki variables. this isn't because it can't cope with the translation; it's because the third party editors are designed to edit HTML, and not HTML-with-embedded-TWiki-variables. Add to this a large number of obscure edge- and corner- cases where the translation from HTML to TML is imperfect, and many people see the Wysiwyg as "badly broken" (to quote a TWiki customer advocate).

Because of the huge complexity of the translation process, it is beyond reasonable expectation that the original author (me) should be expected to support and extend the WysiwygPlugin without financial support. Since ILOG stopped sponsoring the project, I have put a lot more hours of my own into the work. However no-one else has stepped up to the plate, to provide either sponsorship or code (or even testing), so it has stagnated.

Possible directions

There are three possible directions for improving the editing experience for TWiki users:

1. Edit TML
2. Support work already done
3. Abandon TML

Edit TML

The PowerEditPlugin was an early attempt (using Java) to write an editor that supported editing of TML directly in a what - you - see - is - like - what - you - get way. Other attempts have tried to do something similar - though much less powerful - in Javascript (the SmartEditAddOn). Personally I believe that a month of solid work could produce a good Javascript TML editor based on the PowerEditPlugin that would provide a less-than-Wysiwyg, but still vastly improved editing experience. Another month would port it to all the browsers.

Support work already done

To get from where we are today to a really nice editing experience using the WysiwygPlugin is not a huge leap. The problem is 95% solved. All is required to complete a full Wysiwyg solution (including TWiki variables) is:

• For the editor plugins to honour the protection for TWiki variables that the WysiwygPlugin applies. This requires Javascript skills, and should be no more than 7 days work.
• Further extension of the WysiwygPlugin test set, and dedicated user testing and feedback. This should be a community effort.

Abandon TML

Another out-of-the-box approach is to abandon TML altogether. Move instead to a highly constrained set of extensions to HTML (i.e. add new private tags to HTML). Topics would be stored using this "super HTML", and TWiki features, such as automatic topic linking, would either be handled by the editor or simply forgotten about. The WysiwygPlugin could easily be modified to generate this new "super-markup" from existing TWiki topics to support migration, but of course some features would be lost - some TWiki applications rely on being able to rewrite the topic to work. This of course would make non -Wywisyg editing of TWiki topics a thing of the past, but it would enable a vastly improved editing experience for most users.

Where I stand

My personal preference is to finish the WysiwygPlugin, but I am not prepared to put any more unsponsored effort into it; I have already put in far too much free time.

I'd love to take a crack at a Javascript TML editor, but I'm not doing that without sponsorship for pretty much the same reason. It would be an excellent project for anyone who is prepared to throw it away if it doesn't work. I might even take it on as a not-for-free TWiki extension.

If we were to abandon TML, TWiki would get a heck of a lot faster. It would also get a lot simpler. But I just can't see TWiki-ites going for this.

-- Contributors: CrawfordCurrie - 17 Apr 2007

I am	I want to sponsor:			I'm prepared to put up:
	a TML editor in JS	!WysiwygPlugin	abandoning TML
ATWikiUser				$5,000 (specimen)

Discussion

Perhaps worth to mention is another option that is used by WordPress: show a preview of the generated page below the editing area.

-- ArthurClemens - 17 Apr 2007

That's how PowerEditPlugin works.

-- CrawfordCurrie - 17 Apr 2007

Crawford. Thanks for the overview.

Taking up the thread from EditingSummaryNeeded it is important for me to emphasize that the critique of the current Wysiwyg is not an expression of a "job badly done" but an expression of a "job needing more work to get finished". What we have today is better than nothing. But as you also say yourself, just a little bit more than plain text and basic formatting and the plugin has to be disabled in order not to destroy anything in a TWiki app after en edit/save with Wysiwyg.

I think it is important that we set the level of expectations right. TWiki and its TML is not going to go away. That would be to loose the main strength of TWiki. What-you-see-is-what-you-get is never going to happen in the basic meaning of the words. As Crawford points out in the example with the twiki variables above you cannot both have programming and Wysiwyg in the same application.

And personally that is not what I expect.

I expect TWiki to have two modes of operation. Geek/nerd/text/programming mode and Wysiawyg (a=almost) mode.

In Wysiawyg I expect to see

• TML as TML. Not any attempt in interpreting what is going on except.
• I want to see headline as headline instead of code
• I want to see bold, italic and monospaced as formatted text instead of *, _ and = around them.
• I want to see tables as tables I can add rows and columns to and cells in which I can write anything
• I want a new line to mean a new line
• I want to see bullets as bullets. Also nested.
• I expect a topic opened and saved to not having altered or destroyed any twiki variables.

• I do not expect to be able to write HTML in this mode.
• I do not expect to see search results in Wysiawyg mode. I expect to see the raw twiki variables.
• I do not expect to see twiki variables expanded when editing.

I find it perfectly OK that the Wysiawyg is a very limited featured thing showing only the basic formatting and tables as Wysiwyg because this is what is a pain for the newbies and occational users and the PHBs (pointy haired bosses).

If we can help people editing TWiki topic with the basic formatting and just show anything else as the raw TWiki Variables etc etc then I find we have what we need. And I see no need to be able to jump between HTML and TML. That is a near impossible task.

And finally - you cannot make this feature as a hobby project. There is a lot of specifying the features, a lot of Perl code, a lot of Javascript code, and and lot of test cases to be produced and I evaluate the complete job including specs, test cases and tests to be several staff years. Remember you can contribute with resources many ways: code yourself, spec yourself, write test cases yourself, find some staff in your companies and assign them to contribute, and finally and most obviously - fund the experts that have already worked a lot on this task.

-- KennethLavrsen - 17 Apr 2007

Dividing topic parts into "wysiwyg-safe" and "non-safe" parts and applying wysiwyg to the safe parts only has worked ok for us (I don't think this approach is mentioned in the alternatives).

To some extent the current situation can be compared to other parsing text formatters, i.e. LaTeX. Also for LaTeX a lot of effort has gone into different wysiwyg editors, but current status is still sub-optimal and editors like http://www.bakomatex.com/ are state of the art still (~CPU-intensive continuous rendering for preview along with source display).

I have been amazed at how many browser-issues were with the WysiwygPlugin - and how many were "un-fixable". Many issues showed up with IE along the way as not many developers use that.

I think to help ourselves (and the eventual developer taking on the effort) basing next efforts on a framework that possibly offers (semi-)automatic testing on various browser-platforms would help a steady progress tremendeously.

-- SteffenPoulsen - 17 Apr 2007

Automatic testing frameworks for different browsers is a great idea. I look forward to it; TWiki as a whole needs that, not just Wysiwyg. My gut tells me it's a fairly large project, though.

• Agreed. -- SteffenPoulsen - 17 Apr 2007

re: I have been amazed at how many browser-issues were with the WysiwygPlugin - no, you have been amazed how many browser issues there have been with the KupuContrib. Please don't confuse the WysiwygPlugin - which is a TML -> HTML -> TML convertor - with the editor, which can be one of several (Kupu, Wikiwyg, Xinha, TinyMCE). This confusion came about because I originally shipped the Kupu editor in the WysiwygPlugin - it has since been separated.

• Sorry, you are right. I reported probably about 30 of the items against WysiwygPlugin and I was only referring to surprisingly many of the item showing different behaviours on different browsers. -- SteffenPoulsen - 17 Apr 2007

Also, I'm actually quite surprised to hear you say that, because I have only seen a handful of browser issues - see http://develop.twiki.org/~twiki4/cgi-bin/view/Bugs/WysiwygPlugin. Some are killers (the IE newline issue) I grant you.

As a developer, the main issue I have had has been the usual one - people firing reports and criticism without providing testcases or adequate diagnosis, or being prepared to offer test or debug support. Everyone wants it to work, but no-one is prepared to help make it work. And so far, the discussion in this topic is managing to highlight that fact. Steffen, I haven't seen any contribution back that would enable others to divide their topics into "safe" and "unsafe" sections....?

• It is just the patch suggested at TWiki:Plugins.WysiwygPluginDevArchive by Craig (which didn't seem to catch interest then, but made us stick to the version at the time). -- SteffenPoulsen - 17 Apr 2007

-- CrawfordCurrie - 17 Apr 2007

(I added this comment before CC was finished with his reaction)

Browser issues could also be minimized by using Flash. Flash compilers like MTASC (for Windows, Linux and Mac) let you compile ActionScript code without Adobe Flash.

-- ArthurClemens - 17 Apr 2007

Crawford, thanks for putting together this excellent summary of where things are at with Wysiwyg editing.

First of all, as I said in EditingSummaryNeeded, we need to make sure that we remind ourselves (as Crawford pointed out in the History section above) that this is a problem across all (non-commercial at least) wikis.

Lack of Wysiwyg editing is not driving us (the users) away.

I see the current editing facilities as "the bottom rung of the ladder". I believe that there are many more rungs to this ladder and the going will get easier eventually.

But, as a customer advocate, let me start by detailing what I have found to be the most annoying aspects of the current editing situation. There are not necessarily fixed with a Wysiwyg editor, by the way.

Tables

We use tables very heavily here in all kinds of documents. It is very easy for even relatively simple tables to become unreadable in TML. While EditTablePlugin is a help, it is still limited. Now that the TablePlugin functionality is now in the Core, any new editor should allow for better editing of tables. That should include: Resizing, cell colours, borders, bolding (heading) whole rows or columns, aligning whole rows or columns.

Images

A lot of our new development is GUI based, so screenshots are becoming a must. Adding graphics in the normal editing is currently not possible and instead requires a cycle of Saving, Attaching and back into edit.

Contextual Editing

This is probably the biggest complaint I get. The fact that even a CTRL-K (Checkpoint) takes you back to the beginning of the document (which could be very large). My users complain about spending a lot of time getting back to where they were editing, just before. While this can be partly achieved using something like the DblClickEditPlugin in association with the SectionalEditPlugin, this is not a perfect solution as you cannot then easily move to another part of the document to edit. What would be really good would be the Double-click edit which takes you into a full edit with the cursor positioned at the point where you double-clicked. That alone would make a lot of people here much happier about the editing.

I must say that things like SmartEditAddOn have made editing a lot nicer.

I particularly like the way you can search for, preview and insert links to other topics while inside the editor. It also makes the textarea a lot smarter with it's auto-indenting and auto-continue of bullets and numbered lists.

What we need is an Editing priorities topic where we prioritise the fixing of the various issues with editing.

-- DuncanKinnear - 18 Apr 2007

By the way, if we were voting, I'd vote for smarter TML editing (option 1) with a textarea that functioned more like a good text editor (like Textpad) with search and replace, auto-indenting, etc.

By the way, something that has really helped me with editing productivity is the "Clippings" Firefox extension. I cannot praise it enough. It is a very good example of a very simple product adding lots of value. Check it out.

SmartEditAddOn has a similar feature, but it is a total pain to set up and does not cope with multi-line 'clippings'.

Anyway, I'm looking forward to advancing this whole topic, even if it just helps new users figure out what's there and what works/helps.

-- DuncanKinnear - 18 Apr 2007

I'm delighted with the constructive responses to my original post. Let's leave any vestiges of bitching here, and take the discussion on requirements for editing to RequirementsForEditing. I included a snippet of my backgrounder there, and copied Duncan's requirements over there as the basis for discussion. I'd really like to hear from users / customer advocates there.

CC