I have started to provide additional documentation for TWiki in a literate programming style using Leo (Literate Editor with Outlines).

Shortly, maybe even today, I will attach my very preliminary .leo file to this page for download and review.

The file is definitely preliminary and unfinished. (Only some of the TWiki files have been imported; comments may be incorrect; comments are unfinished, incomplete, and poorly written.) The reason for publishing it at this time is two fold:

• Let developers see what Leo is like so they can consider whether they might be willing to continue TWiki development under Leo.

• Let developers and others (including newbies to Perl and TWiki) see where I'm heading and see whether anyone wants to help in the effort.

My goals are to:

• Learn Perl
• Learn TWiki (so as to be better able to install, maintain, and possibly modify it)
• Help others to learn Perl and TWiki by documenting my learning experience

My current plan is to document my learning experience via (possibly) extensive commenting of the TWiki code in Leo. Leo provides a means to allow extensive comments to be added with minimal interference to the efforts of experienced developers.

Aside: Having the developers choose not to develop under Leo will not preclude the goal of documenting my learning in Leo. Having the developers on board would make things better. I will continue to head in this direction until I've learned Perl and TWiki to the extent I want to. I will "publish" the (partially) documented TWiki on WikiLearn (and/or twiki.org) for use (and further development of the documentation) by anyone who is interested.

If you want to review the file, you will have to download and install a copy of Leo. Leo is available for Windows, Linux, and the Mac, but the only version I have actually used is Leo for Windows, version 2.3.

I have limited the markup used to the Leo 1 style, so there is a reasonable chance that any of the other Leos will be able to read my file, although I suspect you may need to convert the line endings appropriately for different operating systems.

(Leo development is currently concentrated on a wxWindows/wxPython version, and a few versions have been released. With a wxWindows version, cross platform availability is built-in for Windows, Linux, and the Mac, IIUC.)

Leo download page:

• http://sourceforge.net/projects/leo/

Leo manual:

• http://personalpages.tds.net/~edream/intro.html#anchor887874

The Leo manual also comes as a .leo file with whatever version of Leo you download. If you install Leo, you can probably browse the TWiki code without reading the manual. I have some hints at the beginning of the TWiki .leo file about settings to change to make viewing easier. (Unfortunately, in the current version, those preference settings cannot be saved between invocations of Leo.)

-- RandyKramer - 30 Sep 2001

• TWiki20010901: Very preliminary version of .leo file for TWiki

• view: view file tangled from above file -- you can see the few changes that Leo makes in a tangled file. (In case you don't want to download Leo, or you can't get your version of Leo to tangle the files.)

I just took a look at this using Leo.py 2.0 (equivelant to LeoPC 2.6), and it looks ok, and I have a couple of observations:

• • Attempting to back code into literate programming doesn't seem to show the flow of the program as well as deriving the code from a literate document.
  • Leo still has a lot of room for improvement. The font and gutter options suggested in the (elderly) sample twiki file are not currently available in (what is presumably a later) Leo (which is very unfortunate).
  • In spite of shortcomings, outlined code development show a lot of promise.

Interested parties might also find "buzz" of interest. It's a logical descendent of Woody and Leo, but is more web-based. http://buzz.sourceforge.net/. Buzz shows potential for syncronicity with TWiki imo.

Another, more outline specific app, is treeline: http://www.xecu.net/dougbell/treeline/index.html

-- DavidLeBlanc - 04 May 2002

David,

Thanks for taking a look at the .leo file and posting here! (I'm sorry, it was probably rather painful -- seeing your comments prompted me to go back and look at the file again -- there is a lot of room for improvement.)

I agree with your point about "back coding" (or "back documenting") -- I don't believe it is impossible though. I think someone who is familiar with the code could do much better than someone (me) who was trying to document the code while learning it (and I still don't really know the TWiki code).

I've set the effort aside since shortly after I uploaded it -- I have a slightly more recent file with some more comments in it, and if someone was really interested in it (doing something with it, carrying it further), I'd be willing to spend a few hours cleaning up problems that are obvious to me now (including making it more clear that only some of the nodes in a Leo outline end up getting tangled into the code -- maybe that's clear, but it didn't jump out at me as I reread today).

I hope I didn't post the wrong view file -- I looked at it today and didn't notice any Leo "sentinels". I then tried tangling the view file again from the version of the .leo file I have at home and it didn't work. My memory is faulty -- I guess there is some chance that when Leo2 tangles a file it does not put any sentinels in it, but I don't remember that being the case.

I took a brief look at buzz and treeline. Look like nice tools with nice potential, but in my brief look I didn't find any evidence that anyone had tried to use them as literate programming tools. It would be interesting to get the comments of anyone who has.

-- RandyKramer - 04 May 2002

Randy;

It wasn't that painful, except to the extent that Leo itself is alpha (beta only by courtesy) quality. I'd very much like to improve Leo itself. It would also be nice if it had more doc on how to use/understand it!

I wondered about the view file, since I did expect to see some comments that would look like information for Leo, but didn't notice any in my brief glance - they could be "vanished" by twiki itself during rendering though...?

Buzz has some nodes in the outline that comes up on it's home page that are quite literate programming expositions of Buzz itself. Treeline otoh is pretty much just a pure outliner.

Are you using the Borland PC version of Leo or the Python version?

-- DavidLeBlanc - 04 May 2002

David,

The last version of Leo I downloaded / used was Leo for Windows version 2.2, dated September 16, 2001. More later -- I want to look for some things -- I have a more up-to-date .leo file for TWiki, and I also have .leo files with:

• some notes on using / learning Leo (IIRC, more than are in the file currently available here)
• some notes on using / learning Perl (probably not of interest to you, might be of interest to someone)

When I find them and look them over, I'll post a notice here -- if anybody wants them I can email them to you (or consider uploading them here -- I'm just hesitant to clutter twiki.org with a lot of large files).

-- RandyKramer - 05 May 2002

Randy;

The last version of Leo or of TWiki? I was asking if you're using the Borland (PC) version of Leo or the Python (Leo.py) version of Leo?

Oops, sorry -- it was Leo for Windows version 2.2, dated September 16, 2001 -- must have had TWiki on my mind!

I'm not sure what you mean by the "file currently available here".

Sorry, that wasn't very clear -- I meant the version of my .leo file (TWiki20010901) that is attached to this page -- I continued development on it after I uploaded it here (not sure how far I carried it, but didn't bother uploading it again because no one had expressed any interest.

I just discovered that the user's manual on http://personalpages.tds.net/~edream/front.html is different from the leodoc.leo file that comes with either Leo1 or Leo2, and the web version has some good internals information for Leo (gawd does his xml su ...err... "be bad").

Just to avoid misleading you, I set Leo aside in October / November of last year, so I haven't kept up-to-date on it. I would hope there is better documentation somewhere. ;-)

I'd be very interested in your notes about learning/using Leo, and if you'd rather email them, "User.DavidLeBlanc" has my eddress (don't forget to remove the spam block).

Ok, I'll email them as soon as I can -- I'm not sure how much is different than the .leo file attached here -- I should have maintained the file in CVS, but didn't really decide how to approach that.

I think that creating a new app inspired by Leo might be a very interesting idea. What do you think?

I'm sure, but are you thinking of a better Leo, or some sort of wiki/Leo combination? Either one would be promising, or even some other apps using outlining as the underlying basis for organization.

BTW, it came to mind that one way to get TWiki into Leo was to create a script that gets the names of all of the files and perl subroutines in TWiki and makes them into items in the outline - probably by inserting those leo "sentinels" as well as other xml-ish wrappings he does to create a .leo flie. I think Perl has some parsing stuff that might assist with that. "leopy.leo" might be a template for how to do this.

Yes, I believe that can be done. I had a procedure worked out for moving pieces of TWiki into Leo, but I can't recall it -- I often do things like that with macros in Word.

-- DavidLeBlanc - 06 May 2002
-- Main.RandyKramer - 07 May 2002

I've set TWikiInLeo aside (i.e., I'm not actively working on it). Frankly, I had hoped to develop some interest in it among the developers so that they might continue development of TWiki using Leo, so that any documentation (including newbie Perl and newbie Twiki) could continue to "reside" in the TWiki codebase without interfering (excessively) with the developers efforts.

One problem was deciding how to use Leo with something like CVS. Ed Ream, the developer of Leo, had not (IMHO) worked out a good way to do this the last time I looked into it.

-- RandyKramer - 24 Jun 2002 ([ MartinCleaver 24 Jun 2002 ] refactored from CodevDocumentationProject)

If you are both convinced as to the value of having Leo annotations rolled back into the code base, then I suggest that you propose this to the CoreTeam, stating how much on-going work it would take and giving us the benefits.

With regard to maintaining the DotLeo file in CVS, why is it a problem? Surely you just check them in? Also, have you considered the pros and cons as to storing them in TWiki instead? The mantra of "Plugin" just occurred to me; so I am wondering, what could a plugin do?

-- MartinCleaver - 24 Jun 2002

Speaking for myself, and the originator of the TWikiInLeo effort, I'm not currently convinced it has value, or let's say not enough value to outweigh the extra work to the developers.

I do think that both the effort and the result would be worth it to newbies to Perl and TWiki. (I say the effort because a newbie to Perl or the TWiki codebase would learn a lot from slogging through the code to the extent necessary to document it at the level of detail I had in mind, and subsequent newbies would benefit from not needing to expend the same effort to slog through it.)

I once had this "romantic" notion that:

• it would be fairly easy for me to learn enough about Perl, TWiki, cgi, html, etc. to make meaningful contributions to TWiki (and enough about C++ and the AbiWord codebase to make meaningful contributions to AbiWord) (both at the code level),
• and that some fairly large number of people might want to do the same thing.

I'm currently somewhat discouraged because of the difficulty and beginning to think this it is unlikely there are a lot of other people that want to help develop and maintain TWiki -- some maybe, but not a lot. Then, what's the benefit to creating a lot of documentation to help newbies to Perl and TWiki? (I might find my enthusiasm rekindled, but right now it's at a rather low level.)

Possibly providing that level of documentation of the TWiki code base might be a good learning tool for Perl newbies regardless of whether they intend to continue to work on TWiki or not, but, I have a hard time slogging through example code that I'm really not terribly interested in.

So why did I write the above? I'm not entirely sure. If there are other Perl and TWiki newbies out there who are enthusiastic about documenting TWiki in Leo, I'd be happy to try to help (taking proper account of the discussion on CVS, below). But, there are still other reasons I'm not very eager, including:

• I started documenting a beta version of TWiki (IIRC, it was the 20010315 version), shortly before the next version of TWiki was due out. I was OK with that (I thought). Then the 20010109 version came out, and I started trying to modify my comments to suit that. It was not fun, and I didn't get very far (due to loss of enthusiasm prompted by confusion). (And, it wasn't too long before "we" realized that there would be another ("bug fix") release fairly soon.) From my point of view, once the detailed comments exist, it's better that those comments be maintained as the code is maintained. (Well, maybe if I had thoroughly understood the TWiki code base for one version, and understood the changes in the next version, I might have been able to more easily modify the comments.)
• There is discussion about major changes to TWiki, like going to an object oriented approach, or the kind of refactoring you (MartinCleaver) are talking about (SlimDownFatCgiScripts).
• From my point of view, if we want the level of documentation I originally envisioned (newbie Perl and TWiki level) it would be best to have a moratorium on development while the documentation is created to match the TWiki code base (so the documentors, if separate from the developers, are not aiming at a moving target) and from then on have the developers, either themselves or with help from "documentors" maintain the comments at the same time as the code. (If there were such a thing as documentors separate from developers, modifications to the working arrangements might be made -- maybe a developer would develop a small improvement, debug it, then hand it off to a "documentor" for finishing the documentation before "officially" making it part of TWiki. (But, in any case, the developer should at least start the documentation, or provide good hints and "consultation" to the documentor until the code is documented at the desired level.)
• IMO, until developers want to see the code documented at that level, and want to use a tool something like Leo to facilitate the maintenance of newbie comments alongside the code (or for their own convenience in navigating and editing the code), there are not enough newbie TWiki developer "wannabees" to justify the added effort.

Still, my efforts were somewhat worthwhile to myself -- I learned some Perl code and some of the TWiki codebase (not enough to remember, but some, and enough to decide that I don't really have time now to learn a lot more, like enough to contribute).

And, I think the next newbie TWiki developer "wannabee" could benefit from the same sort of effort, but I think the documentation you (MartinCleaver) are aiming at is more appropriate at this time. You are not aiming at the rank Perl newbie, but are providing (IIUC) "overview" documentation of how all the modules and files fit together and what each accomplishes. This is very useful (IMHO) to Perl and TWiki newbies, and, I would think, for "casual" TWiki developers. ("Hard core" TWiki developers may maintain such knowledge in their heads, continuously updated as they watch CVS checkins and so forth.)

So, I applaud your efforts, and will try to support them at some level. On the other hand, if someone else comes along and wants to make a renewed effort at moving the codebase into Leo or something similar, I will try to support those efforts as well, or at least until we learn enough to make a decision as to which approach makes the most sense. (The two approaches I'm thinking of are maintaining the TWiki code base in Leo or maintaining it in TWiki. I see the effort you are starting (Martin) as being a step in the direction of moving the TWiki code base into TWiki, but I also recognize that your current goal is not to go that far (IIUC), and, that may never be an appropriate goal -- the documentation you are aiming at might always be a high level sort of thing, maintained in TWiki, with (as you mentioned somewhere) links or pointers of some sort into the actual code. (And, maybe it will evolve to the point where the actual code is maintained in TWiki, like Ward Cunningham's one version of wiki whose code is maintained in wiki.)

Anyway, that's more than enough rambling. (As usual, I should probably come back and edit the above some time in the (near) future.)

With regard to maintaining the DotLeo file in CVS, why is it a problem? Surely you just check them in?

Well, I'll just give an overview of the problem, from memory (which means I probably have some important details wrong):

There are two major versions of Leo (or two major versions of the Leo markup). In both Leo 1 and Leo 2, your code can be organized into an outline form, and you can add extra nodes to the outline, almost anywhere, to contain, for example, extra comments that will not be "tangled" into the code.

OK, I guess I should digress just a little bit more. Leo (Literate Editor with Outlines) is intended to be a literate programming tool, after the literate programming concept espoused by Donald Knuth. In Knuth's original concept (in my own words), a special document is created that it a markup of the code with a markup of comments, probably not in a form easily read by a person or useful to a computer. From that, two types of documents are created by an automatic process based on the markup. By "tangling" that document, source code files are created for the target compiler in the format and sequence required by the compiler. By "weaving" that document, a "book like" (my choice of words (meaning that perhaps no one else would agree with that wording)) description of the program is created. So now, you have three representations of the same program. (So you are violating the "only in one place" rule.) Simply, which one do you put under CVS control, or do put all of them under CVS control? And which one do you work in? Do you debug in that marked up document that is the source for the other two, or do you debug in the derived source, and then carry changes back to that marked up document?

Leo is somewhat different in that it tries to eliminate one or two of those three representations. In both versions of Leo, the intent is that the outline is both the marked up source document from which the code is tangled, but, in addition, the outline is intended to be the human readable document. So, there is no "weaving" to get the human readable form, you view (and edit) the code (and comments and other documentation) in the Leo outline. You can navigate by expanding and contracting nodes and subnodes. You can have high level documentation at high levels of the outline, and include the actual code at "lower" levels in the same outline.

I think if I thoroughly understood the TWiki codebase, I could make a Leo outline to make the TWiki codebase easy to understand. It would include comments at the level MartinCleaver is working towards. (There are some rules about which nodes can go where that may constrain this a little, and these constraints vary between Leo 1 and Leo 2, these are touched on below.)

• Leo 1: As you've learned by now, Leo 1 has two representations of the code, one as the Leo outline, and one as the source code files which are derived ("tangled") from the Leo outline. So, you have two "things" which might be considered the source document (depending on your viewpoint) and two things which might be put under CVS control. The disadvantage of putting the Leo outline under source code control is that it is one big file that contains all the source code for the entire program plus (potentially) a lot of comments. Checkins and checkouts would be slow, and any change to any single line of code, no matter which subroutine it might be part of, is recorded as a change to this one big file. (Aside: At the time I was working in Leo, the Leo 1 markup was the only one I managed to learn enough about to actually use. It has the advantage over Leo 2 (IIUC) that you can organize code blocks in an order other than that required for "compilation".)

• Leo 2: Leo 2 tries to address some of the problems mentioned above for Leo1. It does so by taking the source code out of the Leo outline, or, trying to explain that better, the compilable source code files are a part of the Leo outline, and not duplicated in it. Conceptually, the Leo outline contains comments and pointers to the source code files, rather than the source code files themselves. Now, code and documentation (including comments beyond those in the code) are each maintained in only one place -- the code in the compilable source code files and the other documentation in the Leo outline. However, because of the "pointers" ("sentinels") between the two, as you browse the outline you also see and can edit the compilable source code. And, the advantage is that now you have a fairly small Leo outline file (as it does not include the compilable code) and multiple small compilable source code files that can be maintained under CVS control. (IIRC, you could also have some of the comments maintained in files treated like the source code files -- viewable and editable from Leo, but maintained as separate files.)

Some other comments about Leo2:

• Another slight advantage (maybe!) is that some small changes in code can be made in the compilable source code files rather than firing up Leo. (Ed Ream explains somewhere what those are, but my very conservative rule of thumb (since I don't have real experience with Leo 2) is to stick with changes like fixing typos on a single line. You might get away with adding, deleting, or reordering lines within something like a single subroutine, but if you start reordering subroutines, or moving code from one file to another, I'm fairly certain you will cause problems (for Leo -- your code might still compile and run, but the Leo outline might be corrupted, possibly beyond easy repair).
• As mentioned somewhere above, I stuck with the Leo 1 markup for two reasons:
  • at the time, documentation was not sufficient for me to understand how to use Leo 2 markup
  • Leo 2 is restricted to displaying the code in compilation order, whereas Leo 1 can display the code in any arbitrary order (as needed to make it easy for a person to understand)

Also, have you considered the pros and cons as to storing them in TWiki instead?

Well, yes and no. At the time I started my effort in Leo, there were no real "hooks" in TWiki, to for example, make changes to code on a TWiki page and then run that code. Ward Cunningham does have a wiki with those kind of hooks (it is not the C2 wiki). There still aren't such hooks in TWiki, but if there were, my next concern would be with speed. As I access the Internet over a dial up modem and will do so for the forseeable future, and experience frequent disconnections or interruptions of one sort or another, editing a program on TWiki might be frustrating. (I'm too far from my central office for DSL, and cable modems in my area are one way only (but, real soon now (from three years ago)) we will have two way modems in my area ) Generally, I think I prefer the CVS checkout, modify, test, debug, CVS checkin workcycle, where the modify, test, debug is done locally with no dependence on the speed of my Internet connection. Of course, I can imagine ways of working with the code on TWiki that would provide a similar local modify, test, debug workcycle, preceded by a checkout and followed by a checkin to TWiki.

-- RandyKramer - 24 Jun 2002

On the other hand, having spent just a little time navigating from TWiki page to page looking at some of the documentation modules being added, having that same documentation in Leo would make it a lot easier / faster to navigate.

And have the benefit of some "intrinsic organization" by virtue of being organized in an outline. (For example, subroutines in one file all organized as nodes under that file, or something similar.)

-- RandyKramer - 24 Jun 2002

I found interesting idea to comment TWiki code so more people might contribute code. Questions are:

1. Do we want to do that (or better: does Core group feels urgent need to do that)? Without support from Core Group, decomenters will be shooting on the moving target. Excercise in futility, IMHO.
2. What will be the gains for spent effort? For Core Group, not much (they have all that in heads). I personally would like it to happen, I am getting more familiar with Perl and might want to contribute code, so good comments are good.
3. How we want to accomplish that? Perl has system for embedded documentation, pod. I believe there should exist/or not hard to build scripts to separate pod and source code, and publish both as HTML (pod2html and perltidy will help). So I guess we can have smart comments in perl generating InterWiki links to HTML-ed pod docs, and from HTML-ed pod to HTML-ed perl source. All might be published as twiki pages.

I might be missing some details how to make it all spinning, but making (3) working I can see as more useful and reusable project, than learning Leo. I read about Leo, it is interesting concept, but I prefer no 'smart' tool messing with my code, unless necessary. Other languages might need something like Leo, but Perl has pod.

I believe that some of the Perl gurus around should have something like (3) running? Do you want to share?

-- PeterMasiar - 25 Jun 2002

I don't know much about pod, but I think that:

• The pod approach leads to three representations of the program:
  • The code with pod comments (the "source" document)
  • The code with pod comments stripped
  • The pod comments, stripped from the code

If I (or someone else) wants the newbie level comments that I originally had in mind, my goal would be to maintain those comments in the "source" document. Which leads to the dilemma I was trying to solve with Leo:

• either the developers have a harder time working in the "source" document because of the extraneous cruft from newbie level comments (at least newbie TWiki, probably not newbie Perl as they can be moved somewhere else),
• or, the developers do their work in the code with pod comments stripped, and the source document must be maintained by "backporting" changes to it.

But, I am very interested in comments from others about PeterMasiar's comments (or mine).

-- RandyKramer - 26 Jun 2002