Personal Roadmap for Martin Cleaver

Summary

Martin's thoughts, as they occur to him.

The journey to date

A collaborative, not core team effort

By BeijingRelease, TWiki's design process reached had its architectural limits. Cairo was a strain on everyone: for the core team to incorporate changes and for the commuity to see their contributions ignored. From DakarRelease we acheived an open collaborative effort underpinned by a solid test case infrastructure. The focussed efforts have borne fruit: the work done from Cairo to Dakar is testement to that.

Professionalism of the core

• Object Oriented foundation
• Solid Test case infrastructure
• Bug tracking that works
• Code that people don't fear to change
• Subversion repository
• Tighten development schedules we - now actually do deliver functionality
• Documented, remodularise and rearchitected code (e.g. MultiLevelWikiWebs were brought into the core)
• We've reduced the amount we reinvent the wheel - CPAN modules are used where necessary (CpanPerlModulesRequirement)
• Faster development times / more active alpha testers / fuller automated test suite

Short-term goals

• Increase comprehension:
  • More use of diagrams of key TWiki components, e.g. RenderingPipeline for Plugin handling
  • Deletion / Expiration of junk content on TWiki.org
  • Rework the bugs web to use WikiWords instead of Item numbers.
• Enable the community involved in the push to release:
  • Know how to run tests
  • Create builds
  • Automate the creation of the TWikiInstallerWindowsContrib as this opens up whole new markets
  • Get TWiki winning Wiki:EasiestInstallableWikiContest
• With over 150 plugins it is simply rediculous to expect new admins to forrage through them individually, especially when so many of them duplicate the functionality of others.
  • PluginsNeedCategorizing - visit that topic to see why we are waiting
  • PluginAppraisalReport - the results must be made more prominent, on the download page, so to help people use them to make decisions. A process must be invented so people are encouraged to fill them out. In this environment where people get no pay for their efforts (Real quote: "My motivation on my plugin wained after almost no feedback") it is imperative that developers have a means to comprehend that others appreciate their work.
  • Viewfile must be used on twiki.org so we can track plugin downloads
  • Recommended plugins - we need a voting mechanism to help surface quality.
  • Alignment of plugins and extensions for Dakar: think through what should be in the core and what should be out. Some plugins need pulling in completely (VersionHistoryPlugin) others out (WebStatistics) and still others need architectural adjustments to facilitate (Swish-e / MS Index search).
• Get rid of multiple front end scripts - CommonFrontEndCgiScript
• Document SelectiveModPerl, but for FastCGI and PersistentPerl as well
• Provide a tabbed interface for each topic
  • Provide multiple content areas - e.g. summary and talk whiteboard
  • Provide split controls for each topic: the whiteboards, (including summary and talk), the settings, the permissions, and the 'watch this page' controls. Each could be on separate visual tabs on each topic.
    • Preference settings is too hidden in More Settings
• A mechanism for TWikiApplications
• Rationalise variables documentation: CleanUpTWikiVariables
• Rationalisation of skins: focussed effort to reduce the duplicated efforts. I am all for experimentation but all this amount of non-sharing by skins of what could be in a skins layer is simply wasteful. In particular, how about consolidating NatSkin and PatternSkin? See ConsolidateFunctionalityFromSkins.
  • This has begun: NatSkin now has a Cairo compatible PatternStyle
  • We need more consolidation, is NatSkin the right platform for this?
• Automated Installers for Unix platforms, Windows, Debian
  • These should all be automatically built.
    • TWikiInstallerWindowsContrib has the beginnings for Windows native,
    • TWikiVMDebianStable for Windows on Linux core
• Retire dead plugins:
  • Over the years the community at TWiki.org has changed. Some plugin authors are no longer users of those plugins. New owners need to be identified.
• Modularise functionality in scripts and plugins so that it can be accessed via both the web and the command line.
  • Standardise the command line for plugins by bundling the TWikiShellContrib framework with the core.
    • Plugins can expose modularised functionality through an API
    • TWikiShell ensures it is made available through whatever future means necessary
  • Route all command line scripts to execute through the twikishell framework
    • Standardises command line script initialisation: we can provide consistent start up
    • Aliases to introduce commands directly in the unix/windows shell command namespace can be built for convenience
• ArbitraryTopicNames - real words have dashes and dots, etc. e.g. WebTwoDotZero
• We need PerTopicNotification on TWiki.org: if people have commented on a topic we need to let them know when others reply to them. If others are interested they should be able to put their WikiName into InterestedParties and have TWiki.org notify them of changes. Often what draws people back to a site is an automated email seven months later when someone posts a reply. Without this only regular visitors keep revisiting the site.
• UpdateAttachmentsDontWorkAsExpected: when I say "update" an attachment and I attach a file with a different name I expect to create a new version of the existing attachment, not to add a new completely separate existing attachment.
• TWikiDotPmProposedFurtherModularisation to allow non-cgi apps to use TWikiDotPm
• Build transport-neutral Notifications Framework
• Use standard WorkFlow package for CGI page transitions
• Provide a means for a component to start a background task, and allow notification/workflow to integrate the results with the user base. (SchedulerContrib)
• Tagging or branching in SVN to ensure alpha testers can effortlessly flip to latest beta of TWiki in times that HEAD is unstable.
• Move users out of the Main web
• SplitTheTWikiWeb - having docs mixed with config is insane.
• TopicNameDrivenNewEditTemplate - the name of a topic matching a pattern should be enough to determine the default template used.

Revitalisation of the community

Our goal must be to rally support from the people that make a difference.

• Targeting of PerlMongers groups and TWikiUserGroups to bring in new developers and users.
• We must keep people informed so that they can contribute in line with their interests.
• We must appreciate and cater for what people want
  • Managers who want...
    • Innovation
    • Cross-boundary communication
    • Organisational Coordination
  • Skin designers who want...
    • Powerful frameworks over which to lay their work
    • Leverage of skills and systems they know (and many more know TemplateToolkit than TWiki's templating system)
  • Systems Administrators who want...
    • Quick and painless installs compliant with the conventions of their platforms
    • Fast performance.
    • Ready access to tweaks, and later detection of what tweaks have been made.
    • Easy Upgrade
    • A secure install - secure when downloaded and verifyably so in retrospect
  • Programmers who want...
    • Their work to be used, rated and judged. More prominance of the PluginAppraisalReport
      • "My motivation on my plugin wained after almost no feedback"
    • High quality automated testing
    • Possibly payment so they can sustain their efforts.
  • End users who want...
    • Intuitive tools
    • Interoperability with other things they use (e.g. PPT, VSD, XLS)
  • TWiki development team who want...
    • A means to understand what people want, how many people agree about them
    • A smooth way to help new users contribute
  • New contributors who want...
    • To know what needs doing, how hard it is, and how soon everyone will be able to use their contribution.

Long-term Changing context

• Drive an inspiring TWikiVision

• TWiki embeddable in other packages. e.g. Mason, PostNuke
• CommonFrontEndCgiScript

High-level improvements:

• Interoperability with other packages; PostNuke, Mason etc. (expose the RenderingPipeline, inherit authentication, ... )
• Focus on core competencies / spin and ditch others
• Develop rigorous architecture
• Find more developers and grow development process, e.g.
  • TWikiAdvocacy through PerlMongers, TWikiUserGroups, get users to vote at TWikiAdvocacyAtFreshmeat
  • give instructions for generating patches
• Not be overly afraid to rewrite if we have people willing and capable of helping (Perl 6 & Gallery v2 are complete rewrites)
• Appeal to larger base of non-technical users
  • e.g. Use of the Windows drag and drop metaphor to drop pictures on topics that become attachments + ref to attachment.
• Appeal to larger base of technical users, e.g.
  • PerlMongers user group
  • CPAN authors by providing Plugin examples and advocacy to encourage CPAN authors to bundle a simple TWikiPlugin with their CPAN module to allow Wiki users to call their functionality direct from an authoring process. (Effectively making TWiki for CPAN authors what VB is to windows module authors.)
• Understand and Plug Market Gaps, Exploit Market Opportunities
  • e.g. Why are/did key Wiki projects like Wikipedia not using TWiki?
    • what would it take them to change?

The acceptance of Wiki

• Awareness by the public
• Awareness through management
• Awareness at the CXO level of organisations as authors who are experts on collaboration articulate the value proposition of the approach
• Office interoperability, paste from MS Word, Excel, Visio.

AJAX

A set of standards that set the grounds for new levels of interactivity and usability in web applications. Drag and drop, background communications for web applications. These are normal in AJAX. TWiki should seek to adopt a set of standards as sooner or later innovative plugin authors will use AJAX: when they do we want one AJAX infrastructure layer not a hodgepot of do-as-you please components.

Web 2.0

The way we interact using the web is changing: the Web 2.0 initiative seeks to set a radical vision of society and applications and to pull the world towards it. I imagine WorkFlow is a core component of Web 2.0.

Commercial Competition

As awareness and acceptance rise, we can expect mainstream providers to offer wiki functionality. What will compell organisations that use TWiki that they would be better off continuing to use TWiki?

What appeals to would-be contributors about TWiki? Functionality? The fact that it is pure open source? This would work against KWiki. If it is solely functionality, how do we keep ahead?

For JotSpot ...

The maturing of Perl

How do we leverage all the other open source software out there? How do we avoid duplication, and the waste and incompatibilty that arises? How can the Perl community convince management types that they have a strategy that offers a coherent approach across multiple products? For example, common functions of every web application include: installation procedures, session management, single sign-on, password management, graphic templates and log file management.

Currently applications typically address this individually meaning duplication with the attitude "we can write our own, we don't need no stinking _shared component_".

Organisations seek to minimise the specialist skills training and manpower. The fewer technologies in use, and the more commonality, the easier it is to scale without incurring complexity.

Over the past 5 years there has been a multitude of other enterprise scale Perl software. Organisations will need both applicaions like TWiki and applications like RT. If every application has its own approach to the common functions (above), then the organisation suffers through the burden of each new application. Given the choice, managers will seek those applications that minimise the impact.

CgiApplicationFramework is an upcoming standard ApplicationServer. Based on CGI Application it seeks to provide the hard and boring stuff that is not central to any given application.

• CPAN

TWiki everywhere

Laptop, PDA, Server. With synchronisation and peer to peer updates. SyncContrib is only the start.

The Road Ahead

• Competition from convergence:
  • TWiki on laptops competes with productivity software,
  • Competition from other open source projects - but what do we need to do to collaborate?
• Resource availability: what happens to Perl will affect what happens to TWiki
• Conceptual tools: concept mapping, topic mapping
• Social Network Analysis tools

Goals for EdinburghRelease

The use of a Standard Architecture Framework

Cede context of Sessions, Login, Registration, Cookies, Page flow, to CgiApplicationFramework: New applications are increasingly taking advantage of this framework, it offers several advantages:

• Legitimacy: it builds on the widely popular CGI::Application and is endorsed by CPAN gurus
• Skills:
• Interoperability:
• Functionality

KWiki / SocialText uses HTML::Mason

A consolidated funding model

1. We all want improvements
2. Few people can pay for a whole improvement
3. Most could pay a little towards an improvement
4. There are many of us

So, let's face it, most of us could scrape together some cash towards features we want. Either personally or through the client/firm we are working with using TWiki. Countless unpaid hours go in to TWiki's development. Only those with a professional contracting background typically know how to put in the sales cycles and can be bothered to do so. We need to find a way to pledge money towards improvements, such that when there is enough money to start developing someone interested in GettingPaidToDevelopTWiki can do so without having to sacrifice doing paid work. Hell, some of us might get our weekends back!

Initially this would be for point features. e.g. what's in the RegisterCgiScriptRewrite took me a couple of weeks full-time. People have offered small amounts for me to add more, like, ApprovingRegistrations, but to do this on what's been offered would mean a significant loss in income.

This would be a responsibility of a TWikiFoundation.

Decisions I disagree with but have given up on, for now

The following are issues I differ on but that I simply can't be bothered to argue about. I reserve the right to change my mind and I will probably support you if you agree with me...!

• Use of a standard templating system such as TemplateToolkit

Some things I want:

---

1) High availability, scalability (Support load-balanced reads and writes) This may have to wait until you have finished modulizing the backingstor, to support multiwrite, and database clusters.. (I can go on and on about this. Let me know if you ever want to brainstorm. 2) Major performance enhancements. You guys have done great, but we need better response times. (I have looked at the performance cookbooks, etc.) I have some ideas to improve performance, but I don't have time write now to contribute. 3) Better formal testing of contributions across a number of platforms. 4) A CLI client, that let's me use the wiki, and use my EDITOR to edit the markup. 5) A way to generate a small version of a TWiki that I can push to my PDA/smartphone. This may require both reformatting the pages, and providing custom clients for the different smartphone OSes (In the very far future, I'd want the ability to add local editing and synchronization, just as you are working on with laptops)

-- BrianGupta - 15 Mar 2007

Doh, didn't realize the comment box used markup.

Please feel free to move these comments to where they would be most useful.

• One more, consider distributing a twikitree with statically compiled binaries for various platforms. This will become more and more important, as you integrated with Java, C, C++, Ruby, Python. The versoining of all the various components will become nightmarishly complex, if we don't standardize. (e.g. We would include a JVM and maybe even an Java appserver) GUI configuration scripts for setting up database tables.. and so on.

-- BrianGupta - 15 Mar 2007

• One more thing, speed up the performance of TWiki.org. It is not the showcase of TWiki that you want to show, and turns off many I push towards TWiki.
• Glossify the homepage. (Simplify it and turn it into an advocacy page. Maybe even consider making it static through the publishplugin).

-- BrianGupta - 15 Mar 2007

I thought of another idea that you will love. Write a small interactive shell script that interacts with CPAN module, to download TWiki::site. I say shell, so that you can specify which Perl distribution to use, and what directory to store the downloaded TWiki tree. (If run as root, one of the questions would be what user to install as). Basically as much of the things that you can from configure. This of course would follow all dependencies, and include all the files you need. An apache.conf.twiki would also be generated (not just a generic example). The script would also create any needed directories. (blah blah doesn't exist. Would you like us to create it?)

• I also think advanced options should be added to configure. (Stuff like auto-attach, and what not that are disabled by default and pretty baffling to find out about without looking at source)

The complete install procedure.

1. Download and run shell script.
2. Cut and paste apache.conf.twiki directives into apache.conf. (With minor edits to meet your needs if you have a preexisting apache running.
3. Open your new TWiki site in a browser.

-- BrianGupta - 15 Mar 2007

I like the idea of a sync feature metioned above a lot!

-- MartinSeibert - 27 Jan 2008