Requirements For ATWiki Application Packager

No-one has picked up the ball on a TWikiApplication packager yet, and it's about time we did something.

A TWiki Application packager would be a something that helps you to package up a TWiki Application in such a way that it can be re-used by other folks, or in other places.

The BuildContrib is an effective way of packaging many types of extension; Skins, Contribs, Plugins etc. It can also be used for TWikiApplications, but it falls down in a couple of important ways. Rather than analysing it's shortcomings, it would be more useful to think about the requirements for a more user-oriented, TWiki Application packager. Here are some 10,000m requirements:

Requirements

• "Must" means it must be done
• "Should" means it would be really really nice and a PITA if it wasn't done, but we could survive, just.
• "Could" means it would be really nice, but no-one is going to think we're unreasonable if we don't do it.

Basic requirements

There are several different packaging models that have to be catered for, in order of increasing complexity. Each level of increasing complexity shares all the requirements of the lower levels.

1. Must support uniquely-named topics that overlay a single user web
   • Installation: add topics to the web at install time
   • Can be done with a simple zip/unzip
2. Must support topics with unique names that overlay standard webs (TWiki, Sandbox, Main)
   • Installation: add topics to the web at install time
   • Can be done with a simple zip/unzip, but requires admin rights
3. Should support new web(s), perhaps with subwebs
   • Installation: add new web, maybe again several after first installation
   • Can be done with simple zip/unzip, but requires admin rights
4. Could support topics that overlay existing user webs several times (the same app installed with different names)
   • Installation: must be able to rename topics when installing the app, may need to install several times with different target names

Beyond this there are also Plugins, Contribs, Skins. These are what BuildContrib does.

Additional requirements

1. Must be easy (brainless) to use. Use both packager and installer from the browser
2. Must support packages identified by URL
3. Could support packages uploaded from client
4. Could support incremental upgrades (don't lose revision histories)
5. Should minimise the excise (CPAN and plugins) of the packager
6. Could share infrastructure topics between different instances of the app
7. Should be rule (data) driven (reqts 3, 4)
8. Could automatically resolve external dependencies (e.g. on plugins)

Deployment requirements

1. Should support deployment to a public demo area on TWiki.org
2. Should support upload to a public area (c.f. Plugins web) on TWiki.org

User model

What the user sees

Packager

1. Install the packager (once only; plugin or contrib, probably)
2. Visit the "TWiki.PackageTWikiApplication" page
3. Select topics and webs to package
4. Hit "go". Result is a "save to disc" zip.

Options (reqt 2, 3, 4)

• Add rules for renaming topics and webs to fit in a target installation

Installer

1. Install the packager (once only)
2. Visit the "TWiki.InstallTWikiApplication" page
3. Enter the URL of a package to install
4. Hit "Install"

Options

• Browse local disc for a package to install
• Answer simple questions about where to install (what to rename packaged topics to)

Technical details

What the nerds need to know/discuss

Packager

• One option is to generate a MANIFEST and use BuildContrib to build the package

Installer

• Re-use the BuildContrib installers?

-- Contributors: CrawfordCurrie, LarsEik, SueBlake, ColasNahaboo, KennethLavrsen, SvenDowideit

Discussion

Edited comments on rev11

...Some pages uses plugins, for example TimeTablePlugin, WorkflowPlugin and DirectedGraphPlugin... I would gladly install the "Application packager Contrib" ... I would like a page where I could fill in a form ... where I could ... choose which pages I wanted to include... then on a page ... InstalledPlugins were listed and I could select which plugins ... the MANIFEST could be written ... the DEPENDENCIES would list the plugins ... and also CPAN packages

-- LarsEik - 31 Jan 2008

We could get 90% of the desired result ... by simply putting all of the .txt files into a zip ...

-- SueBlake - 01 Feb 2008

... I fear that creating zips also falls under the "too hard to use" banner.

-- CrawfordCurrie - 02 Feb 2008

... install but also uninstall ... you want to also uninstall the things you pulled in because of dependencies ... If Bob grabs Alice app, modifies it, how does he contribute back his derived version? ... how do we mix development-in-the-browser with desktop-development?

-- ColasNahaboo - 02 Feb 2008

... focus on designing this TWiki application so people can FIND what they are looking for. Using tagging, categories. ... I have never been able to directly use a TWiki application someone else did. I always change it a bit to suit my needs. ...

-- KennethLavrsen - 02 Feb 2008

... this is or should be about making a frontend packager for BuildContrib ... We should target us who are TWiki champions and thus have configure permissions. Other users without permissions ... by asking admin/champion for help.

-- LarsEik - 02 Feb 2008

... dependency management is a horrendously complex thing ... existing tools - apt, yum etc. ... layering over BuildContrib ... requires the BuildContrib installers to be steerable ...

-- CrawfordCurrie - 03 Feb 2008

... more complex ones, apps that contain many many topics. How on earth do you simply find them all? ... No thanks I don't ever want to learn that geekish contrib thing ... if there was a central place I could upload a zipped bundle of txt files ...

-- SueBlake - 03 Feb 2008

... Plugins web is the place. ... like BugsContrib. ... created live on a twiki, then made into a BuildContrib ...

-- SvenDowideit - 03 Feb 2008

I refactored the requirements to take into account the discussion, and heavily edited the discussion to reflect my understanding of who said what. Please refer to older revisions of this topic for the full blow-by-blow.

-- CrawfordCurrie - 04 Feb 2008

Nice process. Maybe an additionnal goodie could be, when one refactor, just leave a marker to the version before the change, e.g: rev11

-- ColasNahaboo - 04 Feb 2008

Comments on latest requirements

We could start the packaging with selecting: a) simple app or b) custom app... Would not that be an approach here? Where simple app meaning "I have this single/few pages of a single web to upload and share, and it doesnt need configure permissions or have dependencies" and a custom app would need configure rights, possible plugin dependencies, manual intervention, multiple web etc.

I mean it is important that end users don't to go installing 20-100 pages of an application they thought would be nice...and thereby messing up a web or two and missing some dependencies at the same time.

By installing the packager plugin there could also be an ApplicationInstallGroup that could limit whether you were allowed to install packages through the packager app. Then superusers be allowed for some expanded possibilities. Otherwise end-users could get by with copying from examples rather than messing up, potentially.

About the zip, adding a pre-created zip could be very nice, but selecting topics from a list/checkboxes, perhaps filtered, would be easier (I believe) rather than manually creating that zip.

So maybe 80-90% would be achieved already if we made that path for chosing "Simple App".

About the MANIFEST and "Custom App" path, I really see this as an build on top of BuildContrib or rebuild if you like. It would be so good that you wouldn't need to shell out to build your own plugin/app/thing with the old BuildContrib, but maybe I'm wrong here, since I actually haven't gotten the build thing going The "Custom App" could make room for easily selecting dependent plugins and also a space to type in other dependenciesand things to remember to do manually. It would be smart...I'm smart, I'm so smart ..s.m.r.t...I mean s.m.a.r.t. (my hero Homer)

-- LarsEik - 04 Feb 2008

I agree, but there is still the problem of mapping (renaming), which the BuildContrib doesn't support. Actually, it does, but not data-driven.

-- CrawfordCurrie - 04 Feb 2008

When we consider putting an app in the Plugins web with an installer, I guess that implies that we're only going to do minimal configuration to use it in the local environment, and we'll expect some easy way to upgrade to newer versions.

I've been thinking of apps as something that I will have to half rewrite myself before using, just someone else's working ideas to get me started, then I have to maintain it locally. For example I've used the TWiki Installations app as a starting point to create apps for all sorts of unrelated tasks, and an installation or upgrade would be irrelevant (though alltxt.zip would have helped initially).

So we're talking apples and oranges -- you're looking more at apps that work out of the box on my TWiki for their original purpose. I find the other kind more interesting and useful, but that doesn't seem to be the topic of discussion here, sorry.

-- SueBlake - 04 Feb 2008

No, you are quite right, Sue, what you describe is well within the remit of this package. In fact, it should be the priority. Upgradability; well, that's the sort of thing the BuildContrib is for, so perhaps we should stop talking about it. I'd be quite happy to drop D. Could support incremental upgrades (don't lose revision histories) - though it is only a "could", not a "must".

-- CrawfordCurrie - 04 Feb 2008

Sue, dont' be blue:) You have more experience than me, I'm relatively new to making apps and getting more out of TWiki. I have this great app (I think, but I'm missing an a). For my app it would not be a problem (maybe..) if I or someone upgraded it on tdo, because it contains only the instructional docs, the forms, flows and logic between them. The data must be filled in for it to have any sensible use. But of course I see that upgrading topics with TWikiForms would have some potential issues, like changing name of field etc. So I can imagine there might be cases where the upgrade like with plugins, must or should be a disabled function.

I fully agree that priority should be to get an easy starting point for using and reusing applications. I think for Sharepoint this is called "Application templates". I saw a presentation of a customer use of the afore mentioned application and I was not impressed compared to TWiki. I'd prefer TWiki a millions times before. But making this application packaging/install a good and useful feature for TWiki is a must to attract new companies and users. Really, manually handling zip archives is not a killer feature for me. But if the packager/installer uses a zip archive those who want can download, edit and install. And I dont think we disagree too much, just reading differently the text here...me is not english.

-- LarsEik - 04 Feb 2008

I think SingleBranchPluginDevelopment will make some of this more obvious - For example, for a TWiki user to create a re-distributeable Extension package, I would envision that BuildContrib would have a TWiki UI that allows a user to select a set (ResultSet??) of topics, and hit a 'make new Extension' button. This would generate the MANIFEST, and bundle them up into a BuildContrib style TWikiContrib. similarly, it could even uload to twiki.org...

Assuming that we work out howto get the BuildContrib / configure infrastructure to drive package and topic upgrading, and dependancies, we may have some very nice tools to support the real senario that Sue talks of.

I see upgrading topics to include giving the TWikiAdmin tools to merge changes into their heavily modified TWikiApps.

For example, the next time I work on BuildContrib, I will be adding query SEARCH support - something that anyone wth a customised BugsContrib would probably love to have merged into their version.

-- SvenDowideit - 05 Feb 2008

TWikiApps don't need a packager of their own as long as you design them to be reusable in multiple webs. Just put them all under Applications.MyNewApp and deploy them using %INCLUDE{"Applications.MyNewApp.RenderProjectOverview"}%. This becomes even more apparent if you developed "middleware applications" that are only used by other applications and thus have no "enduser web" where would be usable. Let's have all TWikiApps under Applications.MyNewApp and package it as a MyNewAppContrib. So where's the point I miss?

-- MichaelDaum - 15 Aug 2008

Applications like BugsContrib cannot be used that way: You end up with a lot of "duplicate" topics that just include the main topic in the app. The solution I found with XpTrackerPlugin is to distribute a template web, so to install the app you just to create a new web.

-- RafaelAlvarez - 15 Aug 2008

Having to INCLUDE app topics to use them is way up there on the geek scale for me. And besides, not everything works as expected with INCLUDEs. Apps that depend on ChecklistPlugin, for example, will work in some cases, but not in others.

-- DavidWolfe - 15 Aug 2008

Well, I'd consider it a design flaw, if the application is not reusable in one TWiki in multiple webs without code redundance. BugsContrib could easily be rewritten using the layout as described. A web template is good and fine, but sometimes you want to mix several applications in one web. Infact, that's why I developed the TWikiWorkbench.

-- MichaelDaum - 15 Aug 2008

I agree with Micha. BugsContrib's fixed web annoys me too - and I wrote the damned thing. I was avoiding re-jigging it until we defined at least some best practices for it, or (preffereably) improved the App installation system to make things relocatable.

-- SvenDowideit - 16 Aug 2008