URGENT!

We really, really, really need to resolve where the masters of the documentation for DakarRelease reside.

PeterThoeny is - I guess - working on the documentation at TWiki.org. This documentation is probably out of step with the DEVELOP documentation.

SvenDowideit wants to use the SVN docs as the masters. This is obviously very sensible, as it allows us to keep the release documentation live on twiki.org without confusing people. But I have reservations, due to there not being any of the ,v files that are need for UpgradeTWiki to work.

I need to update more docs, and I don't want to duplicate/conflict with Peter's changes.

PETER, SVEN, PLEASE RESOLVE THIS. URGENTLY.

-- CrawfordCurrie - 13 Feb 2005

The master of the Dakar docs is on TWiki.org as has been documented / was agreed on in DeveloperResponsibilities when we started with the new development model. Please review that page!

I am the primary tech writer since MikeMannix's departure; it is in my interest to have high quality TWiki docs. I understand the wish from the developer's point of view to maintain the master docs in Subversion for ease of use. However I do not agree at this time for these reasons:

• The docs get updated frequently by people who do not have svn write access
• I maintain two doc sets for quality control:
  • The TWiki web on TWiki.org as the "work master" where anyone can contribute. It is at the state of the MainBranch.
  • A Beta TWiki's TWiki web as the "release master". We have a diff utility to see what changed. This allows me to easily compare and merge topics from the "work master" to the "release master" for a Beta or Production release.

At the time the DevelopBranch gets merged into the MainBranch we can update the master docs on TWiki.org.

We can change the doc model once we have the capability to update TWiki.org's TWiki web docs via a Subversion backend. Until this is in place we should stick with the current doc model.

-- PeterThoeny - 14 Feb 2005

OK, thanks. For the record, the policy on docs changes (storing deltas in the change request topic) is IMHO unworkable, for the following reasons:

1. There may be / will be many branches, and handling this manually is just not workable.
2. Developers can't test the finished docs
3. It loads a huge amount of effort on you late in the development cycle (merging the changes).

However, what we can do it to write a script to delta the docs in SVN against the docs on TWiki.org, and generate a bunch of patch files. That solves problem 1 and 2, but still leaves you with a lot of effort late in the development cycle, resolving the differences.

-- CrawfordCurrie - 14 Feb 2005

This is kind of obvious but I think it's worth stating:

• The documentation in TWiki.org's TWiki web is the documentation for how to use TWiki on this site, and therefore must always be up to date with the current code that this site is running on.

Whatever else you decide in this topic I think this statement must hold true.

-- SamHasler - 14 Feb 2005

Sam, I agree, but historically it has always been the documentation for the next version. This has tripped me up more than once!

-- CrawfordCurrie - 14 Feb 2005

OK, I tried to solve this by diffing the docs we have in SVN against the docs on twiki.org, but the docs on twiki.org don't match anything in the release, so that won't work. The only option I have is to diff the docs in Cairo against the docs in SVN and provide Peter with a patch file for all the changes, which he can then merge with the changes in twiki.org. Is this really the way you want to proceed, Peter? It would be a lot easier to work with docs in SVN hint hint

-- CrawfordCurrie - 20 Feb 2005

I've been struggling to get my head around the best way to maintain the documentation. I've had some experience with docs in SVN now, and it is crystallising my thinking. I'd like any other constructive thoughts on this issue.

1. I really want the ease of use of TWiki for editing documentation, but I'm finding it is absolutely critical to be able to branch docs - otherwise the risk of confusing people (especially testers) is way too high.
2. Controlling docs in SVN makes them easy to process during release generation. Some docs in Dakar are already generated directly from POD blocks in the code, making them much much easier to maintain.
3. I've realised that docs in SVN have to have %META:TOPICINFO normalised. There are a number of reasons for this:
   1. Because TWiki isn't used to handle the docs in SVN, the topicinfo gets out of step
   2. The topic info contains a last author ID, which results in a "bogus user" warning being generated in an installation as TWiki tries to track down the user. The user should be normalised to a standard user, such as "TWikiGuest" (this does not discredit documentation authors; their contributions are still fully recorded in SVN and in the AUTHORS file)
4. UpgradeTWiki and BuildContrib now both check in documentation during installs/upgrades. Revision histores are local to an installation, which is the way it should be.
5. Last an probably least, I can use my favourite editor (emacs) when writing documentation.

At the moment the only version of the documentation for Dakar is in SVN. As described above I have tried to diff the docs on TWiki.org so changes can be merged out to SVN, but the doc versions I can see (in the TWiki web) are clearly not the versions that found their way into the Cairo release, so I am stuck.

-- CrawfordCurrie - 26 Mar 2005

After some experimentation and a minor code change, I discovered that $Rev$ works rather nicely in the revision number field. That lets us keep track of the revision of the docs actually used in a release. I changed all the checked-in docs to use it. I also normalised all the author fields to "TWikiContributor" and cleared any signatures. This gives a fairer reflection of the co-authored nature of the docs, and is more wiki.

While I was there I moved TWikiContributor to Main to create a new user, and made it INCLUDE an attachment called AUTHORS. This attachment is generated by build-twiki-kernel by copying the AUTHORS file from the distribution root.

-- CrawfordCurrie - 27 Mar 2005

changed attribution for attachments to TWikiContributor in r3920

-- WillNorris - 27 Mar 2005

Until and when the code to separate out the TWiki web into documentation and configuration, perhaps it would be useful to embark on categorising those topics.

-- AntonAylward - 28 Mar 2005

It sounds like Crawford and Will are creating a workable doc management system using SVN, but I thought I'd share a brainstorm about how to design a TWikiApplication for managing doc development and posting (almost) entirely within TWiki. Not really seeing how the emerging doc system in DEVELOP works, I don't know if some part of these idea might fit into that system. So here's a series of propsoals that aren't necessarily all dependent on each other:

1. Create a Web somewhere purely for documentation development.
2. Break up documentation into modular chunks. In the absence of MultipleCustomNamedIncludes, break up any topic where sections branch for different versions. Create a WebForm for tracking which version a particular topic applies to.
3. Create master documents for major sections that include and order the subsections, using parameterized includes for pulling in the appropriate versions.
4. When we are ready for a release, use one of generate html or pdf addons to create a static copy of the appropriate documentation version.
5. Distribute the static copy of documentation, appropriate to that release, in a lower profile way - i.e. not as a separate web but simply as a set of html pages within a Doc directory that's linked from a help page.

-- LynnwoodBrown - 24 Apr 2005

Actually, I'd object to point 3: including subsections in a master document is a no-no. They should only be used to add extra structure to the web.

-- MichaelDaum - 25 Apr 2005

the static docs could contain hooks to add comments on particular sections, which would be stored within the local wiki. This way if an admin installs a security update, for example, which changes behaviour from the as shipped state the changes can be documented in context.

-- MattWilkie - 25 Apr 2005

on IRC Will, Crawford and I discussed how to deal with this issue for now. We decided that the BuildDistro script will take the topics and attachements in the Cairo release zip, and add to them the version of the topic that is in the DevelopBranch. These files will then be bundled up into the DevelopBranchReleaseCandidates.

From that point its up to everyone to test those release candidates, and to submit bug reports and fixes. (i'll post more on this later)

-- SvenDowideit - 30 Apr 2005

The following is Crawford's understanding of an AIM conversation with Peter on the subject of documentation in the DakarRelease. Peter will correct anything that is inaccurate or a misrepresentation.

The current issue with documentation is that we have no way of determining the baseline of any TWiki document. Documents are supposed to be developed on TWiki.org, so it is reasonable to assume that a specific version of each document on TWiki.org is the baseline for a release. However, the real release baseline is maintained in a private web ("Beta web") that is not readable outside the core team. Even if it was accessible, the only way to determine which versions of the topics in TWiki.org it is based on is via timestamps, as the transfer from TWiki.org into Beta web has been a purely manual process.

The build scripts on DevelopBranch use the versions of the documentation in SVN, because:

1. a release package has to be built from a branch, because different branches will have different documentation (since they will have different features),
2. Portions of the documentation are automatically generated by the release scripts (code docs, TWikiContributor)
3. Documentation has to be post-processed to determine the differences from the last milestone (i.e. the differences since CairoRelease). The documentation is also processed through a "documentation cleaner" script,
4. Documentation is a key part of testing.

Peter has the following issues with this process:

1. he does not like the removal of signatures from doc topics
2. he feels that it locks out non-developers from doc work

The documentation cleaner script removes spurious text - including signatures - and normalises meta-data in the documentation topics. Signatures are removed on the basis that they:

1. make the documentation look amateurish,
2. the acknowledgements are already made in AUTHORS and TWikiContributor (which is derived from AUTHORS) and
3. they are divisive and inaccurate, as many major contributions are not acknowledged this way.
4. Documentation topics are not really TWiki topics - they should be treated as read-only in an installation, so making them look like TWiki topics is wrong.

It is true that non-developers cannot check in to SVN, and are probably better off contributing on TWiki.org anyway. The question is, how can their changes be reflected into SVN?

Crawford's proposal is that the flow goes like this:

1. developers check docs into DEVELOP
2. the public change docs on twiki.org
3. Peter manually merges twiki.org to Beta
4. A script merges Beta to DEVELOP, based on the Cairo baseline
5. MAIN is merged out to DEVELOP, conflicts are resolved and testing is done
6. DEVELOP is merged (copied) into MAIN
7. DakarRelease is built from MAIN
   • the release script generates a ,v file that reflects changes since Cairo
   • MAIN is tagged

Note that step 4 could be dispensed with if the merge in step 3 is from TWiki.org direct to MAIN. However the merge is arguably easier if both source and target are TWiki topics.

Other inputs are very welcome.

-- CrawfordCurrie - 19 May 2005

sounds to me like it would be significantly quicker and more manageable if all released files were in svn. That way the only merging needed would be to bring in changes done by non-developers into the relevant svn branch.

personally, i'd like to dispense with the need for signatures altogether, even in thread mode. this (I think) could be acheived through clever rendering of changes, with annotations showing who made the changes. (and that plugin that ** wrote to show contributors to a page)

then the work flow would look like

1. developers check docs into DEVELOP as they write the code
2. public change docs on twiki.org (which i suggest should be the docs for the currently released version)
3. developers merge changes from twiki.org into DEVELOP
4. developers proof read and fix docco (we should have an annotation plugin for this)
5. DEVELOP is merged (copied) into MAIN
6. MAIN is released
   • the release script generates a ,v file that reflects changes since Cairo
   • MAIN is tagged.

-- SvenDowideit - 19 May 2005

OR, there could be a separate public web for the current development version.

At the end, it's inevitable that we have 2 sets of documentation: One for the "current release" and other for the "development version". It's natural to have them on different webs.

-- RafaelAlvarez - 19 May 2005

I agree with that - and I was/am intending on finishing the SvnStore implementation to make it possible for that we to be in the appropriate SvnBranch

I also am starting to think we should have one Codev web per release, that way you can find out what the reasons for a feature were for that particualar release, and still totally refactor old discussions.

this would still allow all released files to come form the versioning repository (which to me is a requirement of a repeatable automated build system)

-- SvenDowideit - 20 May 2005

I've been thinking some about how to deal with the docs and would like to float some proposals based on the following suppositions:

• It a valuable part of TWiki culture to list the actual names of contributors to documentation. This is a way to acknowledge even modest contribution of members of the community to documentation.
• References to contributors' name should not point to the local Main web since those names will not be part of the distribution.
• Documentation should be, in the distribution, "read only."

So I propose the following convention for composing the official documentation topics:

• List all contributors at the end of the topic.
• Use the interwiki format on contributors' name to reference back to TWiki.org (e.g. TWiki:Main.PeterThoeney).
• Before packaging the distribution, restrict editing of all documentation topics to TWikiAdminGroup. Perhaps this could be done using a script.

-- LynnwoodBrown - 22 Jun 2005

Regarding using of separate Codev webs for each release, this does not seem necessary to me since, at some point, the documentation discussion has to be consolidated in transferred into the final documentation topics (where-ever that resides). The documenation discussion regarding multiple releases can go on within the same web, as long as we note what version we are discussing.

We should, however, maintain separate "official" documenation webs (i.e. TWiki web) for each release. Prior to release of Dakar, we can do a merge recent additions from Cairo docs that are still relevant to Dakar into the Dakar docs.

-- LynnwoodBrown - 22 Jun 2005

Lynnwood, in DEVELOP signatures are removed from docs. If someone wants to be recognised as a doc author - and many people don't - then they should be added to the AUTHORS file and properly recognised as a TWikiContributor. This should be part of the process of recognising contributions. You know your contribution is recognised and valued when your name gets listed in there, far more so than if you sign something, and then no-one bothers even to remove the signature when they use your work to generate professional standard documentation.

Raf's proposal for documentation webs would allow us to maintain stable, online doc sets for previous releases, which is a good thing, while still providing a forum for development of the next generation. Of course it would have to be one web per branch, not just one per release. How do you propose merging between branches?

-- CrawfordCurrie - 23 Jun 2005

i personally think that there is no choice in the matter. now that we have more than one branch, we have to face the fact that we need to have branched doccumentation too.

the most sensible thing would be to have all of the releaseable files in the same configuration management system - ie subversion.

this is what is happening for Develop today, as it neatly addresses the branch issue, the release issue (and plugins will soon-ish be moving to Subversion too) and the fact that users have been confused but the docco on twiki.org not matching the latest release.

please remember, most twiki users (not just admins) come to twiki.org to find out docco for a released version of twiki, not of the currently under development one.

merging docco between branches then would be an svn operation.., and getting authors would be a meta-data operation.

-- SvenDowideit - 23 Jun 2005

We have executed on docs being in SVN. The docs on TWiki.org are way out of date, and don't even share a common ancestor with Cairo docs, so it's extremely difficult to merge in changes there. However there are scripts to show the diffs; it's the best we can do.

-- CrawfordCurrie - 28 Aug 2005

Update: there is now a basic SVN store interface in Dakar that lets you work on TWiki docs in an SVN checkout area using a TWiki installation. However it does not support commit (commits have to be done from the command-line). Nor does it support repRev or delRev.

-- CrawfordCurrie - 31 Aug 2005

See DakarDocumentationModelIsBroken on immediate actions we need to take for DakarRelease.

-- PeterThoeny - 10 Oct 2005