create new tag
, view all tags

Easy upgrade of running TWikis

As time goes, people run

  • more and more TWiki systems on many servers
  • with more and more customizations

And thus upgrading to a new version becomes more and more involved. (see for instance UpgradingProcedures). What can we do to remedy the situation, and tend to the goal of having just to unzip the upgrade in place and run?

possible points:

  1. Remove all site-related info from TWiki. Everything should go to Main (Webtable) Upgrading would thus replace TWiki with the new standard one. Plugins notably should not add/modify anything in TWiki.
  2. Provide 2 packages:
    • full distrib, as now
    • upgrade: for instance it will not include WebPreferences topics, but patches to add the new functionalities to existing ones.
  3. For templates, provide a list of what is mandatory to change in your templates if they have been heavily modified from the standard and you cannot use blindly the new ones (provide the list of features to implement in your templates)



-- ColasNahaboo - 23 Feb 2002

Sounds wonderful!!

-- RandyKramer - 24 Feb 2002

Option: Provide tools/makefile to assist

I think that TWiki could improve a lot the installation/upgrade procedure by:

  • Using a Makefile or something like that to install. This way, one could perform a lot of operations automatically, e.g. specifying at the top of the Makefile the web user and web group the Makefile could change permissions and such for the administrator.
  • Separating "permanent" data and first time installation data (that is, sample webs, etc.). That way, one could issue make install to install the permanent data and make installdata to install (only the first time) the sample webs and all of that.
  • Modifying TWiki to search for things in several directories. For example, it would be very very nice if TWiki looked into, say, /usr/local/share/twiki/templates and into /var/www/twiki/templates, so one could keep separated every customization from the "base installation". That would really make a difference for the installation and upgrade enhancements.

-- EstebanManchado - 09 Apr 2003

We did an upgrade some days ago, and we spent all the evening at it. We definitely have to ease upgrades. We already have lots of ideas to ease them, and are willing to put effort in it, if the core developers are open to the suggestions and changes (because, of course, most, if not all, of the changes will serve nothing if not incorporated in the TWiki core). Peter, anyone, please, comments?

-- EstebanManchado - 14 Jun 2003

Issue: Dealing with Patched Servers

To put some numbers to this: I have 87 patches against my Beijing-based server. I have a similar number against my athens based one. And I suspect that I am in the minority in keeping an organised list of the patches applied.

Unless I need to upgrade those servers (i.e. there are some compelling features worth having), picking through what functionality I might lose by upgrading to Cairo then I just won't bother. I bet there are very large numbers of pre-BeijingRelease installations out there that people are just not bothering to upgrade. (Is there a status form installed with TWiki that just shows a summary of useful statistics (version, plugins installed, number of users, names of webs, number of topics in each web, etc). This would be useful.)

We need commitment that Easier Upgrades is going to be addressed.

-- MartinCleaver - 15 Jun 2003

Issue: Local customisations (code & settings) versus defaults (code & settings)

I think we have a conflict here between the nifty-ness of having settings and customisations that layer and over-ride each other (from Site-wide to twiki-wide to web-wide to page-wide to user settings from the home page.) then on top of this we have code based layers of core twiki code, plugins, addins and patches. To simplify upgrades we need to remove the redundant portions.

I think that the upgrade i did a while ago from a pre-BeijingRelease to Beijing was made easier by encouraging the features that i needed to be included in the core smile , so the biggest problem i had was with changes in settings. I ended up copying my older versoin of the TWiki and Main webs into the newer ones and then did diffs to see what new setting i cared about. this is just a bit scary and error prone. (and what about user preferences?)

to this end i am thinking that we should seperate the twiki into Data, Docco, Code, and Settings/configuration.

  1. docco (in Main/Twiki webs)
  2. settings / configuration (Settings Web)
  3. data (other Webs)
  4. code (i would like something like CVSWebClient or SandWeb so that i can edit/configure the entire twiki from my browser )

I still would like to advocate using twiki pages for settings (ideed more so than currently). the SettingsWeb would be laid out to allow for the flexibility that we have at the moment


also if we have nested webs we get something like..

then when upgrading you should be able to merge the SettingsWeb and resolve the merge conflicts..

see: SeparateCustomPreferences, BetterDefaults, DataAndCodeSeparation

-- SvenDowideit - 15 Jun 2003

Regarding Dealing with Patched Servers above, what patches do you have applied Martin? Making the list of the patches you've applied publically visible (say attach to your home page would be a good place to starts? (After all if no-one knows they're clobbering people's installs, people's installs will get clobbered smile )

I would suggest after maybe a couple more comments this topic gets refactored to make the key issues clear. (Esp given clear overlap with several other topics)

-- MS - 16 Jun 2003


I have created a TWikiTopicUpgradeScript that will use the rcs versioning info to try to upgrade your TwikiTopics from an older release.

(topics content moved to there)

-- SvenDowideit - 12 Jan 2004

In future to more easily work out which revision was the release revision I see three alternavites:

  1. Revise every topic with an author named after the release. I don't know if this would reduce clarity or improve it when viewing the topics.
  2. Don't distribute version ",v" files thereby setting all topics back to revision 1.1 for an installation. (Is there really any benefit to having these revisions on an installation?)
  3. If revisions were tagged to denote the type of revision (major, minor, spelling etc) as has been discussed elsewhere then it would be simplicity itself to create a new revision for every topic with the same author as the previous revision when a release is made and tag it with the name of the release.

-- SamHasler - 07 Jan 2004

Simple: never, ever, distribute WebPreferences files or any site-modifiable file (like TWiki.cfg, etc...)

Distribute some WebPreferences.sample that can be used to populate a first install, and then if it needs changes provide instructions or upgrade scripts to upgrade the existing ones.

-- ColasNahaboo - 07 Jan 2004

actually, i expect my merging to cater for this, without any problems. (time will tell though) - I'm writing the code right now...

-- SvenDowideit - 07 Jan 2004

Killing the history files would work for the first upgrade, but what about the second, third, etc.? All you are doing is delaying the problem for one cycle.

How come the revisions always start with 1.? Maybe we increase the major rev number for each production release. That way it doesn't matter if the site's webpref is at 1.9 because Dakar would be 2.1.

Webpref.sample is a good (better?) idea too. Maybe a combination of the two.

-- MattWilkie - 08 Jan 2004

NO, no, no!!! I'm not ever going to kill the history files, I am using them to merge the differences. and because of this I think that there is no need to use the sample file idea. I am going to merge the user's customisations into the new version of the file. And this needs to work again and again. I want to update my site every time that a beta is released..

Arbitaily upping the major number won't help, Tagging the repository at each release will help in the future, but i think that not relying on this is even easier.

-- SvenDowideit - 08 Jan 2004

Zapping the history files is the simplest method of preventing RCS issues in a new install. The simplest way of dealing with the "TWiki web" issue is to treat it as part of the system distribution and should not be modified - including TWikiPreferences and Plugin pages . Locally installed plugins going into Main or Plugins (preferably %PLUGINSWEB%), and system distributed plugins have their pages in %TWIKIWEB%. That way adding new plugins to the distribution doesn't clobber people's installations. (Your idea above misses this detail smile ) Also this way users aren't ever going to be in a situation whereby they will have to manually resolve CVS style conflicts.

I would link to where I've discussed this elsewhere, but... some of the text might be view as relevant so I've copied the text (and approach I'm working towards) below: (Largely a c&p, but if I've missed a reference I shouldn't have here, please delete it.)

    Proposal: %SECTION{title}% Split package supplied things from locally changed things, and allow local overrides on everything

    %SECTION{synopsis}% This proposal simple on the basic level, and very inline with current installations - indeed it can largely be viewed as an extension of the current setup. It proposes (among other things) the creation of a system web containing default configuration values, along with system documentation.Like wise, it also proposes local config web. Again this would contain configuration data and documentation.

    The system web would be read only with no local configuration, and supplied by in the tar ball. The local web would be supplied at initial installation by the installer, but not for upgrades. The clear benefit of such a directory is that it would allow upgrades to such a system to happen relatively transparently - since we can be certain that a user has not changed the parts we're changing and they can be happy that we're not changing their changes.

    The default names for these webs will be the same as the values taken for:

    • %TWIKIWEB% - becomes %SYSTEM%

    Specific things that users change:

    • Variables
    • TWiki.cfg
    • Plugins
    • Skins (templates)

    In terms of management the setup would be this:

    • Users downloading and installing modules would be downloaded and installed in the LOCALCONFIG/PLUGINS web.
    • System supplied plugins & so on would be installed in the SYSTEM web
    • In order to facilitate this, it is further suggested that system plugins should reside in a separate directory from standard plugins. This allows the user to install their own plugins without clobbering the system ones and vice versa. (Also this potentially allows a fallback mode to occur.)
    • Similarly, many people currently install skins into the default location of "templates". Whilst convenient and simple, this makes maintenance much harder than it needs to be. (Webs can have the skin templates in a separate directory).
      • It is proposed to split the default templates directory into two - specifically into "supplied" and "localised". The supplied versions can be assumed to be writable over, whereas localised are user controlled & dealt with. Clearly tools to separate out the local from the non-local would be useful.
        • Thought: why not have this in the data tree as in the localconfig directory? If skin designers produce code to go with their plugins force a separation of the "skin" into a skin package & a plugin package, and call that an addon? This could massively simplify things overall ? (Also starts a migration path towards a web being a place for editting a skin, rather than logging into the server - since local customisation would be to copy the default and so on)

    This way if it's decided to start include %RANDOMPLUGIN%, then it will by default pick up the config information from the system web unless the user has previously installed the plugin. In that scenario, the users plugins will be used instead.

    What this proposal does overall is separate the supplied default code & configuration information from the user localised code & configuration information.


    • TWikiPreferences is moved to two locations, with the local version taking precedence.
      • This would be useful to generalise so that one web can inherit config settings from any other, or even better specify overrides in another. (Since that allows logical importing of values from others simplifying ClueSkinConfiguration & OpenProjectConfiguration)
    • Skins are no longer installed into templates. (Or better, defaults actually go somewhere else, meaning existing skin tarballs can be used "without problems"... )


    In the absence of any feedback this is what will happen. If you don't like this, please speak up in the comment box!

    Personally I think this is long, long, long overdue. Data & Code sep helps alot with maintenance, this takes things an extra mile. (I wonder how close to the last mile?)

    -- People.MS - 25 Oct 2003

    This is already partly implemented. It does however need work to make more flexible and less of a security hole.

In the comment above I'm referring to the SecureTWikiPreferences issue, and the fact that this allows at least half the problem above to be solved via SeparateTWikiSystemAndSitePreferences.

Personally, I suspect any "patch" based arrangement rather than the strict "no edit supplied stuff" approach, and "no upgrade clobbers local stuff" approach will be worse for users. (Indeed removing the patching element from the installer is one of the ideal aims for the TWiki Unix Installer )

The discussion here is distinctly related to the UserWeb discussions. And again that benefits from an approach of look-but-don't-touch for system supplied stuff.

-- MS - 08 Jan 2004

Ok, as you all seem convinced that what i am writing the code to do won't work, and I'm convinced that it will, I'm going to try to stop reading this topic until i have time to finish the code.

  • MS - my revised strategy does cater for plugins
  • the RCS files are integral to working out what was changed by the user
the basic technique is to loop through all the current topics, find the highest common revision with the new topic (using RCS) then patch the new vsersion with the differences from the old one. for topics created by the user, copy them to the new web. All other cases are related to these ideas

I'm not really interested in the future propsals for separating out distributed topics and user changes as i think this de-wiki's the distributed topics - and I'm writing this for currently existing installations, not for unknown futures. (so there smile )

-- SvenDowideit - 08 Jan 2004

That seems like a good idea. How far can you go within the existing structures and capabilities and separately considering if re-structuring is appropriate to making upgrades easier.

-- JohnTalintyre - 09 Jan 2004

Ok, as you all seem convinced that what i am writing the code to do won't work, and I'm convinced that it will, I'm going to try to stop reading this topic until i have time to finish the code.

Good strategy smile (When you read this smile ) I'll hold back further comments until your reimplementation of CVS patching RCS files version is available. wink

-- MS - 12 Jan 2004

Edit | Attach | Watch | Print version | History: r24 < r23 < r22 < r21 < r20 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r24 - 2005-07-12 - MattOlsen
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2018 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.