upgrade1Add my vote for this tag create new tag
, view all tags

Feature Request for Upgrade Script

This topic is not a guide to upgrading TWiki. It is an old discussion topic for developers.

At the moment, UpgradeTWiki is a basic, "something is better than nothing" script to help upgrade to Cairo.

It is the first step - a stopgap really - in the path to having an elegant solution for users to upgrade TWikis, as discussed in TWikiUpgradeStrategy.

That being said, if you have a standard installation you should hope that using UpgradeTWiki will give you a working upgraded one with minimal intervention.

Latest version of UpgradeTWiki: 01-Sep-2004 (shipped with TWikiRelease01Sep2004)


  • As far as I'm aware it has not been heavily tested. I've tested as best I can, but more would be good!*
  • If you have used it, successfully or otherwise, could you please note the results in the comments below

Upgrade does two main things:

  1. Merges the changes the user made in TWiki.cfg into the defaults setting, which are now captured in TWikiCfg.pm instead of the file lib/TWiki.cfg.
  2. Merges the changes the user made in their topics using Sven's updateTopics.pl code, which is now in UpdateTopics.pm.

It also:

  • Copies over setlib.cfg
  • Copies over .htaccess, and alerts to changes from the distro default
  • Copies over .htpasswd (if it exists)
  • Provides friendly helpful messages about what it hasn't done that the user might want to consider themselves.

The package is attached.

Feedback welcome. Be warned that TWikiCfg.pm uses symbolic refs, which is generally deprecated, yet in this case rather elegant IMNSHO. I enjoyed coding them anyhow: if they are too unbearable, we'll have to think of a different way! smile

UpgradeTWiki is run in the root of the new TWiki.

You need to have Text/Diff.pm in './lib'*

(or in '.')

Until it is added to the distro, you have to put it there manually yourself.

(Text::Diff in the Distro now - thanks Sven!)

What needs to happen now

Must be done before release (I reckon)

  • Tidy up error handling/reporting
  • Deal with symlinks under ./data
  • Let user specify either TWiki.cfg or setlib.cfg
  • do this and make sure it's OK before doing any work
  • get the permissions right
  • copy over their .htpasswd UpdateTopics does this.
  • unlock/ci merged/copied new files
  • resubstitute back $dataDir etc into new TWiki.cfg
  • Make sure the existing TWiki is all checked in before upgrade
  • There are various additions that need to be made to WebPreferences etc, listed at CairoReleaseUpgradeGuide. Need to make sure that UpdateTopics picks these up, or otherwise allow for them some other way. UpdateTopics does pick them up.
  • Ask the user for a good value for WIKIWEBMASTER and insert that in the new TWikiPreferences somehow. Asked the user to edit this field in TWikiPreferences themselves, as the first test of their working TWiki.
  • The way $OS dependent variables are handled sucks. They end up in a silly place in the new TWiki.cfg. It doesn't have to suck that badly. Yet another day of time is needed for me to fix that.
  • Make sure that the Twiki.cfg info in TWikiCfg.pm matches the latest latest Twiki.cfg that is in the distro

(Nearly there! Any volunteers to review TWikiCfg.pm default info. Probably the easiest way to do that is call WriteTWikiCfg, which spits it all out in the familiar format (instead of reviewing the raw perl hash def)).

Optional, do them later...

  • Think about plugins
    • which mess around with templates, lib, pub at least
  • See about templates, which people may have changed

-- MartinGregory - 05 Aug 2004

Matt's UpgradeTWiki Notes

  1. Unpack twiki distribution.
  2. Unpack upgrade package in same directory.
  3. Change shebang line if not /usr/bin/perl (or start with perl UpgradeTWiki )

Run from the unpack directory.

The target directory is not supposed to exist.

There's a bug with relative paths. E.g. if the target merge dir is /home/me/metoo/upgraded-twiki and the current working twiki is /www/me/metoo/twiki the script fails with "Can't locate ../lib/TWiki.cfg at lib/TWiki/Upgrade/TWikiCfg.pm line 127"

-- MattWilkie - 11 Aug 2004

Just upgraded my TWiki with the script (Gentoo Linux, Apache 2). That worked fine with no major issues but I have some small remarks:

The output does not fit well on 80 character wide xterms.

I included the "setlib.cfg" in the path which was not well accepted by the script.

I also made the mistake that the target directory existed already.

The handling of the *.rej files should be better explained. What are the lines marked with - and + exactly, what are the *.orig files that sometimes appear?

One more thing: I just noticed that the files in the pub directory for one of my webs (the only non default one) did not get copied.

-- HeinrichNirschl - 30 Aug 2004

Just used it, worked really well, very complex linked upgrade.

Reinforcing Heinrich's points;

  • Need to give guidance as to how to resolve conflicts
  • And tell users what the .txt file contains if there's a .rej (i.e. old or new version)
  • And what .rej files contain
  • And use diff -w or diff --strip-trailing-cr - some of my topics were on a FAT disc, and had trailing crs, which seemed to confuse it.
  • And offer to overwrite old topics in TWiki shipped webs (TWiki, Main, Sandbox) if there would otherwise be a .rej
  • Oh, also make sure they appreciate how much disc space is going to be needed

-- CrawfordCurrie - 31 Aug 2004

I agree with Heinrich and Crawford. The upgrade script worked fine for me, too. Better explanation of "no common versions" and rej-files would be nice. I also noted that only standard pub-webs got copied, I had to copy the non-standard ones manually.

-- ChristianKohl - 31 Aug 2004

I hid the attachment to this topic since it has been folded into CairoRelease.

It would be nice if the output was automatically logged somewhere for later perusal (and if it wrapped to current terminal width).

Also would be nice if the script preserved oddball shebang lines in bin/*

  • (and to save others the trouble of searching for it, the one liner to fix it after the upgrade is perl -pi~ -e 's;#!/usr/bin/perl;#!/usr/pkg/bin/perl;' *[a-z] (adpated from WindowsInstallSummary#Edit_the_CGI_Scripts))

in IRC Sven mentioned that for Dakaar it would be nice if the upgrade script scanned for code changes too, not just topic changes as is currently done.

Because I'm lazy and incautious, rather than re-installing plugins and skins I just copied the upgraded twiki overtop of the existing one. So far it seems to work tickety boo.

Awesome job MartinGregory, thanks a whole heap!

-- MattWilkie - 01 Sep 2004

Yesterday I upgraded to Cairo using UpgradeTWiki. Everything went really well, especially considering that this was a beta release!

The only thing that I would change would be to improve the instructions. I was unsure of what to do a couple of times. Things such as, "Do I put the new directories on the same level as the existing TWiki root?" Perhaps adding an example of the "path" to setlib.cfg. Silly me, I gave the path plus the filename.

Again, a wonderful job!

-- RaquelRice - 01 Sep 2004

I really like UpgradeTWiki.pl! But I have some nits:

  1. UpgradeTWiki.pl replaced \f with ^L in TWiki.cfg.
  2. It would be really nice if one could specify the location of the existing TWiki as a command line parameter instead of responding to a prompt.
  3. It would be really nice if all of the output would be captured in an "upgrade.log" file. (A lot of it scrolled off the top of my screen.) Especially since you're asking the user to review the output.
  4. The initial prompt for the path to setlib.cfg was a little confusing; I wasn't sure if I should type the ".../bin/setlib.cfg" oro just ".../bin".

Otherwise, this certainly relieved me of a lot of effort to upgrade. Thank you!

-- BruceDawson - 02 Sep 2004

Thanks for taking the time to provide the feedback.

I'll deal with the issues as soon as I can (of course, y'all can add to the documentation if you want to!)

-- MartinGregory - 02 Sep 2004

In TWikiCfg.pm: 'varname' => 'cmdQuote', 'default' => ' ($OS eq "UNIX") ? "\\"" : "\'" ' - may be:sage: UpgradeTWiki --dakar does the DakarRelease upgrade

'default' => ' ($OS eq "UNIX") ? "\'" : "\\"" ' ?

-- IgorKononov - 04 Sep 2004

Just unpacking this indicates that the zip file contains another zip file by the same name:

unzip -l UpgradeTwiki.zip
Archive:  UpgradeTwiki.zip
  Length     Date   Time    Name
 --------    ----   ----    ----
        0  08-01-04 03:20   lib/
        0  07-17-04 03:18   lib/Text/
    21209  07-13-04 05:15   lib/Text/Diff.pm
        0  08-01-04 03:20   lib/TWiki/
        0  08-11-04 04:40   lib/TWiki/Upgrade/
    37285  08-11-04 03:59   lib/TWiki/Upgrade/TWikiCfg.pm
    16826  08-08-04 03:56   lib/TWiki/Upgrade/UpdateTopics.pm
    14893  07-28-04 03:41   lib/TWiki/Upgrade/UpdateTopicsManualSymlinks.pm
    10283  08-17-04 01:16   UpgradeTwiki
    33337  08-11-04 04:40   UpgradeTwiki.zip
 --------                   -------
   133833                   10 files

It appears there is an old copy included in the zip.

-- MartinCleaver - 04 Sep 2004

There is no need to use the zip anymore, as UpgradeTWiki is now included in Cairo. smile

-- MattWilkie - 05 Sep 2004

This sounds like a packaging error. The package will be needed at the time where MartinGregory creates a new version.

The TWikiUpgradeGuide has this note: "Check first if there is a newer UpgradeTWiki script available, see TWiki:Codev.UpgradeTWiki ". I added a Latest version note near the beginning of this topic. Martin, you can change it to a link once a new version is available, with instructions how to upgrade UpgradeTWiki.

-- PeterThoeny - 05 Sep 2004

Ooops - sorry about the zip in the zip. Bit scary.... no idea how that got there.

I will do as you say Peter, when I get a new version available (currently travelling with work, so won't be this week).

-- MartinGregory - 06 Sep 2004

Am I right in thinking that it just ignores any changes one might have made to the distribution in their installaion? I'm pretty sure I had quite a few changes but I see no .rej files for those. The install went up the screen beyond my scroll buffer so I can't tell exactly what it said - would be useful to have a log.

-- MartinCleaver - 07 Sep 2004

I upgraded to the latest production release using the included upgrade script and it worked very well for me. My only comments are:

  • Better handling of the rejected topics, for example, the TWikiUsers and TWikiAdminGroup topics can be expected to be changed and so should not be rejected.
  • Provide a warning if the $doMapUserToWikiName variable in Twiki.cfg is changed by the upgrade. The variable is set to "1" in the distribution but was set to "0" after the upgrade. It took me a while to work out why the mapping wasn't working.

Thank you very much for providing this tool. It definitely saved a lot of manual work.

-- GrahamMacKenzie - 08 Sep 2004

We upgraded yesterday and all worked well (better than expected wink ). Only thing was that the attachments in pub did not get copied. Great job!

-- UlfJastrow - 09 Sep 2004

Aww nuts. I just (re)discovered that if at any time in the past you deleted the as-distributed history files ( *.txt,v ) the topic upgrade part of the script doesn't work (the code is upgraded though). frown

-- MattWilkie - 10 Sep 2004

Ok, I am having no luck with the upgrade script. First, I am running into what look like two different instructions.

This statement: UpgradeTWiki is run in the root of the new TWiki.


This statement: To perform the upgrade, you need to:

  • Check first if there is a newer UpgradeTWiki script available, see TWiki:Codev.UpgradeTWiki
  • Create a new directory for your new installation: Let's call this distro/
  • Put the distribution zip file in distro/
  • Unzip it
  • Choose a directory for the new installation. I will call this new_twiki. This directory must not already exist.
  • Change directory to distro/ and run: ./UpgradeTWiki

tell me to do two different things. The first statement appears to tell me that the distro location IS the final resting place and not a temporary location. The second text appears to be telling me to create a temporary distibution directory and tell the script to do the install to another location.

In addition. the following doesn't make sense given the above: This fullpath should be either the full directory name of the bin directory (where the script can find setlib.cfg) or the full directory name of the lib directory (where the TWiki.cfg file can be found).

Or maybe I am just reading this stuff too hard. Anyway, UpgradeTWiki just bombs on me with messages like: "Not enough arguements for mkdir at lib/TWiki/Upgrade/UpdateTopics.pm etc.

-- SteveRJones - 22 Oct 2004

working nicely as I type this. However, I did stumble at the very beginning when it asked me to Please tell me a path to either the setlib.cfg (preferred) or the TWiki.cfg... I gave it the path to setlib.cfg - the FULL path (i.e. /WWW/../cgi-bin/cfcl/twiki/bin/setlib.cfg). It wasn't happy. It did tell me why it wasn't happy and I apologized and gave it what it wanted. But I think it could either a) be a bit more clear that it wants the patyh to the directory that contains either setlib.cfg or TWiki.cfg -or- b) it could be smart enough to do a basename on what the user types/pastes and Do The Right Thing if the final path elemtn is one of the two desired files.

-- VickiBrown - 23 Oct 2004

Oh. something else I just noticed. Some of the lines of text in the output from UpgradeTWiki are longer than 80 characters. Tsk tsk.

-- VickiBrown - 23 Oct 2004

Steve - what are you reading? My copy didn't say any of that... Did you get yours as part of the 01Sept2004 release??

-- VickiBrown - 23 Oct 2004

Steve - the wording of the first statement you quoted is not great.

It means that it's run in the root of the new distribution.

If you read it this way, you will see it matches the more detailed instructions.

-- MartinGregory - 23 Oct 2004

First, to answer VickiBrown:

  • I am running the version of UpgradeTWiki that came with the 01Sep2004 distro.

To answer MartinGregory:

  • I am running out of the new distribution directory created by tar xvf. The problem is in the statement provided in bold above. How can full_path be a pointer to the location for my new installation AND the full_path to the directory containing setlib.cfg or TWiki.cfg? Is there a requirement that the distro directory be a part of the new_twiki directory structure?

Thanks for the feedback.

-- SteveRJones - 25 Oct 2004

Ok, I've discovered something important - this script will not run right under Perl 5.8.0. Pointing the script to a Perl 5.6.1 installation got me the first set of dialog questions.

-- SteveRJones - 25 Oct 2004

First, to answer VickiBrown:

  • I am running the version of UpgradeTWiki that came with the 01Sep2004 distro.

To answer MartinGregory:

  • I am running out of the new distribution directory created by tar xvf. The problem is in the statement provided in bold above. How can full_path be a pointer to the location for my new installation AND the full_path to the directory containing setlib.cfg or TWiki.cfg? Is there a requirement that the distro directory be a part of the new_twiki directory structure?

Thanks for the feedback.

-- SteveRJones - 25 Oct 2004

UpgradeTWiki appears to now be running right. Will report the outcome later.

-- SteveRJones - 25 Oct 2004

UpgradeTWiki ran perfectly fine. Kudos to the team. Is it possible to add a check for the Perl version and spit out a message if teh version isn't the right one?

-- SteveRJones - 27 Oct 2004

Yack - I thought Perl tried to be backward compatible.

Adding a check entirely possible ... it goes on the list, though we also need to find out why it doesn't work on 5.8 (if indeed that turns out to be the root cause).

-- MartinGregory - 28 Oct 2004

I couldn't definitively say 5.8 was the root cause, or some missing Perl module in our default distro. We did a special 5.6.1 build for the Twiki 01Feb2003 version because the developers are always liking the latest a greatest Perl - and such behaviors don't normally translate into a highly stabile Twiki smile

The behavior that I got was as I described above - complaints regarding Not enough arguements for mkdir at lib/TWiki/Upgrade/UpdateTopics.pm

Does this help?

-- SteveRJones - 01 Nov 2004

I have successfully used UpgradeTWiki in a first-try test. I have one issue with the end results of the test that, not being a full time web admin, I am baffled with. Let me describe what I did and where I ended up.

  • Server is Fedora Core 2 Linux, Apache 2.0.50
  • TWiki 01Feb2004 is installed at /home/httpd/twiki
  • Ran UpgradeTWiki according to instructions on this page, installing the new TWiki to /home/httpd/new_twiki.
  • Took care of most of the merge and .REJ issues.
  • Test the new version by doing the following commands:
    • mv /home/httpd/twiki /home/httpd/org_twiki
    • mv /home/httpd/new_twiki /home/httpd/twiki

At that point, the new TWiki appears to be working, with the new skin and all. However, all attempts to login (using the /home/httpd/twiki/bin/.htaccess (this is the new one now)) fail. The user cannot login.

Reversing the mv commands gets me back to the original install so I have a safe fallback. But, I must be missing some knowledge about how Apache looks at the .htaccess file or something because the upgraded TWiki will not authenticate a user. What am I missing?

-- AlanDayley - 03 Nov 2004

Alan, if at all possible, join us at TWikiIRC, where we can bombard you with questions until the problem is resolved smile If you can't say so here, and we'll try to help you out (but it will be slower by far.

initial Q's

  • in the system, where is the .htpasswd file
  • can you show us the .htaccess file

-- SvenDowideit - 03 Nov 2004

The issue is solved and I am red faced.

In the process of resolving the left over merge issues after the upgrade, I changed the new .htaccess file to point to the new_twiki path. I did this thinking that I would run both the old and new at the same time. After an interruption of a day, I thought doing the mv rename of the directories would be a faster way to test than changing Apache configuration files, forgetting the path change I made. Well, low and behold, Apache could not find the needed files because the new_twiki path didn't exist after the mv operations. Doh!

The whole thing was my fault. I discovered the blunder with a diff of the original and new .htaccess files. Correcting the path in the new file solved the problem. Thanks for the support.

-- AlanDayley - 04 Nov 2004

I had a similar problem to SteveRJones above: The Update script bombed out immediately complaining about Not enough arguements for mkdir at lib/TWiki/Upgrade/UpdateTopics.pm . I fixed it by changing every mkdir call I could find from mkdir $whatever to mkdir($whatever, 0775). After that, things kicked off and appear to be working.

I'm running Perl 5.005_03 (minimum requirement; very old) on a FreeBSD 4.8 server with Apache 1.3, for reference.

-- RichMills - 07 Nov 2004

I am having a strange problem with the UpgradeTWiki script. It looks as thought all the .rej files are being merged into the .rej file of the first topic that could not be merged. I have dug into the code and have not found anything obvious.

Any ideas would be helpful.

tks louie

-- LouieSherwin - 22 Nov 2004

I have solved my problem with patch. I am running on Solaris 8 and the default patch program does not understand the patch file format that the update script builds. The fix was to change to gnu patch.

Another improvement was to add "-b" to the rcsdiff command. This significantly reduced the number of differences and the number of rejects from patch.

-- LouieSherwin - 02 Dec 2004

Needs to be upgraded (!) for DakarRelease, I guess.

-- CrawfordCurrie - 13 Jan 2005

Have now done an ImproveTestenv in SVN to check for GNU patch, as well as egrep, fgrep and diff. All you TWikiOnSolaris users, please try SVNget:bin/testenv (DevelopBranch but will work fine on mainline TWikis including all older ones, I hope).

I have also put a comment in on the RlsCmd test for TWikis post 01 Sep 2004 saying it's no longer needed - need to be able to check TWikiVersion string, so it's important it is simple and fixed format...

-- RichardDonkin - 14 Jan 2005

This topic was used for CairoRelease and should not be converted to the new form. Please create a new topic to keep track of followup items, e.g. UpgradeTWikiScriptForDakar or the like.

-- PeterThoeny - 18 Jan 2005

I was having difficulty with the upgrade script until I realised I'd not put directories in places it expected to find them. My twiki was in /usr/local/www/twiki, what I'd done was to move it out of the way to twiki-old, then run the update script. Just a heads-up to somebody else who might be surprised by this - leave the old twiki in place. smile

-- MikePatterson - 22 Jan 2005

I have been trying to upgrade my TWiki site, and the upgrade scripts terminates with

Merging your existing twiki data (/var/www/html/twiki/data) with new release twiki data...

Usage: UpdateTopics [<DestinationDataDir]> -bash-2.05b#

I attempted to run the UpdateTopics.pm script from the command line as root, but found that the script was not marked as executable (it was 644). Changing it to 755 let me execute it, but I then got a number of errors which I think are related to missing packages? (see below)

-bash-2.05b# ls -alF UpdateTopics.pm -rw-r--r-- 1 root root 16826 Jan 25 11:26 UpdateTopics.pm -bash-2.05b# chmod 755 UpdateTopics.pm -bash-2.05b# ls -alF UpdateTopics.pm -rwxr-xr-x 1 root root 16826 Jan 25 11:26 UpdateTopics.pm* -bash-2.05b# ./UpdateTopics.pm ./UpdateTopics.pm: line 12: package: command not found ./UpdateTopics.pm: line 14: use: command not found ./UpdateTopics.pm: line 16: use: command not found ./UpdateTopics.pm: line 17: use: command not found ./UpdateTopics.pm: line 18: use: command not found ./UpdateTopics.pm: line 19: use: command not found ./UpdateTopics.pm: line 23: syntax error near unexpected token `(' ./UpdateTopics.pm: line 23: `use vars qw($CurrentDataDir $NewReleaseDataDir $DestinationDataDir $BaseDir $debug @DefaultWebTopics %LinkedDirPathsInWiki $RcsLogFile $TempDir);'

Any clue as to what I'm doing wrong here? Any help would be appreciated


-- FrankPaynter - 25 Jan 2005

Frank - you should be running the UpgradeTWiki script that is in the root of the twiki distribution - the UpdateTopics.pm is only one package that is used as a part of the upgrade.

-- SvenDowideit - 26 Jan 2005

Just upgraded to cairo, and here are some comments on the UpgradeTWiki script: * The instructions say:

    # Create a new directory for your new installation: Let's call this distro/

To me, this first statement is quite unclear. It appears to say that I am creating a directory where my new installation will live. I think it would be clearer to say something like:

    # Create a new directory as a staging area for your new installation: Let's call this distro/. This must NOT be where you will be installing.

  • The instructions also doesn't indicate anywhere that the attachments won't be copied over to the new installation!

  • The debug option should be a command line switch, not buried in the lib/TWiki/Upgrade/UpdateTopics.pm file.

  • The output of UpgradeTWiki should also go to a file. I guess you could tee it, but sometimes that doesn't work really well with programs that require an interactive response.

  • Finally, there is way too little information about the .rej files, as well as the display that goes into creating them. There should be some explanation, maybe:
    • UpgradeTWiki will look at every topic in the current installation. If that topic is also in the new release, UpgradeTWiki will compare the rcs versions in each place. If the topic hasn't been changed in the current installation, UpgradeTWiki will use the new version. If the topic has changed, then UpgradeTWiki will attempt to patch the changes from the last version that matched to the latest version in the current installation into the new release's version. If the patch fails, it will generate a .rej file.
      • The patches are in the file data/patchTopics in the new installation.
      • The patch log file is in data/patch.log in the new installation.
      • UpgradeTWiki prints a letter for each rcs version it looks at. (At least, I think that's what it is doing. The script says "For each file that has no changes...", but then it shows
          .../twiki20040904/data/Sandbox/WebNotify.txt: p
        This is very confusing!)
      • Remember that the .rej files are the result of trying to patch the new release with information from the current release, so the "-" lines are things being removed from the new release, and the "+" lines are things from the current release. (This "feels" backwards, at least to me, so that's why I would add a line stressing it).

  • I wrote this little script to figure out why things were rejected, and to make the fixes manually. It doesn't have any error checking or anything, because I don't really know if it is the "right" method. But it worked for me. For each .rej file, I did a cd into the new installation's data/web directory and ran this:
      web=$(basename `pwd`)
      rcs -u -M $1
      gvim -f -d ../../../twiki/data/$web/$1 $1
      rcs -l $1
      ci -u -mUpdateTopics $1
      sudo -u nobody rcs -l $1
    The two rcs lines, as well as the ci line, came from the UpgradeTWiki library. I took them on faith. gvim is a wonderful implementation of vi: The -d option does a side-by-side visual diff, with differences highlighted in color. One manual step that was required was to change the first line (the META:TOPICINFO one) version="1.xx" to version="1.xx+1" (e.g. 1.17 -> 1.18). This probably wasn't the most elegant solution, but it worked. Of course, the script assumes a unix-y system.

  • To copy pub files without overwriting anything from the new release, cd into the old pub directory, and do this (at least on unix-y systems like linux):
      tar -cf - .|(cd NEWRELEASE && tar -kpxvf -)

-- DougClaar - 03 Jan 2006

Instead of using tee to capture the output from the upgrade, use script [name of logfile].

-- SteenManniche - 29 Mar 2006

UpgradeTWiki.pl feedback, 2005-Feb-08, DEVELOP r3607


* the "Here's what is about to happen" section says "4) I'm going to tell you what you need to do next!* and then doesn't.

* The generated bin/LocalLib.cfg is empty

* There is no lib/LocalSite.cfg generated.

* if I specify path to twiki/lib/ instead of setlib.cfg, followed by path twiki/bin the script bombs with:

Preparing to write new format configuration files for you...

Use of uninitialized value in concatenation (.) or string at lib/TWiki/Upgrade/TWikiCfg.pm line 87, <STDIN> line 2.
Undefined subroutine &TWikiCfg::printToNewConfig called at lib/TWiki/Upgrade/TWikiCfg.pm line 87, <STDIN> line 2.

Comments on Usability

* why can't I use a relative path? e.g. ./UpgradeTWiki ../upgraded ? fixed as of r3613

* The second sentence of:

   - The target directory does not have to be web-accessible.
   That means that there should be all the normal TWiki directories 
   here: bin, lib, pub, templates etc. right here.
appears out of context. I had to read it several times before I realised "right here" meant the directory I was running upgrade from and not the target dir. And in the place it appears there is nothing one can do about it anyway. fixed as of r3613

* the 'where is setlib.cfg' prompt doesn't like it if you tell it exactly where it is: /path/to/bin/setlib.cfg as opposed to /path/to/bin/

* allthetextrunstogethermakingithardtoread. A couple of newlines or rules (------) between each new decision prompt would help. fixed as of r3613 - well, I added some "press return" checkpoints that should help.

-- MattWilkie - 08 Feb 2005

Fixed the bugs, I think. Usability someone else can look at.

-- CrawfordCurrie - 09 Feb 2005

Thanks Crawford, it finishes without error now. There is a warning at the beginning:

Name "TWikiCfg::dataDir" used only once: possible typo at ./UpgradeTwiki line 119.
fixed as of r3613

-- MattWilkie - 09 Feb 2005

As mentioned by RichMills mkdir in UpgradeTWiki requires two arguments when older Perls are used. OTOH, the TWikiSystemRequirements explicitly allow Perl 5.005. I think a note pointing out the release requirements for the upgrade script (> 5.005) could be helpful in preventing disappointment. fixed as of r3613 by adding the second argument

-- GeorgBauhaus - 09 Feb 2005

Tell users early (in the docs, in the script's messages) that they will be able to reuse the old version's directory? The instructions say "use a new directory, without mentioning that it can be renamed later. This isn't for the administrators comfort. wink

Reasons why you want this include (I'm sure you know because the script says so, but only when it is finished):

  • existing web server configurations referring to the old directory
  • backup scripts
  • external links
  • ...
fixed as of r3613 -- GeorgBauhaus - 09 Feb 2005

DEVELOPr3612 - completes successfully and without warnings or error, however there is no data directory to be found.

Another usability note: The script never says "Okay, I'm finished now." So I never know if the script actually thinks it's finished or if it just crapped out without an error. guess this was to do with UpgradeTopics; seems to be fixed as of r3613

-- MattWilkie - 11 Feb 2005

Great job on the fixes Crawford! Even the usability ones you said you were leaving for someone else. wink


-- MattWilkie - 11 Feb 2005

DEVLOPr3617: UpgradeTWiki says it doesn't put anything in the existing twiki install, but that is not entirely true. I found an RcsLogFile in the Main and _default data dirs.

-- MattWilkie - 12 Feb 2005

I had the 'not enough arguments in mkdir' problem on Perl 5.005_03. However, I don't have SVN set up so it's not easy to download UpgradeTWiki, and even then it would be nice to have a separate ZIP to use the latest UT on a standard 02-Sep-2004 release.

I think the top of this page should point to a ZIP file so people can grab the latest version quickly without SVN.

However, this is a really great tool, thanks for writing it - works well on Red Hat Linux 6 (!) with the attached patches. Note that I also commented out the mkdir of the target directory (in first patch) because I wanted to pre-create a symbolic link to a directory with enough space.

My upgrade to 01 Feb 2003 took about 6-8 hours, while the upgrade from there to 02-Sep-2004 has taken just 1 hour so far and will probably be complete in just another couple of hours' tweaking.

  • Perl5.005-patches.tar.gz: Quick patches for Perl 5.005 hosts - check that 0777 permissions are OK for you before using this.

Finally - would be good to have the output of the tool logged to a file (I'm sure Perl can make this elegant somehow) - had to use tee update.log which means typing blind.

-- RichardDonkin - 14 Apr 2005

  • I agree to the above.
  • At my site, the script failed to copy the attachments from pub too. Had to do that manually.
  • The script didn't keep the original owner settings. That should be otimized.
    • can you tell me what did not come across? the script is supposed to bring across TWiki.cfg settings and also keep your changes to distributed Topics
  • Last but not least: Despite the minor problems the script saved a lot of work. You are on the right track!
    • Thanks, good news! smile

-- DetlefMarxsen - 20 Apr 2005


Having finally upgraded my home TWikiSite to CairoRelease, I can echo the comments of others above:

  • The upgrade script is certainly better than no script at all. Thankyou.
  • Still, I think we're all searching for EasierUpgrades
  • This recent upgrade (including the re-install of old PluginPackages and addition of new plugins that require Cairo) took approx. 6 hours.
    • Thanks also to experience from having already performed the similar upgrade on my TWikiSite at work.

Here is a laundry list of the problems I encountered

  • .rej files, as expected
  • UpgradeTWikiDoesNotCopyAllPubFiles
    • In fact, I could consider this a FEATURE! I did have sufficient disk space avail. to duplicate the complete TWiki footprint.
    • However, others may appreciate not being required to duplicate the entire /pub

  • TWikiDocGraphics needed a labor-intensive re-work to reconcile my non-standard docGraphics into the new table formatted list.
    • In hindsight, I would have been better off to have added those non-standard doc graphics to a separate page (had I known)
    • which furthers the argument for DataAndCodeSeparation

  • Similarly, TWikiPreferences was also one of the .rej files, yet even following the .rej was not sufficient for putting the topic back together. I basically had to rebuild the TWikiPreferences by hand.

  • Unix file permissions were more loose after the upgrade than before.

  • My manual redirects from /twiki/bin/index.html and /twiki/lib/index.html MOVED TO... /twiki/bin/view were not copied over into the new twiki root.
    • so had I not looked for this, then the security measure I'd originally taken to prevent web browser directory listing of my /twiki/bin and /twiki/lib dirs would have been lost on the upgrade.

-- KeithHelfrich - 03 Jul 2005

I just tried doing the upgrade, and it failed for reasons I totally don't understand. The upgrade script starts all happy, and then dies with the message:

Usage: UpdateTopics [] Maybe a permissions problem? But I don't know where/why.

-- SiddharthBondre - 30 Apr 2005

Status of work in DevelopBranch

I have changed the script to take parameters on the command line, which means that in the long run it will not be as chatty.

so to fully upgrade your CairoRelease based TWiki to Dakar (creates your LocalSite.cfg and upgrades the topics and pub dir), you can call

  1. ./UpgradeTWiki ../Existingtwiki/bin/setlib.cfg ../UpgradedTWiki

where the UpgradedTWiki dir does not exist yet,


  • reduce the verbosity of the output
  • Log progress to a file

-- SvenDowideit - 18 Sept 2005

Changes to the upgrade process are being tracked in Bugs web, so I'm closing this.

Upgrade of Dakar (Jul 2005 build 5950) to (Sat, 24 Sep 2005 build 6567)

This used the UpgradeTwiki of r6567

  • The /dakar directory has r6567
  • The /oldwiki directory has a working "live" dakar beta at r5950

The first attempt ran:

$ cd dakar
$ ./UpgradeTwiki --upgradeTopics ./bin/setlib.cfg ./data ../oldwiki/data/ ../newwiki/
$ ./UpgradeTwiki --upgradeTopics ./bin/setlib.cfg ./pub ../oldwiki/pub/ ../newwiki/

This ran with some rejects. However the target /newwiki had:

  • all the top level files
  • no /data directory
    • /twiki, /main, /sandbox were all at the top level
  • no /pub directory

    • when calling UpgradeTwiki --upgradeTopics, the output dir is where the combination of the new and old directories goes, so you must call it as you have below (its an advanced feature, but obviously under documented frown ) -- SvenDowideit

So I tried again:

$ ./UpgradeTwiki --upgradeTopics ./bin/setlib.cfg ./data ../oldwiki/data/ ../newwiki/data
$ ./UpgradeTwiki --upgradeTopics ./bin/setlib.cfg ./pub ../oldwiki/pub/ ../newwiki/pub
The first part of this created all the top level files under /newwiki/data but still no /data there.

    • I don't understand but still no /data there - what was the result? -- SvenDowideit
    • Yes, correct. still no /data/ there -- AntonAylward

The second part created a completely separate /newwiki/pub

Yes, I manually created the /newwiki/data and the /newwiki/pub and moved things there but my confidence has been eroded.

Anyway, I did a directory diff between dakar/data/TWiki, oldwiki/data/TWiki and newwiki/data/Twiki. The result puzzled me.

There were, as I expected, many differences betwen dakar/data/TWiki and oldwiki/data/TWiki, but UpgradeTwiki had just copied all the .txt files from oldwiki/data/TWiki to newwiki/data/TWiki.

ALL. I'd expected them to be marked as different or possibly rejected because they couldn't be merged, but no, they were simply copied.

I really can't see the utility of UpgradeTwiki in this context.

    • the trouble i had when doing this, is choosing what was more important, the newly distributed version, or the user modified version - I choose the user's version whenever i wasn't sure. you should however find Topic.rej files in the newData dir that tell you about the failures. It does however sounds like something's very wrong frown -- SvenDowideit

-- AntonAylward - 24 Sep 2005

I suspect that upgrading usertopics from bulletfields to formfields could be trivial. TWiki::User.getXField() will read either type of user home topic, so upgradeTWiki iterate through each home, pick up the set of fields and build a replacement topic using the NewUserTemplate.

-- MartinCleaver - 07 Oct 2005

I ran the UpgradeTWiki script with the --help argument. It said that to upgrade to Dakar I should pass it arguments like this

usage: UpgradeTWiki --dakar oldSetlibCfgFile oldCfgFile outputDirectory does the DakarRelease upgrade

I was upgrading from a Cairo release to a Dakar release. However I was unable to upgrade following the usage information. Here was the command I ran and here is the output.

./UpgradeTwiki --dakar /home/crd/u1/www/DSD/twiki/bin/setlib.cfg /home/crd/u1/www/DSD/twiki/lib/TWiki.cfg /tmp/twiki_new

Loaded TWiki version: Sun, 06 Nov 2005 build 7330 ERROR: old TWiki's setlib.cfg file not found at (/home/crd/u1/www/DSD/twiki_upgrade/setlib.cfg)

I was able to upgrade by using the Upgrade script like it is being used on the web. Which I did like this

./UpgradeTwiki /home/crd/u1/www/DSD/twiki/bin/setlib.cfg /tmp/twiki_new

Which just points to the old setlib.cfg and to a destination directory. I believe the usage information should be updated.

-- MattRodriguez - 14 Nov 2005

Upgrade of Cairo (Sept 2003) to Dakar (Feb 2006)

A Problem After Cairo to Dakar Upgrade

Just ran the UpgradeTwiki script to convert from an older Cairo install. I didn't see any significant errors reported, but some files that seem absolutely essential in the new Dakar version did not get created. I've been adding them manually by copying similar files (guessing in the dark) and have had limited success at repairing things. My Main web transferred fine; problems are with TWiki and Sandbox so far.

Here are the missing files:

  • twiki/pub/TWiki/WebHome/_filetypes.txt
  • twiki/pub/Sandbox/WebHome [directory missing]
  • twiki/pub/Sandbox/WebHome/_filetypes.txt

Also, this one doesn't seem fatal, but my web server error logs apparently show some script looking for a file called "favicon.ico" that doens't exist. It's being looked for in the web server root directory.

-- MarkBearden - 02 Feb 2006

Fatal Issue with Cairo to Dakar Upgrade

I started the Upgrade using:

# ./UpgradeTwiki  ../bin/setlib.cfg ../dakartest

I as prompted regarding some differences to the new .htaccess, etc.

Then the assumed parameters for the upgrade of the data-files where mentioned to me.

But the upgrade thereafter breaks with:

---- which: no patch in (//sbin://bin:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/usr/X11R6/bin:/bin:/usr/games:/opt/gnome/bin) Uh-oh - couldn't see a patch executable on your path! I really need one of those!

Does anyone know why and/or what I have to do against this problem?

Thank you!

-- DirkSchlossmacher - 03 Feb 2006

You need to download and install patch - GNU Patch is recommended, see Google:GNU+Patch or HowToApplyPatch.

-- RichardDonkin - 03 Feb 2006

Thank you for the fast reponse & the good hint! Works now...

-- DirkSchlossmacher - 03 Feb 2006

Have just run Upgrade Script to upgrade Cairo TWiki to Dakar. Script ran fine with no errors. However my TOC's are not rendering correctly. Lots of overlapping arrows.

-- AndrewReeves - 07 Feb 2006

I had simillar problem and the solution was to copy original installation files twiki/pub/TWiki/PatternSkin/style.css and twiki/pub/TWiki/PatternSkin/layout.css over old Cairo version.

-- DaliborSvoboda - 07 Feb 2006

Thanks Dalibor. That has fixed it. Should this be changed in the upgrade script?

-- AndrewReeves - 08 Feb 2006

Can't locate C:\twiki\bin/C:\twiki\binC:/twiki/lib/TWiki.cfg in @INC

Updating Cairo to Dakar on WinXP..I thought I followed directions but... I changed the #! and added a .pl and ran: C:\TWiki-4.0.1>UpgradeTwiki.pl C:\twiki\bin C:\twiki-4_0_1

Now generating new LocalLib.cfg from settings in old setlib.cfg...

C:\twiki\binC:/twiki/lib is a Cairo TWiki
Can't locate C:\twiki\bin/C:\twiki\binC:/twiki/lib/TWiki.cfg in @INC (@INC conta
ins: C:\twiki\binC:/twiki/lib C:/twiki/lib C:/TWiki-4.0.1/lib/CPAN/lib//arch/ C:
/TWiki-4.0.1/lib/CPAN/lib//5.8.7/MSWin32-x86-multi-thread/ C:/TWiki-4.0.1/lib/CP
AN/lib//5.8.7/ C:/TWiki-4.0.1/lib/CPAN/lib// C:/TWiki-4.0.1/lib C:\TWiki-4.0.1\b
in C:/Perl/lib C:/Perl/site/lib .) at C:/TWiki-4.0.1/lib/TWiki/Upgrade/Cairo2Dak
arCfg.pm line 26, <STDIN> line 1.
-- AnnBrady - 10 Feb 2006

Problem with login-related variables

I just upgraded from a Sept 2, 2004 release to 4.0.1 on my site (http://rabbit-island.net). Probably the largest problem I'm currently facing is that all login-related variables aren't resolving. I can point a link directly to /bin/logon and get the proper login prompts (and successfully login), but the variables related to logging in (login, logout, etc.) don't resolve (e.g., see the variables page or RabbitIsland:ClassicSkinLogin).

I've got the configuration set to use TWiki::Client::TemplateLogin and TWiki::Users::HtPasswdUser, though I was able to successfully log in before setting that (away from the "none" default).

Any thoughts?

-- MarcPerkins - 12 Feb 2006

It looks like it might be related to my use of DragonSkin; switching temporarily to PatternSkin seemed to resolve the problem. Should I just switch skins, or is this problem easily fixed?

-- MarcPerkins - 12 Feb 2006

Switching to NatSkin has solved the login problem (and given me a much sleeker and more customizable skin).

-- MarcPerkins - 12 Feb 2006

Email notification does not work - SOLVED

After upgrade from Cairo mail notification doesn't work anymore.

Here is a command and returned message when I try to use tools/mailnotify

perl -wI /srv/www/twiki/bin/ tools/mailnotify
Undefined subroutine &TWiki::basicInitialize called at /srv/www/twiki/lib/TWiki/Contrib/Mailer.pm line 68.
When I tried MailerContrib:

Installation ended without any error and I got the same result as with mailnotify:

# ./bin/mailnotifier
Undefined subroutine &TWiki::basicInitialize called at /srv/www/twiki/lib/TWiki/Contrib/Mailer.pm line 68.
Could someone please give me a hint what could be wrong?

twiki/bin/configure gives neither warning nor error and all other plugins works well.

host:/srv/www/twiki # find . -exec grep -H basicInitialize {} \;
./bin/remove_obsolete_locks:    TWiki::basicInitialize();
./lib/TWiki/Contrib/Mailer.pm:    TWiki::basicInitialize();=
Find returns no basicInitialize definition, does this mean that some files are missing?

Thanks in advance!

-- DaliborSvoboda - 16 Feb 2006

The following works for me on 4.0.1:

cd /srv/www/twiki/bin/ ; ../tools/mailnotify -q

-- HeinrichNirschl - 16 Feb 2006

Thanks for your help, but my problem was, that I installed MailerContrib over 4.0.1 and therefore I got version mismatch.

This issue is solved for me here: EmailNotificationNotWorkingInDakar.

-- DaliborSvoboda - 17 Feb 2006

Don't move existing twiki before upgrade

Just a note that you must not move your existing twiki install from it's original location until after running UpgradeTwiki. This may seem obvious, but I thought I would be clever and move things around before the upgrade so that my new Dakar install would be visible immediately via the web server. Anyway, if you do this, lib/TWiki/Upgrade/TWikiCfg.pm gets the $twikiLibPath wrong and fails with a message:

/www/twiki/lib is a Dakar TWiki
'/www/twiki/lib/LocalSite.cfg' and '/www/twiki/lib/LocalSite.cfg' are identical (not copied) at /data/www/dakar/lib/TWiki/Upgrade/TWikiCfg.pm line 98

Might be worth reinforcing this in TWikiUpgradeGuide, or even better, checking that the $twikiLibPath in the old setlib.cfg actually matches the location of setlib.cfg.

-- MartinRothbaum - 18 Feb 2006

SafeEnvPath is created with variables

After upgrading, there were loads of errors relating to the perl taint checking, telling me there were unsafe directories in the path (see Support.Cantcreatevfile)

The problem was found by the gentle inhabitants of #twiki on IRC. In /twiki/lib/LocalSite.cfg , there was the following:

$cfg{ScriptUrlPath} = '/bin';
$cfg{SafeEnvPath} = "$cfg{ScriptUrlPath};/usr/$cfg{ScriptUrlPath}";
in stead of
$cfg{SafeEnvPath} = "/bin;/usr/bin";

This might be useful to others as well.

-- JosMaccabiani - 12 Mar 2006

Problem with missing permissions for TWikiRegistrationAgent to the Main web

I've upgraded from TWiki02 to Twiki03, then from TWiki03 to TWiki04.

User registration has stopped working - I've found out that I've had Set ALLOWWEBCHANGE = TWikiAdminGroup in WebPreferences of the Main web, and I needed to change it to Set ALLOWWEBCHANGE = TWikiAdminGroup, TWikiRegistrationAgent.

See UserRegistrationProblem for discussion.

I think upgrade script should take care of this.

-- AleksanderAdamowski - 23 May 2006

Topic attachments
I Attachment History Action Size Date Who Comment
Unknown file formatgz Perl5.005-patches.tar.gz r1 manage 0.8 K 2005-04-14 - 13:51 RichardDonkin Quick patches for Perl 5.005 hosts - check that 0777 perms OK for you!
Edit | Attach | Watch | Print version | History: r103 < r102 < r101 < r100 < r99 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r103 - 2006-05-23 - AleksanderAdamowski
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2015 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.