Question

My Company is (finally) planning to upgrade from TWiki 4.04+ on Perl 5.6 to TWiki 4.2 on Perl 5.8.5

Does anyone have any recommendations to share regarding testing such an upgrade?

Environment

-- VickiBrown - 13 Aug 2008

Answer

Why not use TWiki 4.2.0 with ActivePerl 5.8.8 build 822 and Apache 2.2.9 on Windows Server 2003? I do and it works a treat especially with the mod_auth_sspi Apache module for Domain auth. These are the latest versions of Perl 5.8 and Apache 2.2.x. Actually TWiki 4.2.2 is out now but I haven't tested it.

-- JamesGMoore - 14 Aug 2008

Another alternative: I just installed TWiki 4.2.2 with XAMPP on a Windows Server 2003 using the latest Active Perl (5.10). Works fine, too.

-- SebastianKlus - 14 Aug 2008

For each plugin you have installed, have a look on plugin page and check they are 4.2.2 aware.

-- OlivierThompson - 14 Aug 2008

Kenneth's excellent guideline UpgradingTWiki04x00PatchReleases is a start on what needs to be taken care of when upgrading. But I have found that as soon as a TWiki has accumulated some applcations, or simply some thousands of topics, the simple approach of "upgrade and fix any breaks which may happen" will cause too much disruption, It is almost impossible to roll back without losing topic changes if you decide that an upgrade won't work right now. Providing a really smooth experience for your not-so-tech-savvy users might need more work.

For testing an upgrade I usually run both versions side-by-side for some time (weeks to months). The effort to do that will depend on the complexity of your setup and on how much customisation you have made (e.g. whether you are using your own corporate skin), and it requires some knowledge about web server configuration. Basically, the process goes as follows:

1. Backup your TWiki, including web server config (but I guess that was obvious anyway)
2. Copy your existing TWiki installation to a new location, but omit the data and pub directories.
3. Install the new TWiki version unto the copy.
4. Configure the new TWiki to run from that new file locations, and configure your web server to serve it under another port (e.g. 8422 for TWiki 4.2.2), keeping all URL paths identical.
   • If your production TWiki runs under a persistent interpreter like mod_perl, you need to disable this for the new TWiki. But hey, this is only for testing, you should not encounter performance issues.
   • Take care to direct the web server's log files to a separate location. This will make troubleshooting a lot less cumbersome.

At this point you should have a working TWiki under another port, with your intended settings for user management and other options, but only with the distribution topics available, and maybe with no users registered (if you have been using TWikiUserMapping) and missing site preferences. In particular, configure should pick up your copy of LocalSite.cfg instead of creating a new one. Use the Sandbox to check which of your plugins or skins need an update. Manual comparisons can be done easily by just adding ":8422" to the location line of your browser.

• If you use a custom skin, you need to adapt all your templates because in 4.2 the handling of line feeds has changed. This turned out to be really time consuming for my own 4.2 tests.

When these obvious things are sorted out, start the next steps:

1. Make symlinks so that your can share Webs (subdirectories of data and pub ) between the old and new versions, again starting with Sandbox.
2. Check what happens if you alternate between "old" and "new" TWikis for editing topics. This is not necessary for a one way upgrade, but needed for step four below.
3. If that works, make symlinks for other webs.
   • The Main (or "Users") Web will need some attention, depending on your user management and TWikiPreferences settings. I usually copy the Main web, because new user registrations are rare, but I might want to use different settings in the new version.
   • Leave the TWiki web alone in the new installation so that in every version, users have accurate documentation. Check carefully what changes you've made to any topics in the TWiki web.
   • Again, UpgradingTWiki04x00PatchReleases has valuable hints which topics ought to be manually reviewed.
4. Find some "power users" and kindly ask them to use the new version. Ask them to report any anomalies. Testing many topics and many user behaviors is too much of a burden for one single person if a TWiki has been running for some years.
5. If everything is ok, or if too many users want to use the new version because of the enhanced functionality, move the new TWiki to the normal port and abandon the old version. This involves some copying files around and perhaps carefully adapting the web server configuration, but can easily be tested. Redirect the "test" port to the productive port for some transition time.

That might look a bit tedious, but has the benefit that you can postpone the upgrade at any point in time, without any hazard to your day-to-day use of TWiki.

-- HaraldJoerg - 14 Aug 2008

I really hope that the developers have some ideas for how they will some day be able to reduce all this mess to a double-click installation that honors customizations and preferences. It seems to me, that anywhere in the world of software, this list of instructions for installing an update would be laughed at. My company has dozens of applications and servers from simple one-hit wonder applications to enterprise database servers, web servers, production-automation servers, and more. Every single one of them can be easily updated and upgraded by double-clicking an installer which, when run, updates the application without altering any data or configurations. I'm willing to do these things because I really like TWiki, but how many people/companies out there would just say, "forget it?"

-- DavidWolfe - 14 Aug 2008

Please don't over-interpret: You can upgrade TWiki with much less effort. But in my past I've seen too many problems resulting from no-way-back upgrades, with many different types of applications. Most of the time it occurs on the shoulders of the users ("Yes, feature X does not work right now, caused by the upgrade. We're working on it"). And not many applications are offering hundreds of plugins, developed by different people. Just have a look at Vicki's impressive list. Oh, and do you have any feedback from someone who succeeded in upgrading Sharepoint 2003 to Sharepoint 2007?

Also note that most of the steps don't take much effort. I have been through this for my own upgrade (4.0.1+patches to 4.2). Back then, we kept the old version because of problems with user mapping and WYSIWYG, but I'll try again for 4.2.2; that is why I have this writeup ready.

Running a pilot phase is something which can never be handled by a double-click installer. But as soon as an application is critical for your business, or your users really depend on it, every change calls for something like an in-house test. Think of things like Wikipedia:Change_Management_(ITSM) or Wikipedia:Release_Management.

Disclaimer: Maybe I am just paranoic. But developing data center infrastructures is the job I get paid for, whereas TWiki is just a hobby-horse. So please bear with me

-- HaraldJoerg - 15 Aug 2008

I actually did upgrade our TWiki with less effort than the process described above. But it still took way longer than it should have and involved manually poking into every file in every directory, making sure that no customization had occurred before manually copying the files to replace the existing ones. Where there were customizations, making reasonably sure that the customization would not break the new software before copying and pasting the altered text/code into the new version before replacing the old one. Then testing the things that were most likely to break, like basic features, pages rendered using different types of templates, plug-ins, and our own TWikiApps. I know I could have just thrown it over the fence and looked to see what broke. The problem is that I would not have known how to fix it afterward, so it's not an option.

A week after I did the upgrade another came out. I'm in no way complaining about rapid releases of updates or bug fixes. It's just that the process should be a whole lot easier. Thousands of software companies do it. Hundreds of OSS projects do it. TWiki, with its pool of remarkable talent and knowledge should surely be able to do it. I am extremely grateful for Sven's work on installers. Without them, many of us would likely have not experienced TWiki at all. But more work needs to be done on updates.

-- DavidWolfe - 15 Aug 2008

Vicki: I found that from 4.1 to 4.2 there were two main things that "broke": 1) some lists built from SEARCH had to be redone (the Cookbook has the new syntaxes), and 2) some templates had to change, because the default templates (which I had included) had changed (e.g., attachments added at bottom).

More generally, while I'm relatively new to TWiki, I gather upgrades have long been known to be a pain, especially for those of us who like to "tweak" things. It takes me days to re-apply my modifications: if patches contiue to come out every week, I could spend the rest of my entire life doing that :). I can see this upgrade issue having a huge impact on TWiki's competitiveness.

I'm sure we all keep notes of our changes, but I don't know of a way to automatically integrate those into a new release. If TWiki had a change management system for itself and not just the content, those changes could often be automatically merged into a new release (else it throws an exception asking for manual intervention, like how TWiki does it for content!).

An added benefit of this is that we could more easily test changes out on a "dev" system, and then transport them into "production": I have two TWiki instances, but since the changes currently have to be manually applied to each instance, I usually don't bother unless it's something "risky". Then I have to periodically refresh dev from prod, like when I'm testing an upgrade.

-- SeanCMorgan - 15 Aug 2008

James, Sebastian - Heh. :-) know the latest TWiki works great with the latest Perl etc. I've got several up-to-date TWikis running now. It's the moving of all of our data (1000 webs, half a million pages, 8,000 users, and that lengthy list of plugins, all in use) that worries me. We've run into some weird problems lately just updating some of the Plugins individually (problems often due to users misuse of that plugin and the plugin not being bug-for-bug compatible with its earlier version.)

Good news there, we have been updating some of the plugins individually.

Oliver - Duh! Now how did I forget that. Thank you.

Harald, David, Sean - excellent suggestions, excellent comments, thank you. Running both versions side by side is probably undoable in the grand scheme but might be possible for a small number of webs. Interesting things to consider.

-- VickiBrown - 17 Aug 2008

Responding to my last comment :-), I just stumbled on PatternSkinColorSettings (because it was missing from the docs for the current release! I've copied it from an older release, and tweaked it to show previews of the colors). Anyway, I love the idea of how it uses AttachContentPlugin to update a flat file that would otherwise not have TWiki version control. That's what I'm talking about! No need to have a parallel system, or build new hooks into RCS.

Due to security concerns, it can only write to the /pub directory. But to capture my Perl and template hacks, that restriction would have to be lifted (for the admin group only of course).

This wouldn't make upgrades painless, but it should make it easier to identify all the customization points.

-- SeanCMorgan - 21 Aug 2008

Good discussions here! Closing after more than 30 days. Please reopen with more details if needed...

-- PeterThoeny - 06 Nov 2008

The new release page for version 4.2.4 that for Upgrading From TWiki-4.1.x, "The upgrade can be done relatively easily by following the the procedure described on TWikiUpgradeGuide." Yet the TWikiUpgradeGuide provides a long list of vague instructions, with the disclaimer: "The following steps are a rough guide to upgrading only. It is impossible to give detailed instructions..."

Does upgrading from a recent release of the same underlying branch(4.x) really have to be so difficult? I realize that there are many changes between 4.0, 4.1, 4.2, etc., but are they so different as to necessitate making this procedure so daunting?

Of all the comments above, Harald's is the most helpful and provides a bit of a guideline. But even his procedure includes the following: "For testing an upgrade I usually run both versions side-by-side for some time (weeks to months)." Aside from the fact that I simply can't afford to create and monitor parallel installations of a single application for "weeks to months" for the sake of testing an upgrade, it seems a bit on this side of insanity to require such measures for a "minor" software upgrade.

I don't mean to resonate with absolute negativity; if I didn't care at all about the software, I wouldn't bother adding my thoughts and I'd simply move on. But the upgrade process is quite painful, and I see that I'm not alone in this sentiment.

-- JohnDeStefano - 12 Dec 2008

We will soon provide a drop in upgrade package for 4.2.x to 4.2.4.

To upgrade from a 4.0 or 4.1 I recommend these simple steps:

• old server: zip up twiki/data and twiki/pub, excluding twiki/data/TWiki and twiki/pub/TWiki
• new server: install latest TWiki
• new server: unzip old twiki/data and twiki/pub into directory tree of new installation (make sure file ownership and permissions match)
• copy twiki/data/.htpasswd from old to new system (applies only of TWiki internal password management is used)

-- PeterThoeny - 13 Dec 2008

Change status to: AskedQuestions AssignedQuestions AnsweredQuestions ClosedUnanswered