create new tag
, view all tags

Feature Proposals » TWiki to run under the current perl version


Current State: Developer: Reason: Date: Concerns By: Bug Tracking: Proposed For:
UnderInvestigation HaraldJoerg None 2017-04-01     KampalaRelease

Edit Form

DateOfCommitment:   Format: YYYY-MM-DD


This is a recurring action, and right now the topic is "Perl 5.24.1". While it is generally a good idea to check that TWiki works with up-to-date Perl, it gets somewhat urgent when this Perl version is found in the wild.

Description and Documentation

The most notable difference in Perl 5.24 is that CGI.pm is no longer part of the Perl core. It is, of course, available as a package from CPAN and from Debian (libcgi-pm-perl), but both of them come in versions which are not compatible with TWiki. As we have learned, CgiContrib is not a complete solution to the problem.


Right now, Strawberry Perl is available with Perl 5.24, and within 2017 I'd expect the release of Debian Stretch, which ships with Perl 5.24.


Without some action, TWiki can't be installed on Debian Stretch and will not work with current Strawberry Perl.

WhatDoesItAffect: Install


I'd hope that the effort is rather small. One known issue is the early invocation of CGI::Carp in the TWikiScripts which is not caught properly by CgiContrib.

The plan:

  • Run TWiki from SVN using Perl 5.24 and the PSGI Engine to check for other pitfalls. First results:
    • Basic viewing and editing just works, because PSGI avoids the CGI::Carp problem.
    • configure breaks with Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/}{ <-- HERE / at lib/TWiki/Configure/UI.pm line 100. (and perhaps more to come). This is "only" a warning, but in configure, every warning raises a fatal error (which is a good thing).
  • Run the test suite under 5.24 (CPAN:App::perlbrew to the rescue)
    • Doing so, it turns out that there are plenty of this unescaped left braces in TWiki regexes. Unfortunately the TWiki syntax %FOO{ is often examined by a regex /\%FOO{.../ which should in Perl 5.24 be written as /\%FOO\{.../. The escaped version works in any Perl version back to 5.8, so we're safe to change it though it is tedious. -- Harald Jörg - 2017-03-25
    • The count for the unescaped left brace issue is about a dozen in core, and about 200 in plugins. Every request causes several warnings to the web server error log, for persistent interpreters there's just one warning per module. Fortunately, the fix can be applied per program: Write a test script, parse the resulting warnings which are pretty precise about files and lines, and fix these lines. Patch available (2455 lines). -- haj 2017-04-01
  • Finally, of course, fix the CGI::Carp issue.
    • Two news on that item: First, the fix is a small one, albeit a bit ugly. It is not sufficient to simply move TWiki's lib magic. We must also avoid to use the module within the BEGIN block, because use outruns require. An example diff for view is attached. Second, I discovered by accident that TWiki 6.0.2 will work with Perl 5.26 if both the system's CGI (from CPAN or Linux distribution) and TWiki's CgiContrib are installed: In that case, CGI::Carp is taken from the system (before TWiki adds its own libraries), and CGI later on from CgiContrib. I caught the system library by installing CGI::Session which depends on CGI. I would prefer not to rely on such a construction. -- haj 2017-04-01

-- Contributors: Harald Jörg - 2017-03-21


With CGI::Carp, I don't think moving "require 'setlib.cfg';" to the beginning of the BEGIN block of scripts in bin/* is a good idea. Because setlib.cfg reads LocalLib.cfg, which can be big -- mine is 194 lines long. Problems in LocalLib.cfg should be displayed, which is why CGI::Carp::confess or Carp::confess was set to $SIG{__DIE__} at the beginning of the BEGIN block originally.

How about incorporating CGI.pm and CGI/*.pm into TWiki core rather than having CgiContrib? And then split setlib.cfg into two: settwikilib.cfg and setlib.cfg so that settwikilib.cfg will be executed at the beginning of the BEGIN block to make ../lib/CGI.pm and ../lib/CGI/* available.

-- Hideyo Imazu - 2017-06-01

Thanks for this info - it hadn't occurred to me that LocalLib.cfg problems would be an issue. Nevertheless, I think that the whole setlib.cfg + LocalLib.cfg + LocalSite.cfg stuff could use some rework.

  • Unfortunately, incorporating the CGI.pm modules into the core would improve nothing. They still would not be available in time. At the time the TWiki scripts are executed, the lib directory which holds both TWiki core and contrib libraries is not in @INC; all we have is bin. That's why setlib.cfg and LocalLib.cfg are located in bin: It is their job to make the TWiki libraries available.
  • Of course I agree that problems in LocalLib.cfg should be visible to the sysadmins.
    • However, I'd locate the problem in setlib.cfg which simply throws away any errors in LocalLib.cfg by doing a eval 'require...' and then ignores any return information in $@. This is bad. It is fine to continue if there is no LocalLib.cfg at all, but this is no justification to ignore all other errors as well.
    • The difference between Carp (which is available in all versions of the Perl core) and CGI::Carp (which vanished in Perl 5.22) is that CGI::Carp does some nifty apache-like formatting of the error message. Before we start making the bootstrapping stuff even more complicated by splitting setlib.cfg I'd rather replace CGI::Carp by plain Carp in this early stage of processing. I also think that fatalsToBrowser is no good idea in this stage because problems with LocalLib.cfg will affect each and every request, and there is nothing a TWiki user can do about it.

-- Harald Jörg - 2017-06-01

Back to the drawing board with regard to the CGI::Carp stuff:

  • Problems in LocalLib.cfg should be displayed, which is why CGI::Carp::confess or Carp::confess was set to $SIG{__DIE__} at the beginning of the BEGIN block originally.
I still agree. However, after locally reverting the changes in Rev. 30317 and 30319, I checked whether and how errors in LocalSite.cfg are reported by introducing various errors (a syntax error, or a plain die, some warnings) into LocalLib.cfg. As I suspected, neither of the errors is reported. The string eval in setlib.cfg captures fatal errors. $SIG{__DIE__}, which just changes the contents of the error message, never springs into action because there is no death. If LocalSite.cfg itself prints to STDERR with warn, the warnings should in the error log with both the old and new version of the scripts. The difference is that in the current version, they are no longer prepended by the timestamp and location, which is exactly what CGI::Carp provides. But it all started because CGI::Carp isn't available before setlib.cfg has done its job. So we could
  1. revert to the previous versions and make CGI::Carp a prerequisite for TWiki. I would prefer not to do this because CGI::Carp comes with CGI, and recent versions of CGI are neither compatible with TWiki nor included in current Perl core.
  2. Wrap require CGI::Carp in yet another string eval to see whether it is available, and proceed accordingly. This would clutter the scripts with another hunk of border-case garbage.
  3. Keep it as it is and examine in more detail what errors in LocalLib.cfg should be caught. A look at $@ after the string eval would be a very decent start. As far as I know, a "missing" LocalSite.cfg should not be fatal, so some heuristics need to be applied.

-- Harald Jörg - 2017-07-16

CGI::Carp again: Right now I am testing on a machine with Perl 5.24 (i.e. no CGI.pm nor CGI/Carp.pm included), and no system or CPAN version of the CGI distribution. The modules which come with TWiki are the only ones on the machine. As expected, this fails to compile because the TWiki libraries are only available after setlib.cfg and friends have been processed.

So a working solution would be to replace CGI::Carp by a plain Carp before setlib.cfg has been processed, and to replace the $SIG{__DIE__} handler by CGI::Carp::confess only after setlib.cfg has been processed.

By splitting setlib.cfg we could achieve that CGI::Carp gets loaded before "the second part of =setlib.cfg=". However, I would like to avoid yet another configuration item to be documented for installation and upgrade.

-- Harald Jörg - 2017-09-06

Topic attachments
I Attachment History Action Size Date Who Comment
Unknown file formatdiff cgi_carp.diff r1 manage 0.7 K 2017-04-01 - 16:36 HaraldJoerg An example diff how to avoid the CGI::Carp issue in Perl without CGI.pm
Edit | Attach | Watch | Print version | History: r7 < r6 < r5 < r4 < r3 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r7 - 2017-09-06 - HaraldJoerg
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2018 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.