Refactoring Proposal: Setting Lib Path

Motivation

This is the result of TomKagan's comments on TWikiInstallationGuide.

Particularly for TWikiOnWebHostingSites, it may be necessary to move the lib directory to somewhere distant from the cgi-bin directory, meaning that the TWiki scripts' use lib statements won't work.

Description

One alternative to editing all the TWiki scripts, in order to set a different lib directory location, would be to just set the environment variable PERL5LIB, e.g. to /path/to/home/lib. Using Apache, this can be done in the twiki/bin/.htaccess file, e.g.:

SetEnv PERL5LIB /path/to/home/lib

You can use double quotes around the path if it includes spaces. Other web servers may require the server admin to set this, which is not much help for web hosting setups.

Unfortunately this would require Perl TaintChecking to be removed (see perldoc perlrun), which is somewhat questionable security practice, and another global edit.

It is desirable to leave taint checking turned on, even in a production server. Taint checks could find latent security holes if an unusual code path is exercised, and of course the local TWiki administrator may make their own code changes, so it seems like taint checking should remain enabled.

To avoid having to turn off taint checking, how about centralising in one file the statements that set the lib path, i.e.

use lib ( '.' );
use lib ( '../lib' );

If these are placed in twiki/bin/setlib.cfg, and executed by each TWiki script, it would be much easier to relocate the twiki/lib directory. The code at the top of each script now looks like:

BEGIN { use lib '.';  require 'setlib.cfg'; }
use TWiki;

The use lib '.'; is more portable than just saying "./setlib.cfg". The setlib.cfg file looks like this:

use lib qw( . ../lib );
1;          # Required return value for module

This is the only file that needs editing in order to set a non-standard path for the lib ory. I don't think that thee current directory is needed in the use lib, since nothing in the TWiki code (that I can think of) loads modules from the bin directory - omitting it would be a slight performance win.

I've tested this on TWikiAlphaRelease code but not yet committed it. A somewhat related issue is how to do an efficient startup.pl script for ModPerl, to set the lib path and preload commonly used modules such as TWiki.pm.

Any comments? On a related topic, TaintChecking in TWiki needs some improvement IMO.

-- RichardDonkin - 27 Apr 2002

Discussion

I've implemented the above in TWikiAlphaRelease, though not completely at present. See CVS:bin/view and CVS:bin/setlib.cfg for how it works. This also lets users without root access install Perl modules under their home directory and have TWiki use these modules as well as standard ones (see CpanPerlModulesRequirement).

-- RichardDonkin - 28 Apr 2002

Is this setlib.pl file some sort of Perl convention akin to pyhon's "specially named file containing additional path information" scheme? If it's not, is there any compelling reason not to just add it to twiki.cfg as something like PerlPath or LocalPerlPath ? Extra file opens and fetches are expensive, and twiki.cfg already represents a single place for lots of settings.

As for mod_perl, doesn't it have a path setting directive?

-- DavidLeBlanc - 28 Apr 2002

There isn't any Perl convention that I'm aware of. The trouble is that TWiki.cfg is in the lib directory, so I can't put the location of the lib directory there, and I don't really want to move TWiki.cfg into the bin directory.

File opening is not that expensive, particularly when you consider that TWiki already opens a lot of module files. At least on Windows systems, starting a new process is much more expensive. I doubt that this adds a lot of overhead.

-- RichardDonkin - 29 Apr 2002

I do not want to play the devil's advocate, but do we need this added complexity?

It should be possible to install TWiki on a predefined cgi-bin on any hosted server. TWiki.pm does an include of "." and "../lib" for libraries, so for this case you can place the TWiki libs right into the cgi-bin directory if you can't create cgi-bin/../lib.

Also, I do not think the set lib functionality fits the TWikiMission.

So I would vote to just document workarounds of how to install TWiki on a server where you can't use "../lib".

-- PeterThoeny - 29 Apr 2002

I'm afraid I find the 'put everything in cgi-bin' workaround a bit painful to contemplate, but that's just me I suppose! It does seem much cleaner to just edit one file and not to continually search '.' for libraries (at least I now understand why TWiki does this, though). The added complexity is fairly low, in fact it's only 5 lines of actual code in setlib.cfg, and most people won't need to touch it - but when they do, it will make life quite a bit easier. Performance should actually improve slightly, as the extra file access is compensated for by not loading the 'lib' helper module.

As for the TWikiMission: the setlib.cfg implementation also makes it much easier to install additional Perl modules on an intranet server where you don't have root access, by pointing the $localPerlLibPath variable to (say) /home/fred/localperl/lib. This is fairly important for use with plugins that use additional modules, as it avoids having to install another copy of Perl or edit several TWiki scripts to change the library path.

-- RichardDonkin - 29 Apr 2002

Richard, I assure you that any file open/fetch is far far far FAR more expensive then any process creation unless the dumb os thrashes (not impossible with Windows of course ). At a time when a fast CPU was 20 MHZ, the relative speed of a disk access was reckoned in many 10's of YEARS. CPU's have gotten faster, and so have hard drives, but not even close to the advance in processor speeds. (IIRC, the example was that if some particularly expensive CPU operation took 1ms, the equivelant disk operation was 52 years.)

I do agree that "put everything in cgi-bin" is painful to contemplate though

-- DavidLeBlanc - 04 May 2002

I'm assuming that process creation includes a file open to fetch the script or executable (i.e. Unix fork and exec, or the Windows equivalent which is just one API call) - so it is always going to be more expensive IMO. Pure process creation is of course faster if it is just a matter of creating a new empty process, but that's not very useful - you either need to copy all the VM pages across, which can be expensive if disk paging is involved, or you need to find a file on disk and page it in, which is also expensive.)

Anyway, the key point is that the performance impact of the setlib.cfg change should be neutral, because it avoids one file concat(use lib) while adding another.

-- RichardDonkin - 04 May 2002

setlib.cfg would help installation at my work place. This is because we have a company wide standard Perl installation on every unix box and I prefer to keep TWiki libraries separate from others e.g. database libraries.

-- JohnTalintyre - 05 May 2002

I concur. One of the few things that doesn't work on TWikiOnWindowsCE is the construct use lib '.'; I don't yet know why.

-- MartinCleaver - 07 May 2002

It seems like this would be useful in a couple of different places, including intranets, and IMO it is a generally useful feature that makes it easier to install TWiki in various environments. I've tested it on Linux and ActivePerl for Windows.

So... I'd like to put this in - the overhead is minimal in performance, and people who don't need it won't need to worry about it.

-- RichardDonkin - 07 May 2002

The increased complexity I was refering to is about additional files cluttering the bin directory and additional docs describing the lib path extension, docs that do not apply for most of the users.

Lets put his into the release since the majority is for this feature.

-- PeterThoeny - 08 May 2002

Martin, CE actually has virtually no real concept of an hierarchical directory structure, no matter what the CE File Explorer view suggests and all paths must be absolute - think of them as just real long labels of folders that can contain backslashes. "." or ".." mean nothing to the CE storage manager (which in fact really isn't a file system).

-- DavidLeBlanc - 15 May 2002

BEGIN { require './setlib.cfg'; } is currently done just for the view script. To be consistent it should be done for all other scripts as well.

-- PeterThoeny - 12 Jul 2002

I have now put this into CVS - all scripts now require this setlib.cfg file, with exception of testenv, which reverts to the ../lib path if setlib.cfg is not available. See any file in CVS:bin for details.

-- RichardDonkin - 03 Aug 2002

The new setlib stuff is causing me some problems under mod_perl on my Win32 install, where the current directory does not seem to be TWiki's bin. I suggest changing the setlib BEGIN block to this:

     BEGIN { use FindBin qw($Bin); require "$Bin/setlib.cfg"; }

[Later] Well, this turns out to be problematic for a couple of reasons:

• FindBin:: uses CWD::, which fails the taint check on my system (ActiveState Win32 Perl 633)
• FindBin:: uses global variables, which will only be set correctly for the first invocation under mod_perl.

Anybody have a better suggestion?

-- FritzMueller - 29 Sep 2002

Interesting about the FindBin problems. Did you try editing setlib.cfg to use the full pathname on your system? This should work OK, e.g.:

$twikiLibPath = 'd:/twiki/lib';

setlib.cfg is intended for exactly this sort of problem, where the ../lib approach doesn't work. You can also edit this file to point to CPAN modules installed under your home directory, if you are not the administrator of the server.

-- RichardDonkin - 30 Sep 2002

I did end up coding full paths within my setlib.cfg. This in itself wouldn't have bothered me -- but, I also had to edit a full path into the use './setlib.cfg' line of each of the other bin scripts, for the same reason, and that somewhat of defeated the purpose of having setlib.cfg.

Many of the win32 mod_perl recipes (for pre setlib.cfg TWiki releases) suggest adding TWiki's bin and lib to @INC before the fact via a registry hack. This is probably okay for Apache::Registry mod_perl usage, which is all one big "ball of mud" anyway, but seems wrong to me for Apache::PerlRun usage, since it will add TWiki's dirs to @INC for all scripts running under the web server. In any case, these mod_perl folks are going to break when they upgrade to a setlib.cfg TWiki due to encountering the './setlib.cfg' problem at the head of each script.

I think the setlib.cfg approach is almost right (edit paths in one place) but I think it could use some sort of refinement to prevent mod_perl users from having to edit each file anyway. Haven't yet figured out what this would be, though. Perhaps the use line just needs to be made robust in the face of failure, and then the setting of @INC in the win32 mod_perl world or other tweaky environments (CE?) handled by some different, external means.

-- FritzMueller - 01 Oct 2002

Who knew one little comment in the configuration documents of TWiki could spawn such a large comment? I don't have a mod_perl and win32 setup to test, but has anyone tried "require 'setlib.cfg'" instead of "require './setlib.cfg'" ?

-- TomKagan - 01 Oct 2002

I've put a slightly different use of 'setlib.cfg' into the TWikiAlphaRelease, and tested this on CGI but not ModPerl (which I don't have currently):

BEGIN { unshift @INC, '.'; require 'setlib.cfg'; }

This works identically for those without mod_perl, where the current directory is the TWiki 'bin' directory. ModPerl users should set @INC through the registry or whatever, and prepend the full path to the twiki/bin and twiki/lib directories so that no time is wasted by searching the current directory (although mod_perl will cache the loaded modules anyway so this isn't really an issue except after Apache server startup.)

Note that you can just put (the full path to) twiki/bin into the registry etc, and then set the full path to twiki/lib in setlib.cfg, or you can put both in the registry - it's up to you.

Can someone using ModPerl test this out using the TWikiAlphaRelease?

-- RichardDonkin - 14 Oct 2002

These changes need to be added to various plugins which add perl scripts to the core.

-- TomKagan - 24 Nov 2002

And of course none of this works properly if you are using mod_perl, you'll get random 'module not found in @INC' messages in your server log. This is because the dorking around with @INC only lasts until the end of the first run of the script as the aforementioned dorking is in a BEGIN block. See http://perl.apache.org/docs/1.0/guide/porting.html#_INC_and_mod_perl:

"When running under mod_perl, once the server is up @INC is frozen and cannot be updated. The only opportunity to temporarily modify @INC is while the script or the module are loaded and compiled for the first time. After that its value is reset to the original one. The only way to change @INC permanently is to modify it at Apache startup."

See ModPerlUnix for a workaround.

-- AlanBurlison - 11 Mar 2006