Tags:
create new tag
, view all tags

Feature Proposal: Ship mod_perl_startup.pl suggested by ApacheConfigGenerator to improve efficiency.

Motivation

The ApacheConfigGenerator suggests that one would exist in the tools category: but it (4.2.0) does not.

Description and Documentation

Distribute a mod_perl_startup.pl script to improve effiency under mod_perl environments by precompiling all scripts and modules needed and used by TWiki during Apache startup instead of during each CGI request. The ApacheConfigGenerator already suggests ModPerl::Registry, and it should be taken full advantage of.

This script works well but is no longer by http://develop.twiki.org/~twiki4/cgi-bin/view/Bugs/Item5656. (so long as you use 4.2.1 or trunk -- SD)

Examples

PerlSwitches -T
PerlRequire /var/www/html/docs/tools/mod_perl_startup.pl
SetHandler perl-script
PerlResponseHandler ModPerl::RegistryPrefork
PerlSendHeader On
PerlOptions +ParseHeaders

Impact

WhatDoesItAffect: Performance

Implementation

Example mod_perl_startup.pl:

#!/usr/bin/perl -w
use strict;
use warnings;
use ModPerl::RegistryLoader;
use ModPerl::RegistryCooker;
use ModPerl::RegistryPrefork;
use ModPerl::Registry;
use File::Spec;

my $binurlbase = '/twiki/bin'; # must be set by the user
my $binbase = '/var/www/html/twiki/bin'; # must be set by the user
my $libbase = '/var/www/html/twiki/lib'; # must be set by the user
my $registrymodule = 'ModPerl::RegistryPrefork'; # must be set by the user
use lib '/var/www/html/twiki/lib/CPAN/lib'; # must be set by the user
use lib '/var/www/html/twiki/lib'; # must be set by the user

use CGI;
CGI->compile(':all');
use Algorithm::Diff;
use CGI::Carp;
use Config;
use Cwd;
use Data::Dumper;
use Error;
use File::Copy;
use File::Find;
use FileHandle;
use IO::File;
use Text::Diff;
use Time::Local;
use Archive::Tar;
use CGI::Cookie;
use CGI::Session;
use Digest::base;
use Digest::SHA1;
use Jcode;
use Locale::Maketext::Lexicon;
use Net::SMTP;
use Unicode::Map;
use Unicode::Map8;
use Unicode::MapUTF8;
use Unicode::String;
use URI;
use Scalar::Util;
use TWiki;

find({ wanted => \&found, untaint => 1}, $libbase);
sub found {
   if (m/.pm$/) {
      $File::Find::name =~ m|^$libbase/(CPAN/lib/)?(.+\.pm)|;
      my $module = $2;
      my $modbase = $module;
      $module =~ s|/|::|g;
      $module =~ s|\.pm$||;
      print "\nno $module\n" unless $INC{$modbase};
   }
}

my $rl = ModPerl::RegistryLoader->new(
   package => $registrymodule, # must be set by the user
);

chdir $binbase;

foreach my $binscript (<$binbase/*>) {
   my (undef,undef,$scriptname) = File::Spec->splitpath($binscript);
   $scriptname =~ m|^([^/]+)$|;
   my $script = $1;
   $binscript =~ m|^(.+)$|;
   my $bin = $1;
   open BINSCRIPT,'<',$binscript;
   if (<BINSCRIPT> =~ m|^\#\!/usr/bin/perl|) {
      $rl->handler("$binurlbase/$script", $bin);
   }
   close BINSCRIPT;
}

1;

-- Contributors: JoshuaCharlesCampbell - 26 May 2008

Discussion

Benchmarks on my system (Centos 5.1 (Apache 2.0 perl 5.8.8)):

ab -n 100 -c 2 twiki/bin/view

plain CGI: 2.37/sec

ModPerl::RegistryPrefork: 3.63/sec

ModPerl::RegistryPrefork w/ startup.pl: 3.49/sec

ModPerl::Registry (worker): 2.66/sec

ModPerl::Registry (worker) w/ startup.pl: crashes (double free())

Though the mod_perl documentation recommends doing this it seems to make it slower. I also thought worker was supposed to be faster. Maybe it's opposites day?

-- JoshuaCharlesCampbell - 27 May 2008

#!/usr/bin/perl -w
use strict;
use warnings;
use ModPerl::RegistryLoader;
use File::Spec;

my $binurlbase = '/twiki/bin'; # must be set by the user
my $binbase = '/var/www/html/twiki/bin'; # must be set by the user
my $libbase = '/var/www/html/twiki/lib'; # must be set by the user
my $registrymodule = 'ModPerl::RegistryPrefork'; # must be set by the user
use lib '/var/www/html/twiki/lib/CPAN/lib'; # must be set by the user
use lib '/var/www/html/twiki/lib'; # must be set by the user


my $rl = ModPerl::RegistryLoader->new(
   package => $registrymodule, # must be set by the user
);

chdir $binbase;

foreach my $binscript (<$binbase/*>) {
   my (undef,undef,$scriptname) = File::Spec->splitpath($binscript);
   $scriptname =~ m|^([^/]+)$|;
   my $script = $1;
   $binscript =~ m|^(.+)$|;
   my $bin = $1;
   open BINSCRIPT,'<',$binscript;
   if (<BINSCRIPT> =~ m|^\#\!/usr/bin/perl|) {
      $rl->handler("$binurlbase/$script", $bin);
   }
   close BINSCRIPT;
}

1;

This improves performance to 4.10 on this system.

-- JoshuaCharlesCampbell - 27 May 2008

#!/usr/bin/perl -w
use strict;
use warnings;
use ModPerl::RegistryLoader;
use File::Spec;

my $binurlbase = '/twiki/bin'; # must be set by the user
my $binbase = '/var/www/html/twiki/bin'; # must be set by the user
my $registrymodule = 'ModPerl::RegistryPrefork'; # must be set by the user

my $rl = ModPerl::RegistryLoader->new(
   package => $registrymodule, # must be set by the user
);

chdir $binbase;

foreach my $binscript (<$binbase/*>) {
   my (undef,undef,$scriptname) = File::Spec->splitpath($binscript);
   $scriptname =~ m|^([^/]+)$|;
   my $script = $1;
   if ($script !~ m/configure|register|resetpasswd|statistics/) { # don't precompile uncommon commands especially configure which has a ton of unnecessary s***
            $binscript =~ m|^(.+)$|;
            my $bin = $1;
            open BINSCRIPT,'<',$binscript;
            if (<BINSCRIPT> =~ m|^\#\!/usr/bin/perl|) {
                    $rl->handler("$binurlbase/$script", $bin);
            }
            close BINSCRIPT;
   }
}

1;
Best version so far.

-- JoshuaCharlesCampbell - 27 May 2008

I think we should put this into 4.2.1 - as its not going to do anything bu make lives easier.

-- SvenDowideit - 27 May 2008

I have no problem with adding this to 4.2.1.

Joshua - the ApacheConfigGenerator contains a 4 liner startup script today. You are very welcome to replace this by your own better script and maintain improvements there as well. That should also benefit users of older versions if they desire to update just the apache config.

-- KennethLavrsen - 27 May 2008

Kenneth - sure thing.

-- JoshuaCharlesCampbell - 27 May 2008

Note also that as of version 4.2.0, many of TWiki's "occasionally used" modules like e.g. TWiki::Attach are compiled lazily using require during runtime and so will not be caught automatically by use TWiki; in a mod_perl startup script. That said, I'd prefer to distribute only a very defensive subset of required CPAN modules in the mod_perl startup routine. If one of the modules is missing in an installation ( CGI::Session would be the most prominent example), Apache won't even start. In that case, there's no way to proceed to the sanity checks in configure, regardless of whether you run configure itself under mod_perl.

-- HaraldJoerg - 27 May 2008

Harald - Please note that the latest version of the script does not include any CPAN modules directly. It does include them by compiling commonly used scripts (add attach to the regular expression to exclude it if you consider it to be not worth precompiling). The goals of being able to sanity check the existence of modules and preload them obviously conflict. Obviously CGI::Session could very well be used on most if not every view -- not precompiling it defeats the purpose of precompiling at all. The solution to this is to not use the mod_perl_startup.pl script until you've got TWiki working without it.

Yes, this script assumes that your TWiki is configured and installed correctly.

-- JoshuaCharlesCampbell - 27 May 2008

Why not make $binbase look like this: use File::Basename;

my $binbase = dirname( $0 ); $binbase =~ s/\/tools$/\/bin/;

The less an administrator has to edit, the better!

-- TimotheLitt - 14 Oct 2008

I am setting this to parked and no committed developer. Please feel free to flip that and own & implement.

-- PeterThoeny - 2010-08-01

By chance, I've just gotten back to turning on mod_perl on my site. Here's my version of the startup script.

Changes:

  • I found that reading files in /bin to decide what to register was dangerous - some addons blew-up. It seems safer to load from a verified list. I left autoscan as an option.
  • The error detection moved to a BEGIN block because compilation would fail due to the use ModPerl outside of apache
  • Less configuration: the script finds /bin and /lib relative to FILE, and reads LocalSite.cfg to find the base URL.
  • If no LocalSite.cfg, returns success so that the server can start - allowing you to run configure. An error is logged.
  • Added logging (the success logging is off by default, though it's useful when running configtest). A case could be made to turn it on by default...

With these changes, it should work out-of-the-box for most people, although folks with more experience may want to update the 'safe scripts' list.

By the way - it would now be trivial to add a option to configure to control whether mod_perl preloads TWiki modules - I've put the support code into this script, just remove the =1 from $enabled to turn it on.

Enjoy.

#!/usr/bin/perl -w

use strict;
use warnings;

BEGIN {
    defined( $ENV{MOD_PERL} ) && $ENV{MOD_PERL} =~ /mod_perl/ or die "mod_perl_startup called, but mod_perl not used!";
}

# Select registry module for MOD_PERL that matches your server configuration

my $registrymodule = 'ModPerl::RegistryPrefork'; # must be set by the user
#my $registrymodule = 'ModPerl::Registry'; # must be set by the user

# Subdirectory containing scripts and LocalSite.cfg

my $bindir = 'bin';

# Set to automatically scan for scripts (not recommended as plugin and contrib scripts may not be ModPerl-safe)

my $autoscan = 0;

# Set to log scripts being preloaded to stderr (useful when running configtest (you DO run configtest, right?)

my $logsuccess = 0;

# Safe and worthwhile scripts to preload (There's a parallel list in your twiki.conf file)

#my @scripts = split( '\|', 'attach|changes|compare|edit|login|logon|manage|oops|preview|rdiff|rdiffauth|rename|rest|save|search|upload|view|viewauth|viewfile' );
my @scripts = split( '\|', 'attach|edit|manage|rename|rest|save|upload|view|viewauth|viewfile' );

use ModPerl::RegistryLoader;
use File::Spec;
use File::Basename;

my( $binbase, $lib );
BEGIN {
    $binbase = dirname( dirname( __FILE__ ) );
    $lib = $binbase;
    $lib .= '/lib';
}
$binbase .= "/$bindir";

use lib "$lib";

# Find the base url from the configuration file.

my $binurlbase;

my $enabled;

if( open( my $cfg, '<', "$lib/LocalSite.cfg" ) ) {
    while( <$cfg> ) {
   if( /^\$TWiki::cfg\{ScriptUrlPath\}\s+=\s+'(.*)';/ ) {
       $binurlbase = $1;
   }
   if( /^\$TWiki::cfg\{UseModPerl\}\s+=\s+(.*);/ ) {
       $enabled = $1;
   }
    }
    close $cfg;
    return 1 unless( $enabled );
    $binurlbase or die "No ScriptUrlPath found in $lib/LocalSite.cfg - please run configure";
} else {
    my $t = localtime(); 
    print STDERR "[$t] [warning] mod_perl_startup: Unable to open $lib/LocalSite.cfg: $!\n\tNo modules will be preloaded - please run configure\n";
    return 1;
}
my $rl = ModPerl::RegistryLoader->new(
   package => $registrymodule,
);

if( $autoscan ) {
    chdir $binbase;

    foreach my $binscript (<$binbase/*>) {
   my (undef,undef,$scriptname) = File::Spec->splitpath($binscript);
   $scriptname =~ m|^([^/]+)$|;
   my $script = $1;
   if ($script !~ m/configure|register|resetpasswd|statistics/) { # don't precompile uncommon commands especially configure which has a ton of unnecessary s***
            $binscript =~ m|^(.+)$|;
            my $bin = $1;
            open BINSCRIPT,'<',$binscript or die "Can't open $binscript for preload: $!";
            if (<BINSCRIPT> =~ m|^\#\!/usr/bin/perl|) {
                if( $logsuccess ) {
               my $t = localtime(); 
          print STDERR "[$t] [notice] mod_perl_startup: Preloading $bin: $binurlbase/$script\n";
                }
      $rl->handler("$binurlbase/$script", $bin);
            }
            close BINSCRIPT;
   }
    }
} else {
    # Pre-compile only known scripts

    foreach my $script (@scripts) {
   my $file = "$binbase/$script";

   unless( -f $file && -x $file ) {
       my $t = localtime();
       print STDERR "[$t] [error] mod_perl_startup: ERROR: Can't preload $file: " . (-f $file? "not found\n" : "not executable\n");
       next;
   }
        if( $logsuccess ) {
       my $t = localtime(); print STDERR "[$t] [notice] mod_perl_startup: Preloading $file: $binurlbase/$script\n";
        }
   $rl->handler("$binurlbase/$script", $file);
    }
}

1;
-- TimotheLitt - 2010-08-25

Thank you Timothe for sharing this with the TWiki community!

We could take this into the distribution. One thing could be changed to make it work with non-standard directory structure: The $twikiLibPath variable in twiki/bin/LocalLib.cfg points to the twiki/lib path. (TWiki.org for example has a ...../twiki/lib5x0 lib directory, allowing several TWiki versions to be running at the same time). Your script could be enhanced to use LocalLib.cfg if it exists.

-- PeterThoeny - 2010-08-25

Hmm, I didn't see any use of that file & it's modestly ugly because use lib is effectively a BEGIN - I don't like config variables in BEGIN. But since you have a valid use case, and it only took 3 minutes - here's an updated script that does what you want.

Since you want to distribute it, I also moved all the configuration (except the /bin directory name, which is required to find the config file) into configure so that it's packaged as an integrated feature.

I also provided a patch for TWiki.spec so that configure understands what to do. You might want to put it in a different section - but this seemed reasonable to me. I intentionally did NOT provide configure support for the exclusions mask for the AutoScan mis-feature. I think it's dangerous and should be deprecated. I only left it in so that legacy users can upgrade easily. I strongly suggest that they move to listing safe files. (But if they insist, they can still edit the file manually.)

You'll want to remove generating mod_perl_startup.pl from the config generator, and put the file provide here under /tools in the distribution.

One more thing - I don't understand the reference to Item5656 - it seems to be an unrelated bug.

In any case, I'm done - feel free to adjust to suit.

#!/usr/bin/perl -w

use strict;
use warnings;

BEGIN {
    defined( $ENV{MOD_PERL} ) && $ENV{MOD_PERL} =~ /mod_perl/ or die "mod_perl_startup called, but mod_perl not used!";
}

# Subdirectory containing scripts and LocalSite.cfg
# This is the ONLY hard-coded configuration variable; although some defaults
# are coded here, all the others should come from Configure.

my $bindir;
BEGIN { $bindir = 'bin'; }

###############

# Set to automatically scan for scripts (not recommended as plugin and contrib scripts may not be ModPerl-safe)

my $autoscan = 0;

# Set to log scripts being preloaded to stderr (useful when running configtest (you DO run configtest, right?)

my $logsuccess = 0;

# Safe and worthwhile scripts to preload (There's a parallel list in your twiki.conf file)
# From configure

#my @safescripts = split( '\|', 'attach|changes|compare|edit|login|logon|manage|oops|preview|rdiff|rdiffauth|rename|rest|save|search|upload|view|viewauth|viewfile' );
my @safescripts = split( '\|', 'attach|edit|manage|rename|rest|save|upload|view|viewauth|viewfile' );

use ModPerl::RegistryLoader;
use File::Spec;
use File::Basename;

my( $binbase, $lib );
BEGIN {
    $binbase = dirname( dirname( __FILE__ ) );
    $lib = $binbase;
    $lib .= '/lib';
    $binbase .= "/$bindir";
    if( open( my $cfg, '<', "$binbase/LocalLib.cfg" ) ) {
   my $clib;
   while( <$cfg> ) {
       if( /^\s*\$twikiLibPath\s+=\s+"(.*)";/ ) {
      $clib = $1;
      last;
       }
   }
   close $cfg;
   $lib = $clib if( defined $clib );
    }
}


use lib "$lib";

# Find our configuration variables in the configuration file.

my( $binurlbase, $enabled, @scripts, $registrymodule );

if( open( my $cfg, '<', "$lib/LocalSite.cfg" ) ) {
    while( <$cfg> ) {
   if( /^\$TWiki::cfg\{ScriptUrlPath\}\s+=\s+'(.*)';/ ) {
       $binurlbase = $1;
   }
   if( /^\$TWiki::cfg\{UseModPerl\}\{Scripts\}\s+=\s+'(.*)';/ ) {
       @scripts = split( '\|', $1 );
   }
   if( /^\$TWiki::cfg\{UseModPerl\}\{Registry\}\s+=\s+'(.*)';/ ) {
       $registrymodule = $1;
   }
   if( /^\$TWiki::cfg\{UseModPerl\}\{Enabled\}\s+=\s+(.*);/ ) {
       $enabled = $1;
   }
   if( /^\$TWiki::cfg\{UseModPerl\}\{LogSuccess\}\s+=\s+(.*);/ ) {
       $logsuccess = $1;
   }
   if( /^\$TWiki::cfg\{UseModPerl\}\{AutoScan\}\s+=\s+(.*);/ ) {
       $autoscan = $1;
   }
    }
    close $cfg;
    return 1 unless( $enabled );
    $registrymodule or die "No Registry module specified in $lib/LocalSite.cfg - please run configure";
    $binurlbase or die "No ScriptUrlPath found in $lib/LocalSite.cfg - please run configure";
    @scripts = @safescripts unless( @scripts );
} else {
    my $t = localtime(); 
    print STDERR "[$t] [warning] mod_perl_startup: Unable to open $lib/LocalSite.cfg: $!\n\tNo modules will be preloaded - please run configure\n";
    return 1;
}

my $rl = ModPerl::RegistryLoader->new(
                  package => $registrymodule,
                 );

if( $autoscan ) {
    chdir $binbase;

    foreach my $binscript (<$binbase/*>) {
   my (undef,undef,$scriptname) = File::Spec->splitpath($binscript);
   $scriptname =~ m|^([^/]+)$|;
   my $script = $1;
   if ($script !~ m/configure|register|resetpasswd|statistics/) { # don't precompile uncommon commands especially configure which has a ton of unnecessary s***
            $binscript =~ m|^(.+)$|;
            my $bin = $1;
            open BINSCRIPT,'<',$binscript or die "Can't open $binscript for preload: $!";
            if (<BINSCRIPT> =~ m|^\#\!/usr/bin/perl|) {
                if( $logsuccess ) {
               my $t = localtime(); 
          print STDERR "[$t] [notice] mod_perl_startup: Preloading $bin: $binurlbase/$script\n";
                }
      $rl->handler("$binurlbase/$script", $bin);
            }
            close BINSCRIPT;
   }
    }
} else {
    # Pre-compile only known scripts

    foreach my $script (@scripts) {
   my $file = "$binbase/$script";

   unless( -f $file && -x $file ) {
       my $t = localtime();
       print STDERR "[$t] [error] mod_perl_startup: ERROR: Can't preload $file: " . (-f $file? "not found\n" : "not executable\n");
       next;
   }
        if( $logsuccess ) {
       my $t = localtime(); print STDERR "[$t] [notice] mod_perl_startup: Preloading $file: $binurlbase/$script\n";
        }
   $rl->handler("$binurlbase/$script", $file);
    }
}

1;
--- /var/www/servers/twiki/lib/TWiki.spec~      2008-12-15 10:26:07.000000000 -0500
+++ /var/www/servers/twiki/lib/TWiki.spec       2010-08-26 07:15:46.000000000 -0400
@@ -152,10 +152,41 @@
 # default TWiki registration process to store registrations that are pending
 # verification.</li>
 # </ul>
 # $TWiki::cfg{WorkingDir} = '/home/httpd/twiki/working';

+# **BOOLEAN**
+# If you have succesfully configured your TWiki site under apache, you may want to enable
+# Mod_Perl.  To do this, you must modify your apache configuration file (the configuration
+# generator site http://twiki.org/cgi-bin/view/TWiki/ApacheConfigGenerator provides a template)
+# and enable this setting.  Do NOT enable before you have successfuly configured your site, as
+# Configure will not run.  This setting has no effect unless Mod_Perl is configured.  Mod_Perl
+# is an apache-only feature.
+$TWiki::cfg{UseModPerl}{Enabled} = 0;
+
+# **STRING 30**
+# Mod_Perl registers (caches) your scrips for increased performance.  The registry you
+# use depends on your server configuration.  If your server uses Prefork for it's process
+# models, use ModPerl::RegistryPrefork.  Otherwise, use ModPerl::Registry.
+$TWiki::cfg{UseModPerl}{Registry} = 'ModPerl::RegistryPrefork';
+
+# **BOOLEAN EXPERT**
+# Mod_Perl startup normally only logs errors.  If you need success logging (for debugging),
+# enable this feature.
+$TWiki::cfg{UseModPerl}{LogSuccess} = 0;
+
+# **BOOLEAN EXPERT**
+# Mod_Perl startup normally loads only scripts that are believed to be safe under Mod_Perl.
+# If you want all scripts in you bin directory loaded, you can enable this feature.  This
+# is NOT recommended, since some plugins and contribs are known not to be safe.
+$TWiki::cfg{UseModPerl}{AutoScan} = 0;
+
+# **STRING EXPERT**
+# Mod_Perl startup normally loads only scripts that are believed to be safe under Mod_Perl.
+# You can adjust this list with the following feature.  Separate script names with |.
+$TWiki::cfg{UseModPerl}{Scripts} = 'attach|edit|manage|rename|rest|save|upload|view|viewauth|viewfile';
+
 # **STRING 10**
 # Suffix of TWiki CGI scripts (e.g. .cgi or .pl). You may need to set this
 # if your webserver requires an extension.
 $TWiki::cfg{ScriptSuffix} = '';

---End-of-patch
-- TimotheLitt - 2010-08-26

Edit | Attach | Watch | Print version | History: r15 < r14 < r13 < r12 < r11 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r15 - 2010-08-26 - TimotheLitt
 
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2017 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.