mod_perl1Add my vote for this tag performance1Add my vote for this tag scalability1Add my vote for this tag search1Add my vote for this tag create new tag
, view all tags


Native code search for use with CGI accelerators


By default TWiki searches its database by forking the standard grep program in a sub-process to search through the text files in the database cache. This is usually pretty fast, but unfortunately in some environments (specifically with mod_perl) forking a sub-process is very expensive, and another approach is needed.

NativeSearch is a c-code module that can be plugged into TWiki to replace the default searching algorithm. It publishes an XS interface that allows it to be called from Perl, without having to fork a sub-process. It has comparable performance to the standard forking search in most environments, but is often significantly faster than forked search when TWiki is run under a CGI accelerator. The effect is most pronounced with mod_perl, but is also there for SpeedyCGI.

You require shell access to the server to install this module. Refer to TWiki:Codev.NativeSearch for a list of systems where the module has been successfully compiled and installed (and please update it if you find a new one).

Installation Instructions

If you are installing on a TWiki version before 4.1.2, please refer to TWiki:Codev.NativeSearch for instructions. Note that an old version of the package was pre-installed with 4.1.2; that pre-installation will be overwritten by this package.

Before you start, there are some prerequisites.

  1. you must have ExtUtils::MakeMaker installed (available from CPAN)
  2. you must have a C compiler (gcc has been tested). Most Unix/Linux systems will have this already. On Win32 MinGW is recommended.
  3. you will need make (a standard development tool). Unix/Linux users will probably have this already; others may have to search. You are recommended to use nmake on Win32.
  4. finally you will need the pcre library (required for perl compatible regular expressions). This is a standard Linux library and will be available in an RPM or Debian package or whatever for your system. It is also available from GnuWin32 for Windows. You need both the runtime library (binary) and also the developer header files, which are usually shipped in a seperate package.

You do not need to install anything in the browser to use this extension. The following instructions are for the administrator who installs the extension on the server where TWiki is running.

Like many other TWiki extensions, this module is shipped with a fully automatic installer script written using the BuildContrib.

  • If you have TWiki 4.2 or later, you can install from the configure interface (Go to Plugins->Find More Extensions)
  • If you have any problems, then you can still install manually from the command-line:
    1. Download one of the .zip or .tgz archives
    2. Unpack the archive in the root directory of your TWiki installation.
    3. Run the installer script ( perl <module>_installer )
    4. Run configure and enable the module, if it is a plugin.
    5. Repeat for any missing dependencies.
  • If you are still having problems, then instead of running the installer script:
    1. Make sure that the file permissions allow the webserver user to access all files.
    2. Check in any installed files that have existing ,v files in your existing install (take care not to lock the files when you check in)
    3. Manually edit LocalSite.cfg to set any configuration variables.

Building the c-code

cd to the root directory in your TWiki installation. Now, assuming you have administrator (root) access to the server:

$ cd tools/native_search
$ perl Makefile.PL
$ make install
$ perl test.pl -i -l NativeTWikiSearch test.pl Makefile.PL NativeTWikiSearch.xs
(on Win32 you should use nmake)

Make sure that perl test.pl returns something sane, and NOT a Segmentation fault. (test.pl is simply a stub interface to the native search, so it behaves like grep. The test above simply greps for the string "NativeSearch" in the files Makefile.PL and NativeTWikiSearch.xs)

If you do not have administrator (root) access to the server, you can still install. Follow the steps above, but use perl Makefile.PL INSTALLBASE=/your/dir to install to a different directory. Exactly where you install depends on your system configuration.

Now, in configure, under "Store Settings", you should be able to select the search algorithm TWiki::Store::SearchAlgorithms::Native (click the button first if you have TWiki 4.2 or later).

Run a TWiki search, and watch the web server error log for any problems.

Because performance can differ widely between two apparently similar webserver configurations, you are recommended to do your own benchmarking and select the search algorithm that performs best for you.

Contrib Info

Many thanks to the following sponsors for supporting this work:

Contrib Author: Crawford Currie http://c-dot.co.uk
Copyright: © 2007 WikiRing;
© 2007-2011 TWiki:TWiki/TWikiContributor
License: GPL (GNU General Public License)
Contrib Version: 21570 (2011-07-06)
Change History:  
2011-07-06: TWikibug:Item6764: Doc improvements -- TWiki:Main.PeterThoeny
2010-05-03: TWikibug:Item6433: Doc improvements; replacing TWIKIWEB with SYSTEMWEB
2007-09-27: TWikibug:Item4713: Initial version
ExtUtils::MakeMaker>=0Required for building makefiles
Contrib Home: http://twiki.org/cgi-bin/view/Plugins/NativeSearchContrib
Feedback: http://twiki.org/cgi-bin/view/Plugins/NativeSearchContribDev
Appraisal: http://twiki.org/cgi-bin/view/Plugins/NativeSearchContribAppraisal

Related Topics: TWikiContribs, DeveloperDocumentationCategory, AdminDocumentationCategory, TWikiPreferences

Topic attachments
I Attachment History Action Size Date Who Comment
Unknown file formatmd5 NativeSearchContrib.md5 r4 r3 r2 r1 manage 0.2 K 2011-07-06 - 07:09 PeterThoeny  
Compressed Zip archivetgz NativeSearchContrib.tgz r4 r3 r2 r1 manage 9.9 K 2011-07-06 - 07:09 PeterThoeny  
Compressed Zip archivezip NativeSearchContrib.zip r4 r3 r2 r1 manage 14.3 K 2011-07-06 - 07:09 PeterThoeny  
Unknown file formatEXT NativeSearchContrib_installer r2 r1 manage 3.7 K 2011-07-06 - 07:09 PeterThoeny  
Edit | Attach | Watch | Print version | History: r7 < r6 < r5 < r4 < r3 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r7 - 2013-10-16 - PeterThoeny
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2016 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.