create new tag
, view all tags

Native Search

By default TWiki searches its database by forking the standard grep program in a sub-process to search through the text files in the database cache. This is usually pretty fast, but unfortunately in some environments (specifically with mod_perl) forking a sub-process is very expensive, and another approach is needed.

NativeSearch is a c-code module that can be plugged into TWiki to replace the default searching algorithm. It publishes an XS interface that allows it to be called from Perl, without having to fork a sub-process. It has comparable performance to the standard forking search in most environments, but is often significantly faster than forked search when TWiki is run under a CGI accelerator. The effect is most pronounced with mod_perl, but is also there for SpeedyCGI.

You require shell access to the server to install this module.

You can find packages for download and installation instructions in NativeSearchContrib. If you are using a TWiki version before 4.1.2, see below.

To help others, please list below the platforms you were able to install on.

Platform Perl Version Accelerator Whodunnit
Debian 5.8.4 mod_perl CrawfordCurrie
Debian Etch 5.8.8, PPC speedy and perperl TWiki:Main.SimonRaven
Fedora Core 4 5.8.6   SvenDowideit
Fedora 7     PeterStephens
RHE 4 5.8.? mod_perl CrawfordCurrie
Centos 4 5.8.4 mod_perl ArthurClemens
SuSE 9.3 5.8.6 mod_perl StefanScherer
Windows XP (Home Edition) SP2 5.8.8 mod_perl TWiki:Main.CrawfordCurrie

Installing on old TWikis

If you need to install native search on a twiki version prior to 4.1.2, first follow the instructions in NativeSearchContrib. Now, you need to patch the file lib/TWiki/Store/RcsFile.pm. You need to find the searchInWebContent method. Delete it (or comment it out if you aren't brave) and use this version instead:

-- Contributors: CrawfordCurrie, SvenDowideit - 11 Feb 2007

Nice work Crawford and Sven! Totally in line with the TWikiMission!

-- PeterThoeny - 16 Feb 2007

Very good improvement! But I had problems make it work. I'm using SuSE Linux Professional 9.3 with Perl 5.8.6 and Apache 2 with mod_perl prebuilt by SuSE. Compiling the perl module was fine, but after installing and restarting Apache TWiki does not find any body text.

Apache's error_log showed some strange messages: httpd2-prefork: unknown option bit(s) set: Success

So I went into cgrep.c and added some debug messages. A good thing is to show the version of pcre with warn ("DEB: cgrep starts with pcre V %s", pcre_version()); at the beginning of the cgrep() C function. And then I found out what was wrong.

When I start the test.pl script, the version of pcre is V 5.0 (the prebuilt version from SuSE). But when I started a search in TWiki, the error_log showed me V 3.9 02-Jan-2002 -- oops.

Then I downloaded the current version 7.0 and compiled and installed it. Same effect: command line test says V 7.0, Apache says V 3.9.

I cannot find any other shared library of libpcre on my machine. It seems to me that the httpd binary is linked against an old pcre and this version is then used also in the perl module.

So I patched the pcre header file to rename the public function names from "pcre_*" to "mypcre_*" and linked it statically to the perl module.

After that hard patch also my Apache says V 7.0 and the search is working fine and fast.

-- StefanScherer - 29 Mar 2007

I found that building pcre on OS X 10.4 (Tiger) went ok without a problem, but when running the test above the program gave the error dyld: lazy symbol binding failed: Symbol not found: _getline.

Even using gcc_select 3.3 doesn't solve the problem.

I had to (re)install CPAN:ExtUtils::MakeMaker because my version did not recognize INSTALL_BASE.

-- ArthurClemens - 01 May 2007

Thanks AaronPeterson for providing a patch for Solaris. Tracked in Bugs:Item4220

-- PeterThoeny - 06 Jun 2007

I moved the patch to the bug topic. Aaron, please see my comment in Bugs:Item4220.

-- CrawfordCurrie - 08 Jun 2007

Using AaronPeterson's Solaris patch I got NativeSearch working on Mac OS X 10.4.10 (Tiger), Perl v5.8.6.

I had to compile the gigantic gettext package as the patch required though.. wish there was a smaller substitute.

After making the changes to the Makefile and cgrep.c, perl test.pl returned the right result.

-- ClintMarkGono - 25 Jun 2007

Nice one, Clint. Still waiting for someone to pick up the (fairly trivial) challenge of providing a simple function to replace getline, which would obviate the need for gettext.

-- CrawfordCurrie - 25 Jun 2007

Is there any chance of getting this to work with ActivePerl in Windows? It would be really nice to not have any dependencies to grep etc. I'm having some issues with intermittent empty searches which I believe is related to cygwin/grep/mod_perl..

-- ParBohrarper - 07 Aug 2007

Hmmmm - what makes you think it doesn't work on Windows? The C-code is really simple, and AFAIK the XS interface in ActiveState is the same as that used in GNU perl. "All" you need to do is compile it on the windows platform. I never bothered trying, myself.

-- CrawfordCurrie - 08 Aug 2007

I've tried compiling it with Visual Studio 6.0 and nmake. After copying some stuff (getline) from libgw32c and commenting some other stuff out (warn) I managed to compile it. It doesn't work though, it crashes on the test input from above. I suspect it has something to do with getline, but I don't really know how to debug perl extensions. I could write a normal executable and debug that I suppose.

-- ParBohrarper - 14 Aug 2007

Thanks for giving it a try! Could the issue be to do with line endings? CRLF? As I said above, it should be trivial to write a replacement for getline that works on all platforms if it is. I wouldn't have used it in the first place if I had realised it was going to cause issues!

-- CrawfordCurrie - 14 Aug 2007

I changed to linking to libgw32c instead of copying the code, but it still doesn't work. My error: C:\src\native_search>perl test.pl -i -l NativeTWikiSearch test.pl Makefile.PL NativeTWikiSearch.xs Free to wrong pool 272cb8 not 80175 at test.pl line 8. It seems to indicate something fishy with malloc/free (and possibly threads/library boundaries?). It doesn't crash when there are no matches if that is any clue.

-- ParBohrarper - 14 Aug 2007

Oh, and it works if i run it as a standalone executable instead of a perl extension.

-- ParBohrarper - 14 Aug 2007

I tried using MinGW instead of VC6.0, but I get the same error.

-- ParBohrarper - 14 Aug 2007

Got this compiled and working on Fedora 7

-- PeterStephens - 17 Aug 2007

Debugging perl extensions is IME quite tricky. I tried using gdb and the perl debugger, but tracing through all the data type conversions was difficult, and it sounds like that's where the problems are :-(. I ended up using printf.


I rebuilt it as a Contrib to help make it easier to install (version 4.1.2 onwards) and ported it to Win32 at the same time.

-- CrawfordCurrie - 25 Sep 2007


I stripped mine, with: strip --strip-unneeded --keep-file-symbols file

-- SimonRaven - 27 Sep 2007

Actually tested it, it's fricking fast :). nice work

-- SimonRaven - 27 Sep 2007

Topic attachments
I Attachment History Action Size Date Who Comment
Compressed Zip archivetgz native_search.tgz r1 manage 3.6 K 2007-02-11 - 15:22 CrawfordCurrie Unpack in the 'tools' directory
Edit | Attach | Watch | Print version | History: r27 < r26 < r25 < r24 < r23 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r27 - 2007-09-27 - CrawfordCurrie
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2018 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.