Native Search
By default TWiki searches its database by
forking the standard
grep program in a sub-process to search through the text files in the database cache. This is usually pretty fast, but unfortunately in some environments (specifically with
mod_perl) forking a sub-process is very expensive, and another approach is needed.
NativeSearch is a c-code module that can be plugged into TWiki to replace the default searching algorithm. It publishes an
XS interface that allows it to be called from Perl, without having to fork a sub-process. It has comparable performance to the standard forking search in most environments, but is often significantly faster than forked search when TWiki is run under a
CGI accelerator. The effect is most pronounced with mod_perl, but is also there for
SpeedyCGI.
You require shell access to the server to install this module.
You can find packages for download and installation instructions in
NativeSearchContrib. If you are using a TWiki version before 4.1.2, see
below.
To help others, please list below the platforms you were able to install on.
Installing on old TWikis
If you need to install native search on a twiki version prior to 4.1.2, first follow the instructions in
NativeSearchContrib. Now, you need to patch the file
lib/TWiki/Store/RcsFile.pm
. You need to find the
searchInWebContent
method. Delete it (or comment it out if you aren't brave) and use this version instead:
--
Contributors: CrawfordCurrie,
SvenDowideit - 11 Feb 2007
Nice work Crawford and Sven! Totally in line with the
TWikiMission!
--
PeterThoeny - 16 Feb 2007
Very good improvement! But I had problems make it work. I'm using SuSE Linux Professional 9.3 with Perl 5.8.6 and Apache 2 with mod_perl prebuilt by SuSE. Compiling the perl module was fine, but after installing and restarting Apache TWiki does not find any body text.
Apache's error_log showed some strange messages:
httpd2-prefork: unknown option bit(s) set: Success
So I went into cgrep.c and added some debug messages. A good thing is to show the version of pcre with
warn ("DEB: cgrep starts with pcre V %s", pcre_version());
at the beginning of the
cgrep()
C function. And then I found out what was wrong.
When I start the test.pl script, the version of pcre is V 5.0 (the prebuilt version from SuSE). But when I started a search in TWiki, the error_log showed me V 3.9 02-Jan-2002 -- oops.
Then I downloaded the current version 7.0 and compiled and installed it. Same effect: command line test says V 7.0, Apache says V 3.9.
I cannot find any other shared library of libpcre on my machine. It seems to me that the httpd binary is linked against an old pcre and this version is then used also in the perl module.
So I patched the pcre header file to rename the public function names from "pcre_*" to "mypcre_*" and linked it statically to the perl module.
After that hard patch also my Apache says V 7.0 and the search is working fine and fast.
--
StefanScherer - 29 Mar 2007
I found that building
pcre
on OS X 10.4 (Tiger) went ok without a problem, but when running the test above the program gave the error
dyld: lazy symbol binding failed: Symbol not found: _getline
.
Even using
gcc_select 3.3
doesn't solve the problem.
I had to (re)install
CPAN:ExtUtils::MakeMaker because my version did not recognize
INSTALL_BASE
.
--
ArthurClemens - 01 May 2007
Thanks
AaronPeterson for providing a patch for Solaris. Tracked in
Bugs:Item4220
--
PeterThoeny - 06 Jun 2007
I moved the patch to the bug topic. Aaron, please see my comment in
Bugs:Item4220.
--
CrawfordCurrie - 08 Jun 2007
Using
AaronPeterson's Solaris patch I got
NativeSearch working on Mac OS X 10.4.10 (Tiger), Perl v5.8.6.
I had to compile the gigantic
gettext package as the patch required though.. wish there was a smaller substitute.
After making the changes to the Makefile and cgrep.c,
perl test.pl
returned the right result.
--
ClintMarkGono - 25 Jun 2007
Nice one, Clint. Still waiting for someone to pick up the (fairly trivial) challenge of providing a simple function to replace
getline
, which would obviate the need for
gettext
.
--
CrawfordCurrie - 25 Jun 2007
Is there any chance of getting this to work with
ActivePerl in Windows? It would be really nice to not have any dependencies to grep etc. I'm having some issues with intermittent empty searches which I believe is related to cygwin/grep/mod_perl..
--
ParBohrarper - 07 Aug 2007
Hmmmm - what makes you think it doesn't work on Windows? The C-code is really simple, and AFAIK the XS interface in
ActiveState is the same as that used in GNU perl. "All" you need to do is compile it on the windows platform. I never bothered trying, myself.
--
CrawfordCurrie - 08 Aug 2007
I've tried compiling it with Visual Studio 6.0 and nmake. After copying some stuff (getline) from libgw32c and commenting some other stuff out (warn) I managed to compile it. It doesn't work though, it crashes on the test input from above. I suspect it has something to do with getline, but I don't really know how to debug perl extensions. I could write a normal executable and debug that I suppose.
--
ParBohrarper - 14 Aug 2007
Thanks for giving it a try! Could the issue be to do with line endings? CRLF? As I said above, it should be trivial to write a replacement for getline that works on all platforms if it is. I wouldn't have used it in the first place if I had realised it was going to cause issues!
--
CrawfordCurrie - 14 Aug 2007
I changed to linking to libgw32c instead of copying the code, but it still doesn't work. My error:
C:\src\native_search>perl test.pl -i -l
NativeTWikiSearch test.pl Makefile.PL
NativeTWikiSearch.xs
Free to wrong pool 272cb8 not 80175 at test.pl line 8.
It seems to indicate something fishy with malloc/free (and possibly threads/library boundaries?). It doesn't crash when there are no matches if that is any clue.
--
ParBohrarper - 14 Aug 2007
Oh, and it works if i run it as a standalone executable instead of a perl extension.
--
ParBohrarper - 14 Aug 2007
I tried using
MinGW instead of VC6.0, but I get the same error.
--
ParBohrarper - 14 Aug 2007
Got this compiled and working on Fedora 7
--
PeterStephens - 17 Aug 2007
Debugging perl extensions is IME quite tricky. I tried using
gdb
and the perl debugger, but tracing through all the data type conversions was difficult, and it sounds like that's where the problems are :-(. I ended up using
printf
.
CC
I rebuilt it as a Contrib to help make it easier to install (version 4.1.2 onwards) and ported it to Win32 at the same time.
--
CrawfordCurrie - 25 Sep 2007
hey,
I stripped mine, with:
strip --strip-unneeded --keep-file-symbols file
--
SimonRaven - 27 Sep 2007
Actually tested it, it's fricking fast :). nice work
--
SimonRaven - 27 Sep 2007