Suppress usage of "ls |" in Search.pm

Summary: The latest TWikiAlphaRelease does not use piped commands for search anymore. Topic name search is now done natively in Perl instead of an external ls *.txt | grep ... command. TWiki.org's TWiki is updated as well.

-- PeterThoeny - 01 Nov 2003

The search is using extern commands for ls and grep. The problem is that running TWiki on Win32 with PERL "-wT" cause pipe fail.

• Is using such an external command for dir list needed here ?

 [lib/TWiki/Search.pm] [Line 214] :    $cmd = "$TWiki::lsCmd %FILES% | %GREP% %SWITCHES% -- $TWiki::cmdQuote%TOKEN%$TWiki::cmdQuote";  

• My not to use readdir like in Store.pm ?

In fact, the problem is the pipe, I agree with the external grep but TOPIC list could be get from readdir, like in other part of TWiki :

    opendir( DIR, "$sDir" );
    my @TopicList = grep /\.txt$/, readdir DIR;
    closedir DIR;

Ok, no so easy to change Search.pm, but, if you agree with me, I could try to change it. Note that Impact would be restricted to topic list in current WEB.

-- PatrickNomblot - 17 Jun 2003

Getting rid of lsCmd is the right approach, it makes the code cleaner. Only Search.pm depends on it. If you produce a patch we will take it into the core code. It is a little complex since the grep switches need to be done conditionally for external grep and Perl internal grep.

This fix is needed for TWikiMiniWebServer

-- PeterThoeny - 17 Jun 2003

I attached in this Topic the modified Search.pm to suppress use of "ls" and "|". I tried to do it to minimize the impact on grep in TOPIC (no change in fact) and did it so to keep the high level of code quality (I hope). Thanks to check if this code is good for you, I propose to put it in next TWiki release when ok and suppress ls.cmd in TWiki.cfg.

-- PatrickNomblot - 20 Jun 2003

I have just uploaded the last Search.pm : with this last version, TWiki can also search in rcs files too. Simply say scope="rcs" (rather than scope="text") and grep will look also in rcs fileswhich makes the MyEditedTopics working fine !

You will notice that this choice of scope="rcs" keep Search.pm compatible !

: Note that Win32 grep.exe obtained from http://unxutils.sourceforge.net will search into *.txt* rather that what we tell it to do : *.txt, so it allways grep in txt,v files, whichs is wrong in most cases !

I hope the TWiki team will appreciate my changes on Search.pm, I'have tried to made them in the respect of TWiki.

-- PatrickNomblot - 24 Jun 2003

I have not looked into details, but it looks like

@tmpList = grep /.*$token.*\.txt$/, readdir DIR;

does not allow "pattern$" or "^pattern" regex searches. Better to remove the extension (with map) before grep'ing the pattern.

On $theScope eq "rcs", I would prefer a term that is not implementation specific. RCS could be replaced by CVS or a versioned database backend. Not sure what term would be intuitive, "history", "rev", or?

-- PeterThoeny - 24 Jun 2003

Thanks

Works on Unix and saves a few forks. Plus hopefully resolves the issues with my Windows install, where everything works, except forking ls...

Yes

^pattern for names does work ... regardless of the regular expression checkbox!?

No

pattern$ does not work. Never did, as mentioned in DynamicEditTopicTemplate. Try pattern.txt$

Yes

history instead of rcs is a good idea, because, you know, erm ... CoolURIsDontChange

-- PeterKlausner - 24 Jun 2003

topic="rcs" was not a good idea ... sorry ! I have changed it to topic="history" Thank you for the idea !

Search in TOPIC seems to have the same behaviour than original one. Anyway, I like the Peter's idea, the last Search.pm uses map to remove .txt from topic names before grep. I think that it should be possible to have pattern$ working know (but it still does not).

-- PatrickNomblot - 25 Jun 2003

I think it would be great to put this little evolution of Search.pm in CairoRelease together with MyEditedTopics. It runs very well on our system since 24 June now, whith no problems. Note I think that is also important to keep participation of people outside CoreTeam

Many thank's !

-- PatrickNomblot - 02 Jul 2003

I've changed the status of this to FeatureHack - I don't mean that it is one but we still don't have a FeaturePatch? category, without which your hard work is likely to get lost. I don't want you to get demotivated!

PleaseCreateNewCategories holds the discussion for the introduction of this process enhancer.

-- MartinCleaver - 16 Jul 2003

I am now working on this feature among ExcludeWebTopicsFromSearch, SearchTopicNameAndTopicText, ArgumentListIsTooLongForSearch, InlineSearchArgListTooLong.

-- PeterThoeny - 29 Oct 2003

This is now implemented, among with SearchTopicNameAndTopicText, ExcludeWebTopicsFromSearch, ArgumentListIsTooLongForSearch, InlineSearchArgListTooLong, SearchWebHasTooManyParameters, SiteMapIsSlow. A first step of an overhaul of the lib/TWiki/Search.pm module is also done. Therefore I did not apply PatrickNomblot's patch.

-- PeterThoeny - 01 Nov 2003