create new tag
, view all tags

Search Topic Name AND Topic Text

%SEARCH{}% has a new topic parameter. Specification, documented in TWikiVariables:

Parameter: Description: Default:
Limit search to topics: A topic, a topic with asterisk wildcards, or a list of topics separated by comma. All topics in a web

This feature is useful to speed up search in some cases. For example, if you want to search for a flag in the WebPreferences topics in all webs, you can limit the search to just that one topic in each web. This solves, among others, the SiteMapIsSlow issue. Use FormattedSearchFormTesting to try it out.

Don't confuse this with SearchScopeForTopicAndText, which is related, but is a proposed enhancement to search for the same string in the topic name AND the topic text.

-- PeterThoeny - 01 Oct 2003


TWiki should be able to search for topic names AND topic text. This improves the performance for some applications, e.g. it would solve the SiteMapIsSlow problem.

The implementation is not too difficult, it is basically a scope="topic" search, followed by a scope="text" search. CVS:lib/TWiki/Search.pm can be enhanced to understand one more parameter, supplied to the %SEARCH{...}% variable.

What is the right syntax for that? It needs to be backwards compatible, and should not cause confusion with existing parameters. We already have a web parameter to specify web(s) to search, as listed in TWikiVariables. How about a topic parameter?

%SEARCH{ "\* Set SITEMAPLIST \= on" regex="on" web="all" topic="WebPreferences" }%


  • Should the topic parameter be a literal string or can it be a regex?
  • Should it be the same (regex or not) search for text and topic name?
  • Isn't it confusing to have a scope="topic" search one one side and a topic parameter on the other side? Which one is stronger?

-- PeterThoeny - 03 May 2003


In the %SEARCH construct, the scope of topics searched is given actually by the web parameter. We could thus imagine to define "virtual webs" (like virtual folders of some mailers) by a new construct %SEARCHSCOPE
%SEARCHSCOPE{ name-of-virtual-web-to-define body-of-the-search-construct-to-select-the-contents }%


   %SEARCHSCOPE{webpreferences "WebPreferences" scope="topic" web="all" }%
   %SEARCH{ "\* Set SITEMAPLIST \= on" regex="on" web="webpreferences" }%

Of course SEARCHSCOPE can take virtual webs too to cascade searches.

One could thus imagine future operations on this "virtual webs" that could be handled as sets:

  • union: just specify the list of them in the web parameters, as in web="webpreferences webnotifies"
  • intersection: just re-do a search in the scope
    %SEARCHSCOPE{webpreferences "WebPreferences" scope="topic" web="all" }%
    %SEARCHSCOPE{webprefswithnosearch "\* Set NOSEARCHALL \= on" scope="text" web="webpreferences" }%
  • negation: for efficiency, maybe introduce a new parameter "nomatch", that if set to "on" will search for the pages not matched by the %SEARCH expression. With this, I think the expression power of teh system will be complete. e.g:
    %SEARCHSCOPE{webpreferences "WebPreferences" scope="topic" web="all" }%
    %SEARCHSCOPE{webprefssearchable "\* Set NOSEARCHALL \= on" nomatch="on" scope="text" web="webpreferences" }%

-- ColasNahaboo - 03 May 2003

Ping. I would like to get this enhancement implemented. Mainly because it solves the perception issue that TWiki is very slow, based on any WebHome topic being slow because of the SiteMapIsSlow problem.

-- PeterThoeny - 02 Sep 2003

This enhancement is also needed to make MultipleSearchesInSameTopic more flexible, e.g. to be able to do a multiple hit search in one particular topic.

-- PeterThoeny - 29 Sep 2003

SEARCHSCOPE sounds like a nice idea, I could have made good use of the negation capability recently. Something about the syntax confused me. Would it be better to not have the default parameter e.g. change

%SEARCHSCOPE{webpreferences "WebPreferences" scope="topic" web="all" }% to
%SEARCHSCOPE{virtualweb="webpreferences" search="WebPreferences" scope="topic" web="all" }%

If the name of the virtual web is the same as a real web, would it override it?

If we have something like:

   %SEARCHSCOPE{virtualweb="weba" ...}%

   %SEARCH{web="weba" ...}%

Would TopicB be able to use SEARCHSCOPE from TopicA?

-- JohnTalintyre - 30 Sep 2003

The SEARCHSCOPE looks like a powerful feature. Hmm, not sure if we need/want that, it raises the complexity of the interface (for the user) and implementation (for the maintainers).

Virtual webs/topics do make sense if you have multiple queries that have similar webs/topics to search. This can already be done with INCLUDE and INCLUDINGTOPIC. For example, we could define a topic with a generic search/filter for all TopicClassification features / bugs, and include that generic topic in all features / bugs index pages. The classification search is done with an INCLUDINGTOPIC. (sounds complex, would be easier to explain on a live example)

I would suggest to go for an easier (and stateless) topic=".." parameter in SEARCH. In the docs we can state that this parameter takes only effect for a scope="text" search. The implementation is easy and straight forward.

See proposed syntax at the beginning of this topic.

  • Question: Should we support regular expression for topics? This is not really needed in most cases. We could keep that as an undocumented feature. By the implementation, it actually would be a wildcard search, not a regex search since the external ls is called. For example, to search for all topics starting with Web you would search for Web* instead of ^Web.*.

-- PeterThoeny - 01 Oct 2003

  1. For KoalaSkin users, SEARCHSCOPE would simplify the UI.
  2. I'd like to make the scope="topics,text" on my sites.

-- MartinCleaver - 11 Oct 2003

A complementing feature is proposed in ExcludeWebTopicsFromSearch. Both features should have the same spec for specifying multiple topics. Possible solutions:

  1. Support and document whitespace separated topic list, e.g. topic="Bug* Feature*"
    • smile This is consistent with the current spec of the web parameter
    • frown No regex power
    • indifferent Some extra work to implement
  2. Support and document comma separated topic list, e.g. topic="Bug*, Feature*"
    • smile Easy to understand and use (the web parameter doc could be changed, while the implementation still supports the whitespace separator)
    • frown No regex power
    • indifferent Some extra work to implement
  3. Document only single topic support, e.g. topic="FeatureForm", but support regex
    • smile Easy to implement
    • smile Undocumented regex power
  4. Support and document regex, e.g. topic="(^Bug|^Feature)"
    • frown Not easy to understand
    • smile Regex power

Which way should we go?

-- PeterThoeny - 25 Oct 2003

I am now working on this feature among ExcludeWebTopicsFromSearch, SearchWithNoPipe, ArgumentListIsTooLongForSearch, InlineSearchArgListTooLong.

I have implemented option 2, e.g. shell command meta char expansion like topic="Bug*, Feature*" to make it easy to use and understand.

-- PeterThoeny - 29 Oct 2003

This is now implemented among ExcludeWebTopicsFromSearch, SearchWithNoPipe, ArgumentListIsTooLongForSearch, InlineSearchArgListTooLong, SearchWebHasTooManyParameters, SiteMapIsSlow

Is in TWikiAlphaRelease and at TWiki.org.

-- PeterThoeny - 01 Nov 2003 100%

I love it! Wonderful! Thank you!

Now all we need to do is to be able to have nested searches in the topic and excludetopic paramters. Searching of forms data and such like. That way the 'structural' stuff that doens't fit wild-card expansion can be used. Yes, we can have 'Web*' and '*EditTemplate' but there are going to be many others. I use WebForms and tag such topics and would love to have them excluded from lists, so as to "remove" them from the sight of the ordinary end-user or bring them to the attention of the developers.

-- AntonAylward - 01 Nov 2004

Edit | Attach | Watch | Print version | History: r18 < r17 < r16 < r15 < r14 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r18 - 2006-01-03 - PeterThoeny
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2018 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.