Feature Proposal: Regex Search with Keyword Search

Motivation

When users see a search or query form they expect keyword search because Google is the standard.

TWikiApplications frequently have a query-by-example form or a filter. Behind the scene is a SEARCH with a regular expression search. That is, the application expects that the user enters a regular expression search string.

Users should be able to enter keyword search strings in a query-by-example form or a filter.

Description

The above mentioned usability issue can be solved if the regular expression search string may contain an embedded keyword search string.

An escape sequence turns a literal search on and off.

• Turn on: __STARTKEYWORDSEARCH__
• Turn off: __ENDKEYWORDSEARCH__

Impact and Available Solutions

Documentation

TBD

Examples

(This example needs more details)

Example regex search in a SEARCH:

%SEARCH{ "aaa;bbb;ccc" ... }%

Assuming bbb is this:

META:FIELD.*?name=\"Title\".*?%URLPARAM{"Title"}%

The user is expected to enter a regex for the Title field.

Now we add the escape sequence to turn on literal search just for the Title parameter:

META:FIELD.*?name=\"Title\".*?__STARTKEYWORDSEARCH__%URLPARAM{"Title"}%__ENDKEYWORDSEARCH__

Now the user can enter a literal search string for the Title field.

Here is a concrete example: The TWikiInstallation directory has a filter form to query by Installed at organization, Type of organization, TWiki customization, Type of use and Used by/for. This is done by anding the values of the form received by URL parameters. Actual SEARCH (newlines added for clarity):

%SEARCH{
  search="META:FIELD.*?TopicClassification.*?%TOPIC%\";
META:FIELD.*?TWikiInstalledAt.*?%URLPARAM{"qAt"}%;
META:FIELD.*?TypeOfOrganization.*?%URLPARAM{"qOrg"}%;
META:FIELD.*?TWikiCustomization.*?%URLPARAM{"qCus"}%;
META:FIELD.*?TypeOfUse.*?%URLPARAM{"qUse"}%;
META:FIELD.*?UsedByUsedFor.*?%URLPARAM{"qFor"}%"
  topic="TWikiInstallation*"
  nosearch="on"
  ...
}%

Currently, the user is expected to type a regular expression into the "Installed at" and "Used by/for" query fields. A user friendly search that combines regex search with keyword search allows users to enter plain text into the filters. New SEARCH:

%SEARCH{
  search="META:FIELD.*?TopicClassification.*?%TOPIC%\";
META:FIELD.*?TWikiInstalledAt.*?__STARTKEYWORDSEARCH__%URLPARAM{"qAt"}%__ENDKEYWORDSEARCH__;
META:FIELD.*?TypeOfOrganization.*?__STARTKEYWORDSEARCH__%URLPARAM{"qOrg"}%__ENDKEYWORDSEARCH__;
META:FIELD.*?TWikiCustomization.*?__STARTKEYWORDSEARCH__%URLPARAM{"qCus"}%__ENDKEYWORDSEARCH__;
META:FIELD.*?TypeOfUse.*?__STARTKEYWORDSEARCH__%URLPARAM{"qUse"}%__ENDKEYWORDSEARCH__;
META:FIELD.*?UsedByUsedFor.*?__STARTKEYWORDSEARCH__%URLPARAM{"qFor"}%__ENDKEYWORDSEARCH__"
  topic="TWikiInstallation*"
  nosearch="on"
  ...
}%

Implementation

TBD

-- PeterThoeny - 18 Jan 2005

Discussion:

What are good and intuitive names for the escape sequences?

-- PeterThoeny - 18 Jan 2005

That's a pretty horrible solution, isn't it? Why can't you set the type option in the %SEARCH from a select ?

• Horrible or not, the issue at hand cannot be solved with a type option because you need to have control over which part of the search string is a regex search, and which part is keyword search. -- PeterThoeny - 02 Sep 2006

-- CrawfordCurrie - 14 Apr 2005

I would like to bump this feature request back up. In creating a TWikiApplication, I would like to provide users with a keyword search of a subset of topics I can currently only define by a regex search. Regarding the aesthetics of Peter's solution, I'm not qualified to judge, but it would provide exactly the functionality I need.

I'm not sure what you're suggesting, Crawford. Currently, the type option requires one to choose either/or, and we're saying there's times we need both.

-- LynnwoodBrown - 10 Aug 2006

Agreed, both is needed to build customer focused applications. Several times I could have made my apps more user friendly if this is available: I need a regex search to make my app work, but who wants to burden the users with regular expressions that are part of the composed search! May be now is the time to get some constructive feedback to my proposal?

-- PeterThoeny - 10 Aug 2006

Just bumping this feature request back up to see if we can get some more discussion.

-- LynnwoodBrown - 15 Aug 2006

I am looking forward to see some creative suggestions for the escape sequence to switch between the modes.

-- PeterThoeny - 16 Aug 2006

Ah, I misunderstood what you meant. I'm still not totally sure from the examples above.

You want to be able to get from a keyword search string entered by the user, say eggs bacon bread to a regular expression, say (eggs|bacon|bread), yes?

Why can't you use SpreadSheetPlugin? magic to do that?

-- CrawfordCurrie - 22 Aug 2006

I was taking with Crawford some more about this on irc and he pointed out that it is possible to do essentially what I need by a search within a search to define the topic attribute, like this:

%SEARCH{"keywords" type="keyword" topic="%SEARCH{"[T]opicClassification.*\=.*[F]eatureRequest" type="regex" nonoise="on" format="$topic" separator=", "}%"}%

Rather to my surprise, this works. Crawford created a proof of concept in NestedKeywordSearch. I'm not sure about whether it addresses all use cases you see, Peter, nor am I sure about the performance considerations. But at least it gives me something to work with.

-- LynnwoodBrown - 22 Aug 2006

The nested search can be used in some cases, keep in mind though that this is slow. A nested SEARCH cannot be used for a typical query by example form, such as the filter in the TWikiInstallation directory. I updated above example to be more clear.

Any ideas for an intuitive syntax? If we do not find one I suggest we go ahead with the implementation, using the __STARTKEYWORDSEARCH__ and __ENDKEYWORDSEARCH__ tokens to switch between the modes.

-- PeterThoeny - 02 Sep 2006