Feature Proposal: SEARCH - behavour of order and limit
I am a bit worried about the development of the SEACRH feature in Dakar and some of the discussions I have had with key developers the past day on IRC and I feel the SEARCH feature needs to be discussed properly. The topic ends with a proposal for both Dakar and Edinburg.
I have been met with these statements on IRC
"i don't care how it worked in cairo"
"never mind what Cairo did. What should it do logically?"
And I am currently fighting to get an acceptable solution to bug1169.
The important "Protect corporate investment (topic contents) from data corruption and incompatible changes" is so very important.
The time Motorola has invested in installing TWiki and the time we will spend on the upgrade from Cairo to Dakar on the server and the software itself is in total a few staff days. The time spent on the contents
of our TWiki is where the real value lies. There are 100s of staff years behind all the information on the many pages. The different teams have built up many different mini twiki applications and the SEARCH feature is one of the most used features in TWiki and one of the most powerful. We have 100s of topics with searches that shows last modified topics or oldest change requests etc etc. It is essential that the sort order of SEARCH and the way the SEARCH feature "reverse" and "limit" works.
Some call the current Cairo function a bug. And there are for sure some bugs in SEARCH that should be - and has been - fixed. But the function of limit and reverse is for sure not a bug. You may disagree with how they are implemented and defined but you cannot say it is a bug. And I personally find the Cairo implementation logical and very useful. And they have proven to be useful in so many places both on my Motorola TWiki and my Motion TWiki.
SEARCH in Cairo
The way SEARCH works in Cairo is this
- When you sort by topic name the default sequence is alphabetically ascenting. And the reverse="on" behavour is alphabetically descending.
- When have limit="10" you get the 10 first found topics. So normally you get the topics starting from A. And if you reverse the sorting you get topics starting from Z.
- When you sort by an alphabetical field value in a form the default sequence is alphabetically ascenting. And the reverse="on" behavour is alphabetically descending.
- When have limit="10" you get the 10 first found topics. So normally you get the topics with field values starting from A. And if you reverse the sorting you get topics with field values starting from Z.
- When you sort by "modified" or "created" the default sequence is ascenting by date. And the reverse="on" behavour is descending by date. E.g. normal sequence is 1998, 1999, 2003 and reverse="on" is 2003, 1999, 1998. When you sort by modified/created you get the oldest first. And with reverse="on" you get newest first.
- When you have limit="2" you get the 2 first found topics. So normally you get the oldest. And if you reverse the sorting you get the newest. E.g. normal search with limit 2 would be: 1998, 1999. And reverse="on" will give 2003, 1999.
I find the Cairo behavour very natural and very logical and not at all buggy. And it is essential that Dakar continues working the same way or you will fail to protect the huge investment current Cairo users have made in the creation of their many topics.
And the documentation matches the behavour.
| Sort the results of search by the topic names, topic creation time, last modified time, last editor, or named field of TWikiForms. The sorting is done web by web; in case you want to sort across webs, create a formatted table and sort it with TablePlugin's initsort
|| Sort by topic name
| Limit the number of results returned. This is done after sorting if
order is specified
| All results
| Reverse the direction of the search
|| Ascending search
The issue in one sentense: "The limit should return the most relevant topics".
Now this sounds easy. But it is not because we do not all agree on what is relevant.
In a list of bug reports changed you want to see the latest changed. You can do that in Cairo with order="modified" reverse="on" limit="20". That will give you the 20 last modified. The newest modified at the top and descending sort order. I always found this presentation the most logical. But some want the sort order of the latest 20 to be ascending. This you cannot do in Cairo.
In a list of process documents I may want to see the 20 that has been modified the longest time ago. We use that to see what needs to be reviewed and re-validated first. So we use order="modifed" limit="20" (and no reverse). This gives me the 20 topics that most need to be reviewed because noone has touched them for a long time.
So both oldest and newest are needed. But the way Cairo has implemented the order/limit newest is always presented reversed and oldest is always presented oldest first. I personally never found this to be a big problem because to me this is the most logical way to present the two cases to the users. If I want to show the 20 oldest, the most relevent topic is the oldest of the 20. And if I want to present the 20 newest the most recently changed topic is the most relevant.
What is missing?
For sure the feature set in SEARCH could be improved. Here is a list of things that people like Crawford Currie and Sven Dowideit used as arguments for changing SEARCH. And their argument are good and valid. It is the implementation where I think we have to think more carefully so we do not alter existing topics from Cairo, Beijing etc.
Let me list the limitations I can think of.
- You cannot show newest Ascenting.
- You cannot show oldest Descending
- You cannot pick a number of topics in "the middle". You cannot create a search that shows the first 20. And then a new link that shows the next 20.
So maybe as an enhancement
which will be 100% backwards compatible we need an additional SEARCH option called "start" which is the first hit in the list of found topic to show. This start value can be positive which means it counts from the beginning. Or negative which means it counts from the end. Default must be 0 which makes it backwards compatible with Cairo.
Now Crawford and Sven can get what they want.
Show the latest 10 bug reports in ascending order: order="modified" reverse="off" start="-10"
And we get more than that.
Show the latest 10 bug reports before the last 10. order="modified" reverse="off" start="-20" limit="10"
Show all bugs. Browse 20 at a time. Newest first order="modified" reverse="on" start="0" limit="20" and then you browse by having a link that contains a twiki variable which increases by 20. This use is actually so common that the search itself should have the feature of setting a variable that can be used for this purpose.
It is important that the SEARCH in Dakar is fixed to be 100% compatible with Cairo. Wow. CrawfordCurrie did that 15 Dec 2005. Thanks a lot Crawford. -- KennethLavrsen - 15 Dec 2005
The enhancement could be added in Dakar but SEARCH is a complex and slow feature and I fear we rush and risk a bad implementation by doing it in Dakar. It should be an enhancement for Edinburgh.
- 15 Dec 2005
I am concerned about voices like "i don't care how it worked in cairo", especially if it is from key developers. I agree 100% with Kenneth to protect corporate investment (topic contents) from data corruption and incompatible changes, it is in the TWikiMission
and cannot be neglected. I know of two companies that have over 5000 engineers using TWiki, we cannot neglect them and thousands of other companies using TWiki.
If this is a feature needed just for the Bugs web by all means don't do that in Dakar. Defer that to Edinburgh.
- 15 Dec 2005
Thanks Crawford for fixing the compatibility issue
- 15 Dec 2005