Bug: SiteMap is slow
The new dynamically-created SiteMap
is very nice, but somewhat slow - according to MartyBacke
's comment of 4 Feb 2003 on FeedbackOnKnownIssuesOfTWiki01Feb2003
, time to access WebHome
goes from 3 to 6 seconds, which was almost bad enough for him to back out the Feb 2003 release. Similar comments on TwikiFreeBsdPerformance
is commonly accessed, this creates an impression that TWiki is very slow even though it really isn't on most pages.
and compare speed to (say) WebStatistics
Various including ModPerl
- see pages linked above.
| TWiki version:
|| 01 Feb 2003
| TWiki plugins:
| Server OS:
| Web server:
| Perl version:
| Client OS:
| Web Browser:
- 01 May 2003
Not sure of the best fix here - the old SiteMap
was harder to edit, I suppose, but much faster to access. Use of the CacheAddOn
would solve this but is not a generally desirable approach.
- ColasNahaboo : I think this shows the limit of the current approach of TWiki: too much runtime interpretation. I would encourage the CoreTeam to move towards more things being "compiled" when saving a page, to alleviate what is to be done when reading pages. A web page should definitely not load in more than 2 seconds (less than 0.5s is higly desirable). For instance, pre-compiling the sitemap as static data on each WebPreferences save.
This is what I tried to achieve with KoalaSkin, see http://koala.ilog.fr/wiki/bin/oops/Koala?template=sitemap
-- ColasNahaboo - 01 May 2003
Good point, some sort of 'save in compiled form' would be useful here. As a workaround, it's always possible to write a simpler hard-coded version of SiteMap
and embed it into the WebHome
pages - personally, I find the full SiteMap
rather too large and complex for the intranet TWiki I use.
- 02 May 2003
The problem is as follows: The SiteMap gets created dynamically, it lists all topics that have a
SITEMAPLIST = on
setting in the WebPreferences
. The current TWiki does not allow to search for a topic name AND
text within a topic, only one or the other. That is, the SiteMap currently searches all topics in all webs for a
"\* Set SITEMAPLIST \= on"
regex text, which is not scalable.
The correct solution is a SearchTopicNameAndTopicText
feature to search only all WebPreferences
topics for the
setting. This will scale nicely, also if you have 100 webs.
- 03 May 2003
In response to a user complaining of slow WebHome
page loads ( TwikiFreeBsdPerformance
) I just browsed a few WebHome
pages at twiki.org and was amazed at how slow they are now. If this is really due to the site maps, then I'd suggest we don't have site maps on the WebHome
pages at all by default. Even if it wasn't slow, that information is really not necessary there IMHO - does anyone actually find it essential to have the full site map on each web's WebHome
page? I think being able to see a list of the webs at the top is all you really need, and you can provide a link to a Site Map page for those who really want to see the details of other webs (like KoalaSkin
Of course users can customise their individual install to remove the sitemap code from their WebHome
pages, but when a new user starts they are likely to be rather tentative and reluctant to make a change like that until they understand their way around better. I know it took me a while to figure out which parts of the install were required and which were customizable, so I didn't remove anything from the default WebHome
pages for a long time.
- 17 May 2003
Martin, that is an interesting observation. In fact, a few month ago we removed the sitemap at work from the WebHome topics except for the Main.WebHome topic. How about doing that with the TWiki distribution, and also add a SiteMap link to the top navigation?
- 17 May 2003
I think that would be a good solution. Keeping the SiteMap on Main is reasonable as those users who do go there are often looking for more global information anyway. The main thing is to remove the SiteMap from the template used to create the new webs where users will spend most of their time.
- 17 May 2003
I point out that the way TWiki processes settings is extremely inefficient. One of the emasurements I made but didn't report there was of WebHome
. For a 'regular' topic, the processing of settings takes about 25% of the time.
Because of design problems, latency and repeated scanning, it takes about 90% of clock time
and over 50% of CPU time to process SiteMap
As I've pointed out in that thread, a number of performance enhancements are possible of the
- "Get" the settings for each topic seperately and cache them
- Store the settings in a fast database outside of the topic body
- Separate access control from the other settings
The flow of control in processing SiteMap
requires looking though the settings in each webs WebPreferences topic for three - and only those three - items. However scanning rsach of
those topics is O(filesize), compared to a per-topic hash database which is independent of the size
or number of settings, or a linear (e.g. CSV
) per-topic store which is O(number of settings).
In addition, to scan that WebPreferences topic, its access has to be verified.
The code that does this is highly redundant. With settings stored in the topic and
access control as settings, caching the settings on a per topic basis would reduce
the re-scanning. Even better would be storing access control outside the topic (the
site and web preferences as well as the topic in question) in, for example, a DB_HASH
for fast lookup.
I've been experimenting with this and the results are very positive.
In an ideal world, TWiki would be more OO and the storage could be done with a
plug-in that over-rides the "base class method". Its all in the book.
I'm writing this up and will be posting it to Codev shortly. A "fix by design" rather
than a "fix by patching the code".
- 17 May 2003
feature fixed this issue. For your own site, get the latest TWikiAlphaRelease
and add a
parameter to the formatted SEARCH in your TWiki.SiteMap
- 01 Nov 2003
I made some performance tuning: In search, in case topic parameter has no wildcards, that topic parameter is used for topic list instead of getting all topics first and limiting list to topic parameter. SiteMap
page access time:
12 sec, before
4 sec, after adding
2 sec, after performanc tuning
- 02 Nov 2003
nicer - even though TWiki.org is quite fast today, the sitemap here is really speedy. Thanks!
- 06 Nov 2003