Bug: SiteMap is slow
The new dynamically-created
SiteMap is very nice, but somewhat slow - according to
MartyBacke's comment of 4 Feb 2003 on
FeedbackOnKnownIssuesOfTWiki01Feb2003, time to access
WebHome goes from 3 to 6 seconds, which was almost bad enough for him to back out the Feb 2003 release. Similar comments on
TwikiFreeBsdPerformance as well.
Since
WebHome is commonly accessed, this creates an impression that TWiki is very slow even though it really isn't on most pages.
Test case
Access
WebHome and compare speed to (say)
WebStatistics.
Environment
Various including
ModPerl - see pages linked above.
TWiki version: |
01 Feb 2003 |
TWiki plugins: |
|
Server OS: |
|
Web server: |
|
Perl version: |
|
Client OS: |
|
Web Browser: |
|
--
RichardDonkin - 01 May 2003
Follow up
Not sure of the best fix here - the old
SiteMap was harder to edit, I suppose, but much faster to access. Use of the
CacheAddOn would solve this but is not a generally desirable approach.
- ColasNahaboo : I think this shows the limit of the current approach of TWiki: too much runtime interpretation. I would encourage the CoreTeam to move towards more things being "compiled" when saving a page, to alleviate what is to be done when reading pages. A web page should definitely not load in more than 2 seconds (less than 0.5s is higly desirable). For instance, pre-compiling the sitemap as static data on each WebPreferences save.
This is what I tried to achieve with KoalaSkin, see http://koala.ilog.fr/wiki/bin/oops/Koala?template=sitemap
-- ColasNahaboo - 01 May 2003
Good point, some sort of 'save in compiled form' would be useful here. As a workaround, it's always possible to write a simpler hard-coded version of
SiteMap and embed it into the
WebHome pages - personally, I find the full
SiteMap rather too large and complex for the intranet TWiki I use.
--
RichardDonkin - 02 May 2003
The problem is as follows: The SiteMap gets created dynamically, it lists all topics that have a
SITEMAPLIST = on
setting in the
WebPreferences. The current TWiki does not allow to search for a topic name
AND text within a topic, only one or the other. That is, the SiteMap currently searches all topics in all webs for a
"\* Set SITEMAPLIST \= on"
regex text, which is not scalable.
The correct solution is a
SearchTopicNameAndTopicText feature to search only all
WebPreferences topics for the
SITEMAP
setting. This will scale nicely, also if you have 100 webs.
--
PeterThoeny - 03 May 2003
In response to a user complaining of slow
WebHome page loads (
TwikiFreeBsdPerformance ) I just browsed a few
WebHome pages at twiki.org and was amazed at how slow they are now. If this is really due to the site maps, then I'd suggest we don't have site maps on the
WebHome pages at all by default. Even if it wasn't slow, that information is really not necessary there IMHO - does anyone actually find it essential to have the full site map on each web's
WebHome page? I think being able to see a list of the webs at the top is all you really need, and you can provide a link to a Site Map page for those who really want to see the details of other webs (like
KoalaSkin does.)
Of course users can customise their individual install to remove the sitemap code from their
WebHome pages, but when a new user starts they are likely to be rather tentative and reluctant to make a change like that until they understand their way around better. I know it took me a while to figure out which parts of the install were required and which were customizable, so I didn't remove anything from the default
WebHome pages for a long time.
--
MartinWatt - 17 May 2003
Martin, that is an interesting observation. In fact, a few month ago we removed the sitemap at work from the WebHome topics except for the Main.WebHome topic. How about doing that with the TWiki distribution, and also add a SiteMap link to the top navigation?
--
PeterThoeny - 17 May 2003
I think that would be a good solution. Keeping the SiteMap on Main is reasonable as those users who do go there are often looking for more global information anyway. The main thing is to remove the SiteMap from the template used to create the new webs where users will spend most of their time.
--
MartinWatt - 17 May 2003
In
MetaDataSucksMeasurement I point out that the way TWiki processes settings is extremely inefficient. One of the emasurements I made but didn't report there was of
WebHome with the
embedded
SiteMap. For a 'regular' topic, the processing of settings takes about 25% of the time.
Because of design problems, latency and repeated scanning, it takes about 90% of clock time
and over 50% of CPU time to process
SiteMap.
As I've pointed out in that thread, a number of performance enhancements are possible of the
code in
lib/TWiki/Prefs.pm
is redesigned.
- "Get" the settings for each topic seperately and cache them
- Store the settings in a fast database outside of the topic body
- Separate access control from the other settings
The flow of control in processing
SiteMap requires looking though the settings in each webs WebPreferences topic for three - and only those three - items. However scanning rsach of
those topics is O(filesize), compared to a per-topic hash database which is independent of the size
or number of settings, or a linear (e.g.
CSV) per-topic store which is O(number of settings).
In addition, to scan that WebPreferences topic, its access has to be verified.
The code that does this is highly redundant. With settings stored in the topic and
access control as settings, caching the settings on a per topic basis would reduce
the re-scanning. Even better would be storing access control outside the topic (the
site and web preferences as well as the topic in question) in, for example, a DB_HASH
for fast lookup.
I've been experimenting with this and the results are very positive.
In an ideal world, TWiki would be more OO and the storage could be done with a
plug-in that over-rides the "base class method". Its all in the book.
I'm writing this up and will be posting it to Codev shortly. A "fix by design" rather
than a "fix by patching the code".
--
AntonAylward - 17 May 2003
Fix record
The
SearchTopicNameAndTopicText feature fixed this issue. For your own site, get the latest
TWikiAlphaRelease and add a
topic="%WEBPREFSTOPIC%"
parameter to the formatted SEARCH in your
TWiki.SiteMap.
--
PeterThoeny - 01 Nov 2003
I made some performance tuning: In search, in case topic parameter has no wildcards, that topic parameter is used for topic list instead of getting all topics first and limiting list to topic parameter.
SiteMap page access time:
-
12 sec
, before topic="SiteMap"
parameter
-
4 sec
, after adding topic="SiteMap"
parameter
-
2 sec
, after performanc tuning
--
PeterThoeny - 02 Nov 2003
That's
much nicer - even though TWiki.org is quite fast today, the sitemap here is really speedy. Thanks!
--
RichardDonkin - 06 Nov 2003