performance1Add my vote for this tag scalability1Add my vote for this tag create new tag
, view all tags

How Many Topics and Webs Can TWiki Support?

Purpose: Accumulate user experiences on the design and performance aspects related to no. of topics and webs. A best practices section in standard documentation would help.

Hundreds of webs?

Can TWiki sustain hundreds of webs? This could be a need in enterprise deployment, where every activity might require its own web. (Note: Some of the plugins will allow multiple projects in same web; for e.g. XpTrackerPlugin.)

Is such a usage suggested? Which parts of twiki usage will introduce performance problems?

grep on readdir and found that it is used in following contexts:

  • getSubWebs(), getAllWebs() function in Store.pm. getPublicWebList() in TWiki.pm.
    • Used for WEBLIST (and TOPICLIST) macros.
  • SEARCHes, when webs=all is given as parameter. (Is it a default?) Used in TWiki/Search.pm.
  • mailnotify, Statistics and some other scripts are run once in a while, and they scan all the webs.
  • Probably variety of Plugin dependencies.

So in essence, one could control the default meaning of "all" to be a local list defined within WEBPREFERENCES, and make sure that templates know what they do.

There could be performance hit due to filesystem's speed to handle hundreds of directories (and files).

Thousands of Topics?

What should be upper limit before we see significant slowdowns? So far, the performance seems to be affected by too many calls to read topics, rather than no. of topics itself.

To document the use of topic lists:

  • getTopicList in TWiki/Func.pm, so key function used by all other plugins etc.
  • _getTopicList in lib/TWiki/Search.pm, eventually consumed in SEARCH. Possibly consumed in several other places
  • getTopicNames() defined in TWiki/Store.pm, used in TWiki.pm (TOPICLIST etc.), Form.pm (for resolving $users?),
  • copyWebTopics() in TWiki/UI.pm

In essence, there are very few occurances of the same.

Other uses of this analysis

In context of using Stores (Subversion etc.), it will be useful to provide multiple twiki interfaces on same store. (Use cases such as mobility and replication at multiple sites). In that case, synchronization (and merging) will have to be supported.

This analysis shows that the list of webs and topics can easily be updated because of very few calls. (That leaves merging of topics when edits happen from multiple location, and that is easier problem to solve.)

-- VinodKulkarni - 10 Aug 2004

It will be nice if this can be included in Cairo documentation (ManagingWebs) - assuming that the deployments indeed support the main assertion.

A friend has partial implementation of IMAP as backend store with RCS API emulation (i.e. supports versions etc. by appropriate subject lines). The main use is to allow occassionally connected laptop installations to sync with central twiki. This is still work in progress; interested people can contact me.

-- VinodKulkarni - 10 Aug 2004

There is a balance between number of webs and size of each web. In general I tend to recommend fewer webs with lots of topics.

Argument for large webs:

Counter argument:

  • WebNotify notification is not fine graned
  • Namespace pollution

At my workplace we have now near 50K topics in around 100 webs. The active webs are usually large webs with more then 1000 topics; we have many stale webs, those are usually very small (and many have been retired just recently).

Technically there are limits to the number of topics within a web. This depends mainly on how many files the file system can handle efficiently in one directory. On Unix there is usually no big performance degradation if you have less then 10K files, e.g. 5K topics in a web if you have the RCS files in the web directory.

I have not yet hit a limit on the number of webs in a TWiki site. I suspect there is more a practical limit then a technical one.

-- PeterThoeny - 11 Aug 2004

I did a small experiment: Created 1000 webs (replicating _default). It is interesting to note that:

  • There seems to be no difference in response time at all. (I didn't even bother to use ab for performance measurements.)
  • Some topics - most notable WebHome which (by default) list the information from all the webs will be very very slow. These topics will have to be manually identified and changed.

Other than that, the management aspects might suffer: You have centralized groups, users, %WEBLIST% and similar variables. To overcome these limitations, we could use standard approach:

  • Make users and groups be managed at web level i.e. be able to define new groups at web level, be able to override some groups and so on, exactly like other preferences.
  • Allow one web to "borrow" the configuration from another web. The idea here is that you setup some dummy webs only for purpose of configuration (such as common groups, users, skin, ...). And use this using a simple Include or similar syntax.

This might be useful for ISPs too.

- VinodKulkarni - 26 Aug 2004

The KoalaSkin approach to this was to avoid using %WEBLIST% and pre-compile all references to other webs. This way we could handle 1000s of webs without problem.

Also we moved web administration out of the web themselves but everything is listed on one page, see http://wikix.ilog.fr/wiki/bin/view/TWiki/KoalaSkinWebList with some global settings inherited. This make managing webs a breeze. Just recompile after changes, it takes 5 minutes for 100 webs, but it is not a problem.

Basically I would advise going to a "compile mode" where we should precompute things after major changes (adding a web or a plugin) to keep normal page view overhead to a strinc minimum, or we wont scale. And as a colloraly, try to precompute a lot of things on topic saves, and store them in a cache (topic parent/children info for instance, webchanges list of topics...) to avoid having to do %SEARCHes for common tasks.

-- ColasNahaboo - 13 Jan 2005

Edit | Attach | Watch | Print version | History: r5 < r4 < r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r5 - 2005-01-13 - ColasNahaboo
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2018 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.