SID-02445: How many Topics can be supported by a Web

Status:	Answered	TWiki version:		Perl version:
Category:	TopicClassificationAddOn	Server OS:		Last update:	5 years ago

Hi,

I would like to know how many number of topics can be supported by a Web. We have a web page which has more than 5 lakhs number of topics and its extremely slow. so, to troubleshoot further want to know the number of topics it can be supported

-- Sukanya Reddy - 2020-05-22

Discussion and Answer

There is no hard limit on the number of topics per web. How many topics do you have in the big web?

The biggest web on TWiki.org currently has 76k topics, which means 152k files in twiki/data/Main. The system gets noticeably slower with that many files in a directory.

Ways to mitigate slow webs with many topics:

• Split up the big web into multiple webs:
  • Move topics from twiki/data/BigWeb/ to twiki/data/NewWeb1/, twiki/data/NewWeb2/, etc.
  • Move attachments from twiki/pub/BigWeb/ to twiki/pub/NewWeb1/, twiki/pub/NewWeb2/, etc.
• Create and use multiple new webs, and keep the big web as a (slow) archive:
  • Copy over frequently used topics to one of the new webs
  • In new webs, add links to less frequently used topics in the big web, such as: BigWeb.SomeLesserUsedTopic
• Use a fast disk, e.g. SSD
• Use a file system that is fast with many small files, e.g. ReiserFS
• Use plenty of RAM to cache file I/O; configure your server to serve the twiki/data out of RAM, see https://serverfault.com/questions/43383/caching-preloading-files-on-linux-into-ram

-- Peter Thoeny - 2020-05-23

Thanks for the response peter, please find the below details:

so under Tsim web we have 5,37,936 files and this web takes long time to open. we are trying to crawl the items but it always fails.we are planning to migrate the data from this web.

Is it possible to move the topics from this web to some other web?

Thanks, Sukanya Reddy

-- Sukanya Reddy - 2020-05-25

Oh, that's 250k+ topics.

Yes, with this many topics it makes sense to split up the big web into multiple smaller webs. I updated my previous answer accordingly. Cross-links will be broken though. Therefore you can try to keep current content, and move stale content out to the smaller webs. For example, if you have stale ProjectFoo* topics, move out twiki/data/BigWeb/ProjectFoo* and twiki/pub/BigWeb/ProjectFoo*.

-- Peter Thoeny - 2020-05-25

sure Peter, Thanks for the response

-- Sukanya Reddy - 2020-05-26

Hi Peter,

Since the topics are huge, is there any way to find who modified the files last. From front end it will be difficult as we have huge number of topics, do we have any command or script to perform the same.

Thanks, Sukanya Reddy

-- Sukanya Reddy - 2020-06-03

You can do that on the shell level. Here is an example:

[twiki data]# pwd
/var/www/twiki/data
[twiki data]# grep 'META:TOPICINFO.*SukanyaReddy' Support/*txt|sed 's/:.*//'
Support/SID-02446.txt
Support/SID-02447.txt
Support/SID-02449.txt
[twiki data]# 

-- Peter Thoeny - 2020-06-04

If you have too many topics in the web you can first generate a file that contains the names of all topics, followed by grep search in that file. To generate the file:

[twiki data]# find Support/ -name '*.txt' | xargs egrep '[%]META:TOPICINFO' | sort | sed 's/:.*author="/  /; s/".*//' > allfiles.txt
[twiki data]# 
[twiki data]# grep SukanyaReddy allfiles.txt
Support/SID-02446.txt  SukanyaReddy
Support/SID-02447.txt  SukanyaReddy
Support/SID-02449.txt  SukanyaReddy
[twiki data]# 

-- Peter Thoeny - 2020-06-04

Thanks Peter.

Instead of moving the topics users suggested the below for performance issue:

https://twiki.org/cgi-bin/view/Codev/TWikiCache

do you think, this will help? I could see it has lot of dependencies which has to be installed

Can we just add anything to the webpreferences of that particular topic to enable the cache which improves the performance.

Also i observed only parent level web takes forever to open.

But topic with in web open in 10 seconds

why this is happening?

-- Sukanya Reddy - 2020-06-10

-- Sukanya Reddy - 2020-06-11

If you don't have user-specific content you can install the TWikiGuestCacheAddOn. We run it on all public TWiki.org TWiki pages for many years.

-- Peter Thoeny - 2020-06-15

Hi Peter,

Thanks for your suggestion, but for TWikiGuestCacheAddOn, the site should be opened for all, so we cant take this into consideration.

How to make a TWiki page static? So that it does not render every time it is loaded, rather it is just one static page which would require manual editing based on user requirements. for this i could see a article https://twiki.org/cgi-bin/view/Plugins/PublishWebPlugin , can we try this?

Also can you please provide the possibilities for below:

1) Create Static page in TWiki for that particular Web. 2) Set Static page as home page for a dummy TWiki web 3) Roll back and make dummy TWiki’s original home page as home page again - How can we do this, i dont see any particular article.

Thanks, Sukanya Reddy

-- Sukanya Reddy - 2020-06-26

Hi Peter,

can you please help me with the requested details?

Thanks,

Sukanya Reddy

-- Sukanya Reddy - 2020-07-07

The PublishWebPlugin is primarily intended to generate static pages for a website, e.g. use TWiki as a web-authoring platform. Example website powered by this plugin: http://www.otopy.com/. You could use it on the intranet though.

I do not understand your 1), 2), 3) scenario.

If you have slow pages with dynamic content such as SEARCH, you can use the VarCachePlugin to speed up the page load.

-- Peter Thoeny - 2020-07-07

Hi Peter,

My apologies, my query wasn’t phrased correctly, this is a two part question.

Here is what i mean.

1) How can we create a static\html page in TWiki and set that as home page (webhome) for an existing TWiki web? 2)And in case if we require to roll back the same, how can we switch back to the previous home page (which was the actual home page of the TWiki web)?

Thanks, Sukanya Reddy

-- Sukanya Reddy - 2020-07-09

You could use wget or the like to generate a static HTML page from a WebHome topic.

Then you could add an Apache rewrite rule, such as:

    RewriteEngine On
    RewriteRule ^webxhome\.html$ /cgi-bin/view/WebX/WebHome

For a more sophisticated solution you'd need to write a custom plugin.

-- Peter Thoeny - 2020-07-11

If you answer a question - or someone answered one of your questions - please remember to edit the page and set the status to answered. The status selector is below the edit box.