Question

For the past month my Twiki installation has been running very slowly. All page views, edits, saves, attaches, etc take up to one minute.

My sys admin says this about the debug.txt file:

The file starts on March 20 at 00:44, and ends (currently) at 03:12 today. It's 57,990 lines of this:

| 26 Apr 2008 - 03:12 | WebStatistics: Bad logfile line | 26 Apr
2008 - 01:44 | TWikiGuest | changes | Sandbox |  gsa | 132.239.180.72 |

Any ideas?

Environment

-- CarlMcKinney - 26 Apr 2008

Answer

If you answer a question - or someone answered one of your questions - please remember to edit the page and set the status to answered. The status selector is below the edit box.

gsa looks like you use Google Appliance. We have blocked this crawler because it would lock up all resources. But you may a bit less drastic approach with a robots.txt file.

-- ArthurClemens - 26 Apr 2008

Are you on a load-balanced setup? If so you may get corrupted log entries. They can safely be ignored though.

There are many reasons for a slow TWiki: Slow LDAP authentication, BlackListPlugin interference, including a page in the sidebar that has a slow SEARCH, high load by spiders, high load by users, etc. See TWikiScalability for some ideas.

I would start by monitoring the system:

• Run top
• Scan Apache and TWiki log files for IP addresses that access TWiki most often. Here is one command I use frequently on twiki.org to identify agressive spiders:
  alias ti5='grep " view " /home/twiki/data/log`date +%Y%m`.txt | tail -500 | sed "s/ \|$//; s/.* //" | sort |uniq -c|sort -r | head -20'

-- PeterThoeny - 26 Apr 2008

Blocking crawlers seems to have solved the problem, but I'm still curious why they were showing up in the error log?

-- CarlMcKinney - 11 May 2008

Closing this question after more than 30 days of inactivity. Please feel free to re-open if necessary.

-- PeterThoeny - 23 Jul 2008