See TWikiOnSourceForgeSlow. Because I was experiencing lots of timeouts on twiki.org (at SourceForge), I decided to start collecting some statistics.

After just two hours or so, I noticed some results that seem to be significant and understandable (in that I think I basically know why they occur). Because I (think I) understand what's going on, I may not continue to collect these statistics, unless I start to notice some different and surprising behavior.

Notes about the Testing Procedure

In this testing, I had only one page request at a time pending for twiki.org. Sometime I may try to see how multiple pending requests impacts the situation. But, this testing was enough to confirm that there is a problem that is not related to throttling to deal with aggressive indexing robots.

To be a little more precise about the testing, I always waited for an operation to complete (or fail) before initiating another request. For example, when a preview is successful, the "save changes" button is available before the preview is completely loaded. I intentionally waited for the preview to completely load before clicking "save changes".

(Interim) Summary

• By far, I get the most timeouts on preview, which is also the script that (AFAIK) requires the most processing effort on the server side (I think that even some of the RCS processing occurs in this script).

• The view script, under normal circumstances, has rare timeouts. Special instances of the view script, like calling the WebSearch page or attempting to view a nonexistent page experience more timeouts. (I often create pages by typing a new page name into the browser's address bar, which results in an attempt to view a non-existent page followed by a chance to create that page.)

• If I repeat an operation shortly after it fails, it succeeds the next time almost consistently (I had only one occurrence where I had to retry preview twice in this short sampling period). It's almost like the operation is started by the first request, and the second request is in time to get the results of the first request. (I don't know whether such a thing is possible on the Internet, or whether results are cached somewhere, or whether an operation (like checking into RCS) goes faster the second time -- oops, wait, that makes some sense -- if preview does include some or all of the RCS checkin, on the second request twiki might see that there are no changes, and respond that much faster. At some point I will let a preview fail, and then retry it like 45 minutes later to see if the second try still succeeds (almost consistently).

• UPDATE: I'm not ready to come to a conclusion, but I walked away for five to ten minutes after pressing preview on this page, when I came back, the preview had failed. When I retried it, it failed again. When I retried it again, "immediately" after the second failure, it succeeded quite quickly.

• FURTHER UPDATE: If the UPDATE above is significant, it means that the success of the second preview immediately after the first is not due to some successful partial RCS processing (I think). My next theory would be that it is due to some magic which I don't understand, but which might be "overcome" if I could increase the timeout on my browser. (I'm assuming the timeout is at my end rather than the server end.) Unfortunately, I've never noticed an option to increase the timeout on the browser. I'll try looking again.

• I don't know if anything can be done in TWiki to improve the situation.
  • I do want to call this to the attention of MartinCleaver and other's considering the Codev.SlimDownFatCgiScripts issue -- maybe something can be done to improve this situation in conjunction with anything done there. UPDATE: I did notice Martin's response there, in essence that nothing done there is likely to address this situation (IIUC). I'd still like to encourage the developers to consider anything they can do to improve the situation. Just as a (probably incorrect) example, if RCS check-in during preview was part of the problem (I don't suspect that at this time), maybe something like caching the preview on the server side might help. (But maybe that would require cookies?) Anyway, I don't really know what I'm talking about, just trying to encourage that we consider ways to improve the situation.)
  • Also, I don't know whether SourceForge uses the mod-perl module, or whether this would help the situation or not.

Statistics

Notes

• view(1) is just an ordinary view
• WebSearch is a view of the WebSearch page
• view(2) is a view of a non-existent page
• results are the result of a search -- in all cases so far, only a few pages were returned -- I suspect I will have more problems with searches that return more pages
• create is the result of clicking create after a search for a nonexistent page

• TWiki version: Current on SourceForge (03 Jul 2002)
• Web server: "
• Server OS: "
• Web browser: NA
• Client OS: NA

-- RandyKramer - 03 Jul 2002

Answer

.