Question: Save times out

After editing the page I click save. The browser attempts to save the page, but the server process locks up. ps displays an apache process that is serving the oops page. However, non administrative pages seem to save just fine. This same problem occurred when trying to preview changes on the Main/WebHome as well as TWikiPreferences. The bug seems unrepeatable, yet it has occurred at least twice in the afternoon. After a delay with no save attempts, the problem resolved itself.

Test case

Go to TWikiPreferences. Edit topic Preview Save

I do not know the initial steps that get the systems into this state. There doesn't appear to be any relation except to the

Environment

TWiki version:	TWikiRelease01Feb2003
TWiki plugins:	DefaultPlugin, EmptyPlugin, InterwikiPlugin, StylePlugin
Server OS:	RedHat Linux
Web server:	Apache 2.0.47
Perl version:	5.6.1
Client OS:	Windows XP
Web Browser:	IE 6.0.2800

-- ErikGrimes - 05 Aug 2003

Answer

I moved the bug entry to the Support web, this is a local issue.

I do not have a good answer with the information you gave. Could be a recursion problem caused by a Plugin or a variable setting. Try to disable Plugins, skins and do some TWikiDebugging to narrow down the problem.

-- PeterThoeny - 06 Aug 2003

I have also had this problem. It appears to be a bigger problem on long/big files then smaller ones. Which is why TWikiPreferences would be an issue. I started getting the problem once I put KoalaSkin in. But even if I took koalaskin out, I would still then have the issue.

-- JeffLink - 14 Aug 2003

This is similar to, if not the same as, a problem I have. My server is also on Redhat (9), running Apache 2.0.40-21, perl 5.8.0-88. I also see perl processes stuck. An strace will show the process has issued a write on stdout, and it always seems to be writing the same data (i.e., it is in the same place). I have a bunch of plugins enabled normally, but I disabled them all and found that the hang still occurred, but the strace showed different data in the buffer (not surprisingly). The larger the topic, the greater the chance that it will hang (or maybe it just hangs longer), but none of these topics are super long. If it hangs long enough, I'll eventually get a "Server error" message. For many topics, I've taken to editing them with outside of TWiki, which of course loses the version capability that I so like about it. I can't see anything with top that is hogging the CPU at the time of the hang. If anyone can give me pointers to tracking this down, I'm willing to attempt it. I'm not sure of the best way to approach debugging this beast. I'd very much like to fix this so that I can have a usable TWiki system.

-- DavidBright - 15 Aug 2003

On debugging TWiki:

• make sure your either do not use mod_perl, or do an restart of apache after each change, otherwise you may have some apache processes in a weird state, explaining the randomness
• Try to fix permissions, see CantSaveChangesToWebMenuEdit
• Find the debug statements in TWiki code, and uncomment them and/or set their $Debug var to true if they have one - backup your perl TWiki file before -, then look at the output in data/debug.txt

Do your apache server has some timeout variables set that would abort the process if it takes too long to render a page???

PS: I never saw this behavior, but I am still on apache 1.3

-- ColasNahaboo - 15 Aug 2003

I have tried mod_perl, but had problems with that (after working through the path issues, I found it truncating the topic when I edited, so I went back to mod_cgi; but that's another issue...). I, like you, have a script I run to set permissions. They all look correct to me (i.e., in accordance with the TWikiInstallationGuide). I've turned on the debug just about everywhere and see nothing in the log. Wait, I may have only turned on debug in the plugins, I'll check that. I am pretty sure the server does have timeout variables set to abort if the process takes too long to render a page, but after waiting for 5-10 minutes, I want to abort anyway! I'm reasonably sure this has something to do with running under Apache 2.0, which I know is not recommended; nonetheless, I'd like to figure out how to make it work.

-- DavidBright - 15 Aug 2003

OK, I turned on debug every place within the .pm files in lib/ that I could find and then tried to preview an edit. It got stuck and at that point, I did a ps and strace and got this:

# ps 
apache    2463 30297  1 10:06 ?        00:00:01 /usr/bin/perl -wT /var/www/extranet/twiki/bin/oops template=oopsauth
# strace -p 2463
write(1, "=\"/twiki/bin/view/Plugins/WebHom"..., 2428

At the same time, the debug log showed:

15 Aug 2003 - 10:06 ===== After trim, Headers are:
Content-Type: text/html; charset=ISO-8859-1^M
                                                                                                                                                                               
15 Aug 2003 - 10:06 ===== Final Headers are:
Content-Type: text/html; charset=ISO-8859-1^M
^M

                                                                                                                                                                               

After about 5 minutes, it picked up again and the log continued from the point above:

15 Aug 2003 - 10:11 got useLocale = 0
15 Aug 2003 - 10:11

and so forth.

-- DavidBright - 15 Aug 2003

Yea, I'm running, apache 2 and perl 5.8 too. But when we installed, the notes on those two were that there were problems with a windows install of this, and I'm on redhat 9.

-- JeffLink - 18 Aug 2003

JeffLink posted a related note in KoalaSkinDev asking if this problem may be related to the use of that skin (he noted above that the problem started after he installed that skin). I will note that I do not currently use KoalaSkin, but when I was first getting my TWiki site up, I experimented with both KoalaSkin and GnuSkin and the save issue seemed to come up sometime after that, but I have no direct evidence that this is at all related to either of these skins.

-- DavidBright - 19 Aug 2003

Yea, DavidBright is correct. It appeard sometime after I put KoalaSkin on, but it seemed to "break" sometime after I did that. Meaning it was working fine for a while. And when I revert back to normal, not using KoalaSkin, I still get the problem.

-- JeffLink - 20 Aug 2003

Otherwise, it seems that you noth use redhat 9, so this may be the problem. You (David & Jeff) may want to contact TomTo (see ApacheTwoExperience) to see if he has problems, and if not, what differs from your setup (TWiki version may not be the same, too).

-- ColasNahaboo - 21 Aug 2003

Some Perl 5.8 weirdness on Red Hat 9 was solved by i18n tweaks - I know you are on Perl 5.6.1 but this may be worth a try.

-- RichardDonkin - 08 Sep 2003

I tried the i18n tweaks. But no luck on that. Still getting long saves.

-- JeffLink - 12 Sep 2003

How about trying a complete re-install of TWiki, in case there's something left behind from the Koala skin install? Last resort really - if you keep the old TWiki install tree, a diff of the directories might reveal what's changed.

-- RichardDonkin - 12 Sep 2003

I've encountered this same problem. Some files taking a very long time to save - something like 5 mins. It doesn't happen all the time, and it's not always very long files. I observed the problem on two different systems after installing the Koala skin, one system running RH9.0 the other running RH7.2

I assume a lot of people use Koala skin without a problem, but now I'm wary.

It definitely seems to hang on /usr/bin/perl -wT /home/apache/twiki/bin/oops template=oopsauth and then waits for 300 seconds (the apache timeout in httpd.conf) and then proceeds OK.

I get this on a save and I've also got it on an attach. Both hang on the oops script, and when I did a strace on each, I got the same thing - write(1, "2;color:white;}\n.bg1-Test{backgr"..., 3124. Exactly the same for different pages and one for a save and one for an attach.

Anyone know how to fix this?

-- PeterWhiting - 29 Sep 2003

This seems to be KoalaSkin specific, through the write call is probably from the last print statement in the CVS:bin/oops script - this skin does use a different save script so that could be relevant. You might want to play with the $| variable (see testenv for example) to turn buffering on or off, and to make sure you aren't using mod_perl. ColasNahaboo really needs to investigate this.

-- RichardDonkin - 30 Sep 2003

Could it be a browser issue? KoalaSkin uses its own save script ( savemulti ), so if it was a problem with savemulti it should not persist when using other skins. What browser do you use? Do you use a web proxy?

I never enountered this, but we only heavily use mozilla and IE (I use opera and konqueror to test things, but not long enough to reproduce a random problem)

What optional patches did you apply? (of «5. Apply the following fixes if they are not already on your system:»). Does the browser is also on RH9? Does the problem happens on direct saves or on saves after preview?

-- ColasNahaboo - 01 Oct 2003

I've always used Mozilla. I just tried Konqueror and I get the same problem. No web proxy.

I applied all 5 patches. The browser is also running on RH9. The problem happens whenever I click Save, Quietsave, Checkpoint, Preview or even Cancel. I always get a /usr/bin/perl -wT /home/apache/twiki/bin/oops template=oopsauth hanging until it timesout, but then it completes as normal.

Interestingly, if I go into TwikiPreferences and remove the "koala" from the set SKIN line, then all saving works without any hangs. If I put "koala" back in TwikiPreferences, the hanging starts again.

-- PeterWhiting - 04 Oct 2003

I did a bit more experimenting. I went to my wife's Mac and accessed my twiki site, and using Mozilla on the Mac, I got the same timeout problems. However, when I used IE on the Mac, everything worked fine. No timeouts.

What does that mean?

-- PeterWhiting - 04 Oct 2003

I've tried saving with both koalaskin on and off and they both hang. I've also tried netscape 4.7, msie 6, and opera and they all have extreme save times too. Opera's counter is still going and I just hit the three minute mark. I'm kinda inclined to say its not a browser.

-- JeffLink - 10 Oct 2003

Since I will (this week normally) try to integrate savemulti to the core distrib, I will first do a mini-skin for beta testing this. I will mail you when it is ready (and announce it here), so that we can track the exact issue.

-- ColasNahaboo - 14 Oct 2003

I've just had the same problem. I've started to use the KoalaSkin and also switched authentication on. On saving the registration page, it hung.
I then changed back to the default skin and had the same problem. Finally, i renamed my .htaccess to htaccess (thus switching off authentication) and started a new session. Under TWikiGuest, the topic saved just fine... Will carry on experimenting.

-- AdamWhite - 20 Oct 2003

I've had this problem ever since I upgraded to RedHat 8 and then 9, using the default TWiki theme. It seems to happen on more complicated content pages. I've noticed that if I log into the server (as root) and kill the perl process, the web browser seems to proceed just fine, and the topic is saved (don't know if it gets revisioned in RCS though).

This is not a KoalaSkin specific problem. It is a TWiki or a perl problem.

-- JaredRobinson - 29 Oct 2003

Well I'm glad its not just me anymore.

-- JeffLink - 03 Nov 2003

We just installed the KoalaSkin and have the same hang problems on large files (It will eventually save the file but it will take anywhere from 1 - 4 minutes). When we reverted back to the default skin, we experienced no problems on any pages. I think it would make sense to address this as a KoalaSkin issue first and then maybe a Twiki problem? Many thanks.

-- AliceKang - 01 Dec 2003

I just tried taking KoalaSkin off of our Main web and it didn't change times of saving. I'm going to try to install Twiki in another location on the server with no bells and whistles, see how that goes.

-- JeffLink - 03 Dec 2003

You might want to try running the PerlPtkdb debugger on the save script, to see where it is blocking - this can be used to display the debugger GUI remotely from the server as long as you are using X Windows on Unix/Linux on both client and server. Or you can VNC into a Windows server to see the debugger. Also, the output from truss/strace would be informative and much less hassle to get (just do a shell wrapper around the Perl script).

-- RichardDonkin - 05 Dec 2003

Is there any more progress on this issue? I am encountering a similar problem -- all pages seem to hang on a Save or a Preview, but if I wait long enough, the pages do get saved. I am using Koala Skin, and am on RedHat

-- RahulAnand - 09 Jan 2004

I, too, am experiencing this problem, but on a slightly different setup. I'm running a from-scratch Linux system with Perl 5.8.2 and Apache 2.0.47, the most recent stable TWiki release WITHOUT the Koala skin (I have never used this skin). I've seen this behavior in both the original (default) skin and in the VoidSkin. On both previews and saves, the server seems to hang on a twiki/bin/oops exec, only to resume after a long timeout. Furthermore, no errors are reported in any of the apache logs. I'll be looking into the issue more myself, but I thought I'd drop a line to let you know this is NOT just a Koala issue.

-- DavidJackson - 22 Jan 2004

I can duplicate this problem, reliably, using Apache 2 (recent version - ie including latest security updates) , recent TWiki and the latest version of CGI.pm The problem is not repeatable with small topics, and only happens when topics get over a certain size. This smacks to me of issues in the popen handling in apache2 rather than CGI.pm - the same code/etc works perfectly fine under other servers, and indeed different versions of CGI.pm.

The problem scenario appears to be:

• Large file upload to server
• Large file being served to client
• Large file being written to disk

These smack of issues in the select handling and hand off between CGI app and clients, with the CGI not being polled sufficiently often result in it writing data, and failing to get a response.

Specifically, running strace on the scripts as they run show that the CGI scripts are hanging writing to stdout. This is sensible & default behaviour for CGI scripts, and it's possible this same behaviour could be demonstrated using a simple CGI script.

-- MS - 29 Jan 2004

I have a (rather dirty) fix for this problem. I'm running Red Hat 9 (which has Perl 5.8), with the 01 Feb 2003 release of TWiki and the rather lovely KoalaSkin. I get the problem with a wide variety of browsers, so I think it must be a server issue (it seems to happen most often when saving or previewing large pages). As lots of other people have noted, the problem seems to be caused by a Perl process hanging: /usr/bin/perl -wT /home/apache/twiki/bin/oops template=oopsauth. Killing this process on the server wakes everything up again.

So, having tried everything else, I decided the easiest way to get around this was to write a simple Perl script to automate the process of looking for hung processes and killing them. The script is here. It runs on the server as a daemon, looking every 2 seconds (using ps) for processes with signatures including the string template=oopsauth. If it finds the same process in two successive scans it assumes it's hung and kills it. The script has to run as root to have permissions to kill Apache processes. It keeps a log of processes killed.

This script works fine for me, but use at your own risk! I'm not a Perl whiz by any means and your systems may differ from mine. It's certainly saved me quite a lot of trouble and I hope it's useful to someone.

-- JonBlower - 30 Mar 2004

Couple of points from my testing: If you "cancel" an edit on a small page first, you won't have the problem again for a while, even if you upload big files. Restarting your browser causes this to recur, as does restarting the web server (so perhaps keepalive means it's just for the life of the server?). "MS-" on #twiki says it's an issue with Apache 2.0 / CGI, and that for some people turning on mod_perl fixes it and for everyone downgrading to 1.3 fixes it. I'm downgrading now, so I don't know which if either of these will work.

-- BrettThomas - 10 Apr 2004

a patch resolving apache 2 hanging has been posted at Owiki:ApacheTwoHangs

---

The patch is duplicated here since it's a security fix. Patch for this issue:

Index: setlib.cfg
===================================================================
RCS file: /home/cvs/cvsroot/projects/owiki/system/bin/setlib.cfg,v
retrieving revision 1.1.6.1
retrieving revision 1.1.6.2
diff -u -d -r1.1.6.1 -r1.1.6.2
--- setlib.cfg  5 Feb 2004 01:34:14 -0000       1.1.6.1
+++ setlib.cfg  16 Apr 2004 09:33:44 -0000      1.1.6.2
@@ -18,6 +18,16 @@
 #
 # Used to configure non-standard locations for TWiki and Perl modules.
 
+# ------------ Work around for bugs in Apache 2 hanging when size_output(STDERR)>4096
+# See also http://owiki.org/ApacheTwoHangs
+# You could change this to something like
+# /var/log/owiki/error.log
+#
+$logfile = "/dev/null";
+unless(open(STDERR, ">>$logfile")) {
+   use IO::Handle; STDERR->blocking(0);
+}
+
 
 # -------------- Change these settings if required

See http://owiki.org/ApacheTwoHangs for full discussion of problem and solution.

-- MS

I encountered the same issue with Apache 1.3.31, just started happening since the server admin upgraded the Apache (don't recall the previous version). I don't think this is a Koala specific issue, but is seen more often with KoalaSkin, because of the much larger amount of HTML it outputs. I suspect it is related to processing of large form data, as well as a large amount of HTML output, causing a deadlock in Apache (a known issue). The above patch did not work, but a modification of it did - the single line:

use IO::Handle; STDOUT->blocking(0);

bin/oops, right after use strict;

Note that it is STDOUT not STDERR, and it is not conditional. This fixed the issue for myself and all my users.

Adding this to the setlib.cfg works as well, but causes large files to fail to parse out completely.

-- ChrisJacobson - 27 Jul 2004

Just a note that after upgrading to Apache 2.0.49, the problems I had seen with hanging have disappeared. After a long time of not being able to use TWiki, and many experiments with other wikis in the meantime, I am now back to using TWiki and very happy to be so.

-- DavidBright - 29 Jul 2004

Chris: your solution with STDOUT->block() seemed to work for me. (In my case, the bin/view script was hanging in the final "print $tmpl" command for 2-3 mins, presumably for some other process that was locking STDOUT). A warning, though - with your patch, it seemed to work until I noticed that longish pages (e.g. WebPreferences) are not getting completely flushed. This may be localized to me, so am still looking for the solution... [The other thing that seems to be specific in my case is that this 'hanging' seems to happen only once per user session ... but only for longish or system (e.g. preferences) pages]

-- MaheshPanjwani - 19 Aug 2004

Thanks to everybody on this topic for being so helpful and explaining what was happening in your own circumstances.

I can confirm that I have been having this problem. This is what I found for my own case:

• Browser independent
• Skin independent (and I have never used Koala)
• 'Long' topics would not save correctly.
• ps showed oops processes which were not terminating
• Get the process ID for the oops process and manually kill it, and the saved page will instantly load. This confirms what JonBlower says above.
• Implement the use IO::Handle; STDOUT->blocking(0); patch as detailed by ChrisJacobson and problem disappears. All 'large' pages load completely fine.
• However, I do agree with MaheshPanjwani that the downside is that these pages are not flushed correctly. This has resulted, for me, in edit conflicts - i.e. Person A has page open and being edited. Person B clicks on 'edit' on the same page, and instead of getting a nice 'Sorry, it's locked by Person A' message, Person B sees a "500 - Internal Server Error" message. This, for me, is much less of a problem that having the page be un-saveable in the first place.

-- RossC - 17 Sep 2004

I've discussed this a bit in TWikiOnApache2dot0Hangs. Like many others with this problem, I'm running redhat.

-- DougClaar - 24 Jun 2005

I've found the problem. At least, I'm pretty sure I have. See TWikiOnApache2dot0Hangs. When/if I find the solution, I'll post it there.

-- DougClaar - 17 Apr 2006

Workaround

If for whatever reason you just can't save TWiki prefs, put your customizations in %MAINWEB%.TWikiPreferences instead, which has priority over all the other pref topics.

-- MattWilkie - 21 Dec 2004