Question

On Twiki sites indexed by search engines, debug.txt and log files grow huge .

I installed twiki on a site indexed locally by altavista... every hour! consequently, the logs files are huge, and the debug.txt filed filled the disk (it weighted 256M).

The main culprit is Diffs that outputs all the diffs in the debug file!

-- ColasNahaboo - 04 Dec 2001

The real problem is that the line in the template

    <meta name="robots" content="noindex" />

http://www.w3.org/Search/9605-Indexing-Workshop/ReportOutcomes/Spidering.txt

So a value of "NOINDEX" allows the subsidiary links to be explored, even though the page is not indexed.

So the real solution would be to change most lines to NONE execpt the pages we want to be indexed (view.tmpl only I think)

Answer

My solution was to run each night a small shell script via cron truncaturating debug.txt and keeping only the last 2 log files. I append it here, and I suggest this problem to be at least documented in the install process

#!/bin/ksh
# local wiki data dir
cd /net/servers/www-sop/files/wiki/sophia/data
# truncate debug.txt
>debug.txt; chmod a+rw debug.txt
# dumb but portable way to keep only last 2 log files
ls -1r log* | (read;read; while read file; do rm -f $file; done)
# keep browsers happy
echo 'Contents-type: text/us-ascii';echo; exit 0

.