create new tag
, view all tags

TWiki on thttpd

thttpd is a simple, small, portable, fast, and secure HTTP server written by Jef Poskanzer. I've been running thttpd with TWiki under FreeBSD for many years and recently updated to TWiki 4.0.X so I thought I would document the steps here. I have TWiki setup to be viewable by all guests, but only authenticated users can edit topics, etc. I haven't modified the process substantially since setting this up with Beijing release of TWiki in early 2003 and I'm as yet unfamiliar with TWiki v4 so apologies if I miss anything obvious.

thttpd Installation

Under FreeBSD, thttpd can be installed as a port, but it can be easily ported to other platforms. By default thttpd supports a very basic form of htpasswd authentication (since .htpasswd file either protects the whole hierarchy from the web root directory, or only a single directory, not subdirectories). I enhanced this somewhat by enabling password protection of directory hierarchies, as per usual (eg. with apache). As at the time of writing, the current release of thttpd is 2.25b. My patch for beefing up htpasswd support applies against at least 2.23beta1 onwards with no problems. Strictly speaking you can use thttpd successfully without this patch but it is probably more secure with it and is a useful feature to have.

You can then either:

  • download, unpack, apply my patch and compile thttpd yourself, or
  • If using FreeBSD ports, just run "cd /usr/ports/www/thttpd; make patch", then cd to the work directory (eg. /usr/ports/www/thttpd/work/thttpd* by default) and patch the source there. Then go back to /usr/ports/www/thttpd and continue to make the port.

This is not intended to be a complete guide for installing thttpd - please refer to the usual documentation for that. However, I highly recommend running the webserver under a non-privileged user such as www. I will assume this is the case below when setting up authentication. You should also setup a group called twiki and make www a member of the twiki group. Note that by default thttpd ships to run in chroot mode. Personally, I don't bother with this since thttpd is by design small and secure, however if you wish to do that you will need to take steps to ensure that all of the utilities needed to support TWiki are available (ie. perl, rcs, etc). I haven't tried this so assume a standard non-chroot install below, but I'm not aware of any changes that would be required.

thttpd Configuration

The remaining steps are to configure thttpd specifically to allow certain TWiki functions. The main things are:

  • Tell it that the CGI scripts are valid. This is done in the thttpd.conf file, which by default will be in /usr/local/etc/thttpd.conf by setting the cgipat variable. I have it set to the following (as I have other non-twiki pages which use a more traditional cgi-bin directory too):

The reason for the .protected entry will become apparent below when we set up the authentication. You can be more explicit than the above if you are particularly paranoid. If you want to use the htpasswd patch mentioned above, you will also need to set recurspasswd in your thttpd.conf file.

TWiki Installation

If you are making a new TWiki site or upgrading to a new version of TWiki (that is, you will be working with a fresh TWiki tree), follow the usual installation steps, unpacking, and running the UpgradeTwiki script. Copy over your user and group topics and webs as necessary. Ensure you have a working perl and required modules to support TWiki. If you are just going to take an existing TWiki site and convert it to use thttpd, take a backup first as we will be moving some files around and changing permissions and ownership on things.

The format of password file supported by the standard login manager is incompatible with thttpd (as it includes a 3rd field, email address, on the end). You need to install a password manager which handles the simpler htpasswd file used by thttpd. See the ThttpdHtPasswdUser.pm attachment below. This should be installed into the lib/TWiki/Users directory. After doing this it will appear in the configuration screen as one of the available Password Managers. (I haven't investigated what effect the lack of email in the password file has. I seem to be able to do password resets ok by using the email in the User topic. It seems weird that anyone can reset any password, even if not their account. Not sure how this can be avoided since you can't login to reset it as you presumably don't know your password. Maybe it should only send a request to the administrator?)

You will need to apply the patch for TWiki.pm which allows it to find out what script is running, as the existing code only supports Apache variables (SCRIPT_FILENAME and SCRIPT_URL). This is a one-liner. Things will mostly work without it, but some links within a page don't resolve properly, depending upon the template you are using.

In order to use thttpd, file permissions will need to be changed as thttpd won't serve files that aren't already publicly readable, and won't execute CGIs scripts that aren't world executable. It also barfs on files that are executable but not CGIs. Additionally we must ensure that the TWiki data files are writable by the web-server, but not viewable by a roving command-line user (since there may be confidential pages). My preference is to have all the twiki code owned by root and the TWiki data to be owned by www. (In fact, you can just make data and pub group writable by twiki but new topics and webs end up being owned by www anyway so there isn't that much point in it.)

To expedite all this I wrote a script which is run from the root of the new twiki installation as the root user. Please go through the script very carefully before running it to see what it does; I take no responsibility, etc, etc. I repeat - be careful, you need to run this script as root. Note, this script uses the following possibly non-portable features:

  • find -print0 and xargs -0. This is probably unnecessary for the main TWiki code, but people sometimes name things with spaces in them;
  • chmod o+X should be fairly widespread now. It is the same as o+x but will only add the execute permission if it is already set for one of the other user classes (eg. it's a directory)
  • the chown user:group files... syntax (some older systems use user.group instead)
If your operating system doesn't support these, you'll need to modify the script accordingly. Note, if you subsequently install plugins, it might be worth rerunning the script later to fixup any of the new files that get installed. This can be done harmlessly and it will stop when it gets to the bit that can't be repeated.

As noted by the script, you need to manually install your .htpasswd file into the twiki/bin/.protected directory (so only registered users may run perform those functions) and make a hardlink to twiki/data/.htpasswd.

TWiki Configuration

Perform the usual TWiki configuration using .../twiki/bin/configure. Be sure to set the Password Manager in the Security Settings to ThttpdHtPasswdUser. By this stage everything else should be working.

Other Issues and Gotchas

  • Take care that you've explicitly set $twikiLibPath correctly in LocalLib.cfg and not relied upon the default which works it out from the path of the CGI or you will find that scripts requiring authorization (eg. edit) will silently fail. This is because they won't be able to find the lib directory correctly using the relative path as they now live in the .protected subdirectory.
  • I have only briefly tried SpeedyCGI. It looks worthwhile.
  • By default thttpd has a CGI execution time limit of 30 secs which is specified at compile time. In practice this seems to be long enough for most TWiki operations even on slow hardware, but be aware of this and check your webserver logs for errors notifying you of long-running processes being killed.

-- Contributors: CallumGibson - 12 Dec 2006


Thanks Callum for sharing this with the TWikiCommunity!

-- PeterThoeny - 13 Dec 2006

Callum, I'm delighted to hear that someone has decided to try grasping this nettle! I've been using IndigoPerl on my laptop, but have always felt Thttpd offered a better solution for personal wikis.

How about we bundle up a TWikiFor with Thttpd + preconfigured TWiki, for personal twiki users? How hard do you think it would be?

-- CrawfordCurrie - 13 Dec 2006

Is this the right place to continue the discussion? I'm not used to all these modern features. wink

Anyway, what's involved in making a TWikiFor bundle? Is it a prebuilt binary distribution (for a particular OS) or something else? I pretty much use all the defaults anyway so even for personal use there isn't much to do (just look at how sparse my instructions are above). For personal use, however, I reckon it would be even simpler as you could not worry about setting up any permissioning at all.

-- CallumGibson - 13 Dec 2006

One approach would be:

  1. Identify the setup in configure, and generate the appropriate LocalSite.cfg
  2. Create a new tools/MANIFEST that
    • includes that LocalSite.cfg,
    • hacks down the distribution to remove irrelevances (e.g. would we really need all the plugins?)
    • adds the twiki-specific thttpd config files in the right places in the TWiki tree
  3. Use the automatic build to generate a release package
That would suffice to generate the TWiki bits (TWikiForThttpd). It might be neat to then bundle it together with a pre-configured thttpd using Windows Installer to create TWikiForLaptops, for example. This would give laptop users a one-touch TWiki setup, which would be really 8-).

-- CrawfordCurrie - 14 Dec 2006

OK, but so far this is only being done on FreeBSD, and by one person?

Well, just last week I threw a new disk into an old laptop and installed FreeBSD, leaving apache and TWiki on the to be installed list. Over the holidays I'll see whether I can replicate Callum's success with thttpd or not. Either way, you'll hear the screams here.

A personal use TWiki for Windows laptop users is a great idea! Even though I don't use it myself, I'd love to see it done.

On a FreeBSD (or Linux etc) laptop, I think the mindset is different. There's always going to be a temptation to put the little twiki on a foreign network for a while, making accounts for new acquaintances at gatherings, etc. After all, why would you have a unix laptop if you weren't going to show it off? smile

I don't know about other people, but as a server geek I set up my laptops with similar security to a server, since I never know whose dirty network it'll be touching or who I'll be letting login to feel the power. So down the track I'm likely to forget and assume that the TWiki can be safely accessed by others. Therefore on non-Microsoft platforms I'd prefer that standard security measures be followed, i.e., the same setup as for a server, unless there's a strong reason to do otherwise and then build in protection from accidental exposure by an overly enthusiastic future self.

-- SueBlake - 14 Dec 2006

Yes, AFAIK Callum is the only person to try this. I have no idea how well it maps to Windows. Heck, if I did, this topic would be no fun, would it? wink

Imposing some sort of default ACLs is essential, I think. Initially that can be Callum's script, but in the future should be pushed back into the installer tech for TWiki (BuildContrib).

I look forward to hearing how you get on, Sue. I still think we need to hit Windows, sadly, to appeal to the common herd.

-- CrawfordCurrie - 15 Dec 2006

Good to see thttpd being used to simplify TWiki setups. However, for a Windows laptop oriented installation, it's probably easiest at present just to download and install TWikiVMDebianStable, which can be up and running within about 5 minutes after download and has proven quite popular. (This installs a VMware virtual machine containing pre-configured Linux and TWiki.) A good Windows installer for TWiki would be less memory intensive and easier to update, but it might be interesting to do a TWiki VM that is based on FreeBSD and thttpd, particularly if you want to spread the use of these tools.

-- RichardDonkin - 15 Dec 2006

Agreed. However the TWikiVMDebianStable is a bit of a monster, and I am interested in any work towards a lighter solution. All TWiki really needs is a CGI-capable web server (of which there are many for Windows), a perl (perhaps the biggest challenge) and TWiki. The footprint of all this could really be quite small.

-- CrawfordCurrie - 16 Dec 2006

TWikiOnMemoryStick does look promising for Windows people, will certainly try it out. There definitely should be more TWikiFor's.

-- FranzJosefSilli - 17 Dec 2006

Let me add, that I've made no modifications to anything that are FreeBSD specific, I'm just using the ports collection, so it should work on your favourite flavour of Unix. I make no comments on Windows other than to offer my deepest sympathy.

I am having some strange hangs when accessing the FreeBSD server from Windows with IE, and it looks like the ci process is hanging (I have 4 sitting in disk wait). I think this is actually a FreeBSD locking problem since I have /tmp as a swap-backed partition on this box. I'm running a very old release of FreeBSD.

PS. Hi Sue. Haven't seen you on #bugs lately. Let me know if you have any questions on the setup.

-- CallumGibson - 17 Dec 2006

Preliminary feedback - I've installed but not configured/tested yet.

New box, FreeBSD-6.2-RELEASE, Thttpd-2.25b, TWiki-4.1.0 (not the port)

The instructions were easy to follow and nothing bombed out really. The patch applied to this version of thttpd just fine. There's a little error in twiki-postinstall.sh, line 92 is missing its closing quote, but it only affects display of the message at the end.

Since this is a fresh TWiki install, I don't have an old .htpasswd, but I'll know to copy the new one correctly when I get that far.

It might help to give the command to create the hard link. It's not something people do every day and I can imagine it being misinterpreted as a symlink which obviously wouldn't be right.

Is this setup meant to be without chroot? It looks like chroot is in the default config for httpd, so we might need to tell people to deal with that one way or the other because it's easy to overlook.

Lots more to do still. I'll be back...


Re-reading the comments, you know, I think people are looking at this as a little toy web server ideal for serving a few pages off a creaky old computer. Aha, that's why all the puzzling stuff about personal setups. No, I see thttpd as an Internet server which is more efficient in a high load environment, and which is used by some of the busiest places on the net for that reason. It's used by sites that demand more than apache, it's not just something to make do with when you need less. Sure it might be good for a personal TWiki too, if you're interested in that good on you, but that's not what I'm doing and I don't think that's what Callum set up these instructions for either. It's a TWiki site that happens to use an alternative to apache. Please correct me if I'm mistaken.

-- SueBlake - 21 Jan 2007

OK, sorry folks, I can't get thttpd itself working and I don't have any more time, I want to have a web server up today. It's only some stupid error I expect, but it's costing too many hours. I'll try it again with another box when there's longer time frames.

-- SueBlake - 22 Jan 2007

Found a bug where anchors within a TOC didn't resolve properly due to a mangled base value in the header.

-- CallumGibson - 14 Feb 2007

Oh, fixed the bug in the post install script which Sue noted above, too.

A comment on the chroot issue - I don't believe there are any additional issues with running TWiki under chroot with thttpd than there would be with any other webserver. The biggest issue is you need to install all the perl tree and any other utilities you need within the root, of course. I'll make a comment to that effect, since chroot is the default config for thttpd (easily changed), but I don't assume either case and it's not really a thttpd-specific issue that I can see.

-- CallumGibson - 14 Feb 2007

One question. How is performance with thttpd? Does it surpass performance running Apache-httpd+mod_perl? I ask because my experience with thttpd has generally been quite favorable, and am curious if the technical benefits of running this configuration outweigh running a nonstandard configuration.

-- BrianGupta - 25 Mar 2007

For general performance comparison, see http://www.acme.com/software/thttpd/benchmarks.html. But as twiki is heavily CGI dependent, I don't know that there are any performance benefits from using thttpd, apart from it's smaller footprint on small machines. As I note above, I think there would be some benefit from using SpeedyCGI, but I've still only experimented briefly.

-- CallumGibson - 14 May 2007

Topic attachments
I Attachment History Action Size Date Who Comment
Perl source code filepm ThttpdHtPasswdUser.pm r1 manage 7.8 K 2006-12-18 - 07:09 CallumGibson Password Manager for thttpd htpasswd file
Unknown file formatdiff scriptname.diff r1 manage 0.4 K 2007-02-14 - 02:56 CallumGibson scriptname patch for TWiki.pm
Unknown file formatdiff thttpd-recurspasswd.diff r1 manage 5.2 K 2006-12-13 - 02:53 CallumGibson recursive htpasswd patch for thttpd
Unix shell scriptsh twiki-postinstall.sh r3 r2 r1 manage 2.9 K 2007-02-13 - 01:03 CallumGibson Script to set permissions for thttpd
Edit | Attach | Watch | Print version | History: r27 < r26 < r25 < r24 < r23 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r27 - 2007-05-14 - CallumGibson
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2018 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.