Feature Proposal: Set User Agent for TWiki's Include Requests

Motivation

TWiki's INCLUDE{} can bring in content from other webservers. But, it does not set the UserAgent in its requests, therefore many servers reject the request as being too spammy.

Description and Documentation

See http://koala.ilog.fr/twikiirc/bin/irclogger_log/twiki?date=2008-07-25,Fri for the conversation that led to this.

It comes in as e.g.

- - [25/Jul/2008:12:14:50 -0500] "GET /~twiki4/cgi-bin/view/Bugs HTTP/1.0" 403 310 "-" "-"

A config option to set the UA should be simple to add. Just define it as a $TWiki::cfg{Net}{UserAgent} option, then include it in external requests (TWiki::Net::getExternalResource)

Here's the Moz standard: http://www.mozilla.org/build/revised-user-agent-strings.html - the RFC is at http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.43

peterthoeny recommended to specify in order of importance, hence: TWiki/4.1.2/twiki.org i.e. "TWiki" + twiki version + host name

I think it's best to put as a config item as it means we can hardcode it on t.o. for now

We elected to have the UA include a site identity string, i.e. if the site knows itself as twiki.org to include that to differentiate it from other twiki sites on the same server

Once this is complete we can show the Bugs web summary on each the Plugin page.

Examples

Impact

Implementation

-- Contributors: MartinCleaver - 25 Jul 2008

Discussion

Can we have this for TWikiRelease04x03?

-- RafaelAlvarez - 25 Jul 2008

It is a good idea.

Note one little thing. Today we recommend setting up TWiki to deny itself to avoid an obvious DOS attack where you let two TWiki's eat themselves up.

If one includes the other that includes the first the including goes on until the TWiki time out. And then it only takes a little fun with ab to get two Twikis to burn each other up.

So if we implement this we also need to make sure we can setup TWiki to not accept itself as user agent.

-- KennethLavrsen - 25 Jul 2008

Note - without a committed developer this proposal cannot be accepted - only discussed.

A bug item was raised but it is too early for that.

Target release would be GeorgeTown which is 5.0. There is no 4.3 on the roadmap. Next release must address the important roadmap items and not just small feature creaping.

-- KennethLavrsen - 28 Jul 2008

Ping! This looks like a small and useful enhancement. Anyone taking it?

-- PeterThoeny - 2010-06-23