Question

Hello,

I have used TWiki to set up a few websites with dragon skin and hand over the new address to several search engines for websites. So far I found out, is the chance to get a good place quite poor. I am not sure, why the search engnines have problems with TWiki. Any idea, where the problem is?

Eg: I made www.destedt.de, which should be the main site for that village, but good even after months does not find it properly only a bad link with an oops/help line. How can that be?

Environment

-- JensRoeder - 01 Feb 2005

Answer

Only finding "oops/help" pages is a problem with the way search engines crawlers traverse TWiki sites that can be solved. You can uses a robots.txt file to PreventGoogleToIndexRevisions (should work on other search engines provided their crawlers obey robots.txt).

There is also work in progress to add to pages with url parameters so that SearchEngineIndexOnlyPlainView.

That should prevent search engines wasting time in parts of the site that it doesn't need to look at. Which should hopefully mean it indexes and returns results for the parts of the site with proper content.

See also SpeedUpTipsForTWiki20040901#The_robot_problem_is_caused_by_s although the topics above should be enough. The suggestions on that topic are rather drastic IMO but let us know how you get on.

-- SamHasler - 01 Feb 2005