We relaunched the TWiki.org project with an expanded TWiki charter, and we invite you to participate! The TWiki.org Code of Conduct agreement took effect on 27 Oct 2008. We ask existing twiki.org users to opt-in. You need to opt-in to participate in the Blog, Codev, Plugins and TWiki webs. -- PeterThoeny - 27 Oct 2008
You are here: TWiki> Wikilearn Web>RobotExclusion (07 May 2003, RandyKramer)
Tags:
create new tag
, view all tags
There are ways to keep sites or pages from being crawled by search engine indexing robots.

See:

Contents

Notes

Last time I paid attention to which WikiLearn pages were being indexed by Google, I found a lot of duplicates because Google was indexing pages like WebChanges and other pages with dynamic searches.

Because a robots.txt file must be used at the top level of a site (like http://twiki.org/robots.txt) and the WikiLearn site is currently hosted at twiki.org along with several other webs, I can't use the robots.txt approach.

The alternate, which doesn't work for all robots, is the robots meta tag:

from HTML Author's Guide to the Robots META tag: 

<html>
<head>
<meta name="robots" content="noindex,nofollow">
<meta name="description" content="This page ....">
<title>...</title>
</head>
<body>
...

I may try this at the top of the content of some WikiLearn topics to see if it will work even though it will not at the required/recommended location (in the head).

See the last link under resources (Hack #100) for "noarchive" and other possibilities.

Page being tested:

  • RobotsTest
  • WebChanges (added 7 May 2003)

Resources

See Resource Recommendations. Feel free to add additional resources to these lists, but please follow the guidelines on ResourceRecommendations including Guidelines_for_Rating_Resources.

Recommended

Contributors

  • () RandyKramer - 15 Apr 2003
  • If you edit this page: add your name here; move this to the next line; and include your comment marker (initials), if you have created one, in parenthesis before your WikiName.

Page Ratings

WebForm
PageStatus Scribbles
Edit | Attach | Print version | History: r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r2 - 07 May 2003 - 13:42:18 - RandyKramer
 
TWIKI.NET
Copyright © 1999-2008 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback
See TWiki's New Look