Re: [ic] HELP - Inktomisearch stuck on ord/basket.html page

[Jamie Neil <suppressed>]

Jamie Neil wrote:
> Andrew Rich wrote:
>
> > I initially stopped it by adding an entry to robots.txt for
> > /cgi-bin/cartname/ord/basket.html and this pushed it off the page.
> >
> > I then thought orders were reduced and removed the entry from
> > robots.txt.  Inktomisearch came back and became stuck on the page again.
>
>
> I don't see any reason why adding pages (or directories) to your 
> robots.txt file would affect orders - this is the preferred way of 
> limiting spider activity on your site.
>
> Use something like:
>
>   User-agent: *
>   Disallow: /cgi-bin/cartname/ord/
>   Disallow: /cgi-bin/cartname/scan/
>   Disallow: /images/
>   Disallow: /cgi-bin/cartname/login.html
>
> As well as "ord" we also ban robots from our images directory (to stop 
> our graphics turning up in image searches), and "scan" to stop them 
> indexing "more" links.

I should add that banning robots from "scan" will prevent them indexing 
_any_ category/search links on a stock foundation system, which is 
obviously undesirable. We use action maps to do most of our links, e.g.:

  /cgi-bin/cartname/category/spanners.html

and so this is not a problem for us.

We also feed spiders slightly different search results to normal users: 
normal users get 10 results per page while spiders get one page with all 
the results on it. This allows the search engines to index all the 
products in a category without having to follow any "more" links.

--
Jamie Neil | <suppressed> | 0870 7777 454
Versado I.T. Services Ltd. | http://versado.net/ | 0845 450 1254
_______________________________________________
interchange-users mailing list
suppressed
http://www.icdevgroup.org/mailman/listinfo/interchange-users