Hi, Over concerns that Google got 'angry' at my robots.txt (i.e. seemed to go away when I put too much stuff in it) i just made it very simple: User-agent: * Disallow: (i.e. wide open) This may just be paranoia but in my experience when it comes to search engines the question isn't are you paranoid, it's are you paranoid _enough_ But whether your robots.txt is simple or complex I suggest you search on google for 'robots.txt validator' and check yours there! Cheers, Bryan On Mon, 1 Nov 2004 09:00:22 -0600, Steve <suppressed> wrote: > > > > So I tried simply switching to low > > traffic mode and both orders and googlebot seem to be doing OK > > (Googlebot grabbed over 25,000 pages today). So for now I'll stay > > with this I guess! Thanks for everyone's help. > > > > Bryan > > > > p.s. I posted about 'what is rpc mode' ... does anyone have detailed > > references about what low, high and rpc mode are and what all of their > > configs mean? > > Hello Bryan, > > What does your robots.txt file look like? > > I have been modifying mine slightly to keep inktomi and googlebot from > access certain pages like: > basket.html, process.html, order.html > > It really helped with inktomi getting stuck on certain pages.... > > > -Steve > > > > _______________________________________________ > interchange-users mailing list > suppressed > http://www.icdevgroup.org/mailman/listinfo/interchange-users > _______________________________________________ interchange-users mailing list suppressed http://www.icdevgroup.org/mailman/listinfo/interchange-users
Mail converted by mhonarc 2.6.15
This archive provided courtesy of JSW4.NET, Internet Hosting Services for Small Business.