Grant <suppressed> wrote: > > > I just had a look at robots.cfg and I think I see a few opportunities > > > for false positives. I would think "agent" could be bad, and there > > > are browser toolbars for GetRight and Yahoo which probably alter the > > > UA. Is there a crucial set of NotRobotUA entries to go along with > > > robots.cfg? > > > > > > Is anyone using robots.cfg and actively watching for false positives? > > > > > I'll look into those. Do you know the UA names for the various toolbars? > > I can probably look that up somewhere. I wouldn't have thought that > > "agent" would be a risk. > > I have a good system set up for rooting out false positives. I'll put > the current robots.cfg file into effect and keep a close eye on > things. I'll have some specific info for you soon. > That'd be good - thanks. I had a quick look around and couldn't find any information on Yahoo (etc.) toolbar user agent names. If you want to propose entries for the various directives in the new robots.cfg file then send them my way and I'll see that they get added. Well, I'll either add them or challenge you to justify them. :-) Most of the current robots.cfg file is derived from the user agent strings found in the www.interchange.rtfm.info log files. -- _/ _/ _/_/_/_/ _/ _/ _/_/_/ _/ _/ _/_/_/ _/_/ _/ _/ _/ _/_/ _/ K e v i n W a l s h _/ _/ _/ _/ _/ _/ _/ _/_/ suppressed _/ _/ _/_/_/_/ _/ _/_/_/ _/ _/ _______________________________________________ interchange-users mailing list suppressed http://www.icdevgroup.org/mailman/listinfo/interchange-users
Mail converted by mhonarc 2.6.15
This archive provided courtesy of JSW4.NET, Internet Hosting Services for Small Business.