[ic] Re: interchange-users Digest, Vol 47, Issue 18

[Carl Bailey <suppressed>]

On May 24, 2007, at 12:00 PM, suppressed wrote:

> Date: Wed, 23 May 2007 19:24:40 +0100
> From: Kevin Walsh <suppressed>
> Subject: Re: [ic] Rolling big tables (mysql)
> To: suppressed
> Message-ID: <suppressed>
> Content-Type: text/plain;  charset="iso-8859-1"
>
> Grant <suppressed> wrote:
> > > > I do keep a separate table of robot UAs and match traffic rows to  
> > > > them
> > > > with op=eq to populate another table with robot IPs and non-robot  
> > > > IPs
> > > > for the day to speed up the report.  Don't you think it would be
> > > > slower to match/no-match each IC request to a known robot UA and  
> > > > write
> > > > to the traffic table based on that, instead of unconditionally  
> > > > writing
> > > > all requests to the traffic table?  If not, excluding the robot
> > > > requests from the traffic table would mean a lot less processing for
> > > > the report and a lot fewer records for the traffic table.
> > > Perhaps you should create a column called "spider" in the traffic  
> > > table
> > > and save a true or false value depending upon the [data session  
> > > spider]
> > > value.  You can then generate reports "WHERE spider = 0", for  
> > > ordinary
> > > users, or "WHERE spider = 1" for robots etc.  An index on the spider  
> > > column
> > > would be nice, of course.
> >
> > I let this roll around in my head for quite a while and I ended up
> > writing the IC page accesses to my traffic table based on [data
> > session spider] like you suggested.  This should mean a much smaller
> > traffic table and less processing when running a report on it.  We'll
> > see how much time it buys me before running the report takes too long
> > again.  I also need to set up indexes.
> Also, you may as well grab the latest robots.cfg file from CVS and
> "include" it into your interchange.cfg file.

... and where would that be in CVS?  I read the MANIFEST and it cited  
only /debian/robot.cfg but the file is not present here:
http://www.icdevgroup.org/cgi-bin/cvsweb/interchange/debian/? 
only_with_tag=REL_5_4_2

I guess either the file is missing or the manifest is wrong.

Grateful if you can point me in the right direction.

Carl
- - - - - - - - - - - - -
 Carl Bailey
 Triangle Research, Inc.
 tel: 919.323.8025
- - - - - - - - - - - - -

_______________________________________________
interchange-users mailing list
suppressed
http://www.icdevgroup.org/mailman/listinfo/interchange-users