Re: [ic] Performance hit from having many images in one directory?

["John1" <suppressed>]

On Tuesday, August 23, 2005 1:39 PM, suppressed wrote:

> Quoting Jeff Fearn (suppressed):
> > On 8/23/05, John1 <suppressed> wrote:
> > > At the moment we have all our item images in a single directory
> > > (and all the thumbnails in one directory).  There are currently
> > > only about 1000 items, but this could increase significantly as
> > > time goes on (especially as we tend to leave any images of old
> > > discontinued products there just in case search engines etc still
> > > have them catalogued).
> > >
> > > We run the Linux ext3 filing system.  Is there a performance hit
> > > from storing loads of files in a single directory?  I am sure I
> > > read somewhere once that there *is* a performance hit, but I have
> > > no idea how many image files we would need to store in one
> > > directory before the performance implications became significant,
> > > bearing in mind that each page loads between 10 and 20 images from
> > > the thumbnails directory and 1 image from the main items directory?
> > >
> > > Please can anyone enlighten me on this?  Thanks.
> >
> > Good discussion at
> > http://answers.google.com/answers/threadview?id=122241
Thanks Jeff - good overview.

> There are two major options once you hit 10,000 files or so:
OK, so 2000 or so files in the same directory on a Pentium 4, 3.0GHz with 
2GB RAM,  Western Digital Raptors SATA mirrored drives should be OK on ext3 
then?

Presumably the 10,000 is very much a ball park figure?  I guess the more 
important factor is how many times the file system has to access that big 
directory to serve up a single web page - yes?  In my set up, there might be 
15 thumbnail images on a typical webpage, so that's 15 searches through the 
1000 files then - does anybody have any subjective experience of when 
performance I am likely to see a noticable performance hit? 2000, 10000, 
50000 files?

>    1. Switch filesystems to XFS or ReiserFS for their tree-based
>       scheme. They don't degrade like ext2.
I like this idea.  Are there any good reasons why I *shouldn't* switch to 
one of these filing systems?  If not, then I guess this has to be the way to 
go with any new partitions?

>    2. Use a hashing scheme of some type to store images
>       in subdirectories. It can be as simple as manually
>       by category, or something automated based on say,
>       the first two letters of the image name. We do
>       this for session and temporary files, and we have
>       some routines which support finding the file name
>       automatically.
This is an option would not be too difficult for me to implement though not 
as nice to live with - it's so much nicer for maintenance and admin of the 
images to have them all stored together in one directory.

Thanks Jeff & Mike for your informative replies - much appreciated.

_______________________________________________
interchange-users mailing list
suppressed
http://www.icdevgroup.org/mailman/listinfo/interchange-users