TIFF performance: overviews and internal tiles: Surprise!

[Frank Warmerdam]

>> The conclusions are quite surprising:
>>
>> * The presence of overviews makes little difference in the performance
>> otherwise, about 1/20 of a second in the most dramatic case.
>>
>> * Internal tiling seems likewise ineffectual, a difference of several
>> microseconds.
>>
>> * However, internal tiling will hurt performance substantially in a case
>> where the request ends up grabbing the entire image anyway.
>>
>>
>> Is anybody else able to replicate these findings, or to achieve
>> different results in a controlled experiment? Our hardware is rather
>> beefy (8 CPU cores and a 4-disk RAID-5) so perhaps that's confounding my
>> experiment? Perhaps these performance enhancers are mostly useful on a
>> slower single-disk system?

Folks,

As Umberto wrote, this test is really only a valid test of timing when
everything is in the in-memory file system cache.

The slowness of tiled images without overviews for an overview draw is
not surprising since all tiles will be read and subsampled.  Scanline
oriented files will only have the required scanlines read, (for a 10x
by 10x reduction in size, only 1/10th of scanlines would be read) so
overviews of scanline oriented products without overviews are not
dramatically bad.

Overall, I think this test demonstrates the value of in-memory caching
of data, but does not really measure performance for large datasets
of which only a very modest portion will ever be cached when requested.

Nevertheless, I'm very pleased to see some organized effort to measure
performance in different scenarios.

PPS. I think it is also wrong to draw any conclusions about differences of
1 or 2 percentage points in performance.  Such results should be considered
as equivelent for all meaningful purposes.

Best regards,
-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | President OSGeo, http://osgeo.org