[Gdal-dev] performance problem with several DEM reading/updating at the same time

[Frank Warmerdam]

On 5/20/05, LALLAURET Fabrice - <SSII> <fabrice.lallauret at thales-tts.com> wrote:
> Hello,
> I'm playing with gdal and I don't understand some performance problem I
> have:
> I'm reading several geotiff dems pixel by pixel at the same time (I
> know, this is not optimal but for my application, it's simplifying the job).
...
> When I only read a single dem, the test program takes 0.95 s to run. If
> I read a second dem at the same time (as above), the program takes 50 s.
> Why this difference of performance ? (With a writting process performing
> the same test, the program reach 12min !!)

Fabrice, 

My guess is that once you go to the second file you are getting into
a cache thrashing problem.  This is especially likely with some DEM 
formats such as USGS DEM or DTED that are not row oriented.  
If the two files together take more memory than the cache is configured
for, then each read will force the other files data out of memory causing
huge amounts of extra work.

The obvious solution is upping your cache limit, for instance via the
GDAL_CACHEMAX environment variable.  Try setting that to 100
(for 100MB) and see if it makes a difference. 

If not, you may need to provide the full configuration (commandline
program + data files) for me to try out.  Lots of issues can have an 
impact including size of dataset, format, etc). 

Best regards,

-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | Geospatial Programmer for Rent