[Gdal-dev] Python USGS ASCII DEM

Frank Warmerdam warmerdam at pobox.com
Mon Jul 26 16:35:57 EDT 2004


Norman Barker wrote:
> Hi,
> 
> We are writing an GDAL program using the python wrappers and everything 
> is going well (the wrappers
> around GDAL are really cool!).
> I am having problems with a bottleneck with the USGS ASCII DEMs that we 
> are using.  I understand from
> http://www.remotesensing.org/gdal/frmt_usgsdem.html that they are 
> treated as one tile and are thus loaded in
> completely.
> 
> The line we are using is
>        dem_src_data = dem_band.ReadAsArray(0, (start_row + iY) , 
> src_tif.RasterXSize, 1)
> so we are trying to read a section as long as the input tif file from 
> the DEM.  The DEMS are over 100mb.
> 
> All the other code is very fast thanks to GDAL, but is there a work 
> around for this bottle neck?

Norman,

There are two related issues here I think:

  1) Reading any part of a large USGS DEM is slow because the GDAL USGS DEM
     driver processes the whole file on disk.

  2) If you GDAL cache size is smaller than the size of the one big tile in a
     USGS DEM file, then you are even worse off as that one tile will be
     freed before the RasterIO() call returns.  To deal with this ensure you
     set the GDAL cache size at least large enough to hold that one tile.

There isn't much that can be practically done about problem 1.  Note that in
the file, USGS DEM data is stored in vertical profiles.  So, regardless of
how a driver is written, reading part of one horizontal scanline will always
result in most profiles having to be scanned.

As Norman Vine suggested, you should really consider converting these DEMs
into another format if you plan to work with them alot.  Otherwise ensure that
at least your cache size is set large enough for the big tile to remain
resident.

Best regards,

-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | Geospatial Programmer for Rent




More information about the Gdal-dev mailing list