[GRASS-dev] Re: r.in.gdal: how to speed-up import with huge amount of bands?

[Markus Neteler]

On Mon, Mar 29, 2010 at 9:00 AM, Markus Neteler <neteler at osgeo.org> wrote:
> On Mon, Mar 29, 2010 at 8:35 AM, Markus Neteler <neteler at osgeo.org> wrote:
>> Hi,
>>
>> I have received temperature map time series of 50 years of daily data
>> in single Geotiff files (Tmin, Tmean, Tmax). Each GeoTIFF has around
>> 21550 bands, 4GB file size.
>>
>> Problem: the import takes "forever" despite using a superfast disk, i.e.
>> 120 seconds per band (size is only 464 x 201), so 29 DAYS for each file.
...
> Index: raster/r.in.gdal/main.c
> ===================================================================
> --- raster/r.in.gdal/main.c     (revision 41604)
> +++ raster/r.in.gdal/main.c     (working copy)
> @@ -666,6 +666,7 @@
>     /*      Select a cell type for the new cell.                            */
>     /* -------------------------------------------------------------------- */
>     eRawGDT = GDALGetRasterDataType(hBand);
> +    GDALSetCacheMax (2000000000); /* heavy caching */
>
>     switch (eRawGDT) {
>     case GDT_Float32:
>
> It allocates way more RAM (2GB) but the speed remains exactly
> the same: 120 seconds per band.

Ha! Setting the cache to the file size + minor overhead helps. Now it
takes 5 seconds instead of 120...

At this point I would implement this as cache= parameter. The question
is how to preset it. Or make it a flag "make cache as large as input file"?

Markus