[gdal-dev] gdal_translate is slow

[Denis Rykov]

Thanks guys for clarification!

On Wed, Sep 4, 2019, 5:53 PM Even Rouault <even.rouault at spatialys.com>
wrote:

> On mercredi 4 septembre 2019 17:26:14 CEST Denis Rykov wrote:
> > Thanks for quick reply, I've uploaded grib file here:
> > https://transfer.sh/5JCVX/download.grib
>
> Turns out that my guess wasn't so bad after all. The uncompressed file
> size is
> 3601x1801x14(bands)x8(bytes_per_pixel) = 693 MB
> whereas a GRIB dataset has an internal cache by default of only 100 MB.
> As you write to a pixel-interleaved GTiff, there's constant back and forth
> between bands when reading chunks and thus the GRIB cache has no effect.
> So 2 possible workarounds:
> - increase GRIB_CACHEMAX to 1000 for example. Limited to a GRIB dataset
> that
> can fits uncompressed in memory.
> - add "-co interleave=band" to generate a band-interleaved geotiff. That
> one
> can work with an arbitrarily large GRIB file
>
> I've committed an improvement, so if you now row master with --debug on,
> you'll see a hint
> """
> GRIB: Maximum band cache size reached for this dataset. Caching only one
> band
> at a time from now, which can negatively affect performance. Consider
> increasing GRIB_CACHEMAX to a higher value (in MB), at least 693 in that
> instance
> """
>
> As far as I can see in
> https://github.com/mapbox/rasterio/blob/
> e1c984bf0e4a18e039569bb3bafe6667bb5b3a69/rasterio/rio/convert.py#L76
> it presumably ingest the whole dataset into memory (Sean, correct me if
> I'm
> wrong), and thus those caching issues don't trigger since the input
> dataset
> will be read band-per-band.
>
> Even
>
> --
> Spatialys - Geospatial professional services
> http://www.spatialys.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20190904/8b989d46/attachment.html>