[gdal-dev] [GRASS-user] Slow import of GHSL

[Markus Neteler]

On Wed, Mar 22, 2017 at 9:28 PM, Markus Metz
<markus.metz.giswork at gmail.com> wrote:
> On Wed, Mar 22, 2017 at 8:12 PM, Markus Neteler <neteler at osgeo.org> wrote:
...
>> Nikos, for an even bigger map try
>>
>> Global Surface Water (2000-2012, 30 m, Data coverage is from 80° north
>> to 60° south):
>> http://landcover.usgs.gov/glc/WaterDescriptionAndDownloads.php
>> by USGS. 1.6GB in size.
>>
>> Using gdalbuildvrt I created a VRT from the 504 GeoTIFF files.
>>
>> After import into GRASS GIS, here the timings:
>>
>> # final map size:
>> g.region -p
>> ...
>> rows:       493200
>> cols:       1296001
>> cells:      639187693200
>>
>> (handling only works in GRASS GIS 7.3.svn since Markus Metz's recent
>> improvements on global data import are needed).
>
> (my changes were bug fixes, not improvements)
>
>>
>> Benchmarks:
>> - Import took 2h while reading the data from a CIFS mounted storage
>> box (slow) and writing on SSD.
>> - Displaying the entire map (639 giga-pixel) in GRASS GIS' display
>> (d.mon) took ~15 sec over a ssh tunnel from my laptop to the server,
>> since I am at a conference.
>>
>> Fair deal I would say :-)
>
> A bit more information would help to compare:
>  - what is your GDAL version?

GDAL 2.1.2

>  - are 504 GeoTIFF files compressed? If yes, which method?

Yes, COMPRESSION=LZW

>  - what are the block dimensions of the input GeoTIFFs?

Size is 36001, 36001  - Block=36001x1
Type=Byte

>  - what kind of GRASS compression did you use?

Default raster + NULL compression enabled. I.e.,

r.compress -p watermask2010
<watermask2010> is compressed (method 2: ZLIB). Data type: CELL
<watermask2010> has a compressed NULL file

Again, the fact that I had to read from an attached storage box likely
slowed down the import.
Just thought to post these numbers here.

markusN