[GRASS-user] Slow import of GHSL

[Markus Metz]

On Wed, Mar 15, 2017 at 6:03 PM, Nikos Alexandris <nik at nikosalexandris.net>
wrote:
>
[...]
>
> Nikos:
>
>>> Some messy rough timings:
>>>
>>> 1) i7, 8 cores, 32GB RAM, Base OS: CentOS -> Three r.in.gdal processes
>>> for "p2.tif", each stuck at 3% for almost 14h
>>>
>>> 2) Xeon, 24 Cores, 32GB RAM, Base OS: Windows -> Three gdal_translate
>>> processes with -projwin, the VRT file as an input and GeoTIFF as output,
>>> at 40% since yesterday afternoon
>>>
>>> 3) Xeon, 12 Cores, ? RAM, Base OS: CentOS.jpg -> Same processes as in
>>> 1), stuck at 0% of progress for more than 16h.
>>>
>>> SSD can be seen as a "necessity".
>>
>>
> Markus Metz:
>
>> Hmm, not really.
>
>
> In a laptop (i7-4600U CPU @ 2.10GHz with 8GB of RAM with SSD) it was
> progressing, in a quite acceptable manner.
 What is the gdal version you used? I use gdal 2.1.3.

>  I had to break the process,
> unfortunately, because I don't have a lot of free space :-/

maybe because you forgot the enable compression ;-)

>
>> With the p1 tif and GRASS db on the same spinning HDD, and
>> 6 other heavy processes constantly reading from and writing to that same
>> HDD, r.in.gdal took 2h 13min to import the p1 tif. 360 MB as input and
1.5
>> GB as output is not that heavy on disk IO. Most of the time is spent
>> decompressing input and compressing output.
>
>
> p2 is a harder one!

export GDAL_CACHEMAX=10000
gdal_translate -co "COMPRESS=LZW"
GHS_BUILT_LDS1990_GLOBE_R2016A_3857_38_v1_0_p2.tif p2_test.tif

finishes in 28 minutes.

you could try gdal 2.1.3, maybe 2.1.3 has a more efficient cache regarding
block-wise reading than gdal 1.11.4

Best,

Markus M
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-user/attachments/20170315/c1d411fa/attachment.html>