[gdal-dev] [GRASS-user] Slow import of GHSL

[Nikos Alexandris]

[..]

Nikos:

>>>> Some messy rough timings:
>>>> 1) i7, 8 cores, 32GB RAM, Base OS: CentOS -> Three r.in.gdal processes
>>>> for "p2.tif", each stuck at 3% for almost 14h
>>>> 2) Xeon, 24 Cores, 32GB RAM, Base OS: Windows -> Three gdal_translate
>>>> processes with -projwin, the VRT file as an input and GeoTIFF as output,
>>>> at 40% since yesterday afternoon
>>>> 3) Xeon, 12 Cores, ? RAM, Base OS: CentOS.jpg -> Same processes as in
>>>> 1), stuck at 0% of progress for more than 16h.
>>>> SSD can be seen as a "necessity".

Markus M:

>>> Hmm, not really.

Nikos:

>> In a laptop (i7-4600U CPU @ 2.10GHz with 8GB of RAM with SSD) it was
>> progressing, in a quite acceptable manner.

Markus M:

> What is the gdal version you used? I use gdal 2.1.3.

Well, yes! 2.1.3 in the laptop, 1.11.4 for the rest.

>>  I had to break the process,
>> unfortunately, because I don't have a lot of free space :-/
>
>maybe because you forgot the enable compression ;-)

I should!

>>> With the p1 tif and GRASS db on the same spinning HDD, and
>>> 6 other heavy processes constantly reading from and writing to that same
>>> HDD, r.in.gdal took 2h 13min to import the p1 tif. 360 MB as input and 1.5
>>> GB as output is not that heavy on disk IO. Most of the time is spent
>>> decompressing input and compressing output.

Is it an 10000rpm disk?

>> p2 is a harder one!
>
>export GDAL_CACHEMAX=10000
>gdal_translate -co "COMPRESS=LZW"
>GHS_BUILT_LDS1990_GLOBE_R2016A_3857_38_v1_0_p2.tif p2_test.tif


I did not emphasize it enough, but cache size was among my questions
initially.  I wrongly assumed that it can't be more than 2047 due to the
reference in <https://grass.osgeo.org/grass72/manuals/r.in.gdal.html>:

--%<---
memory=integer
  ..
  Options: 0-2047
  ..
--->%--

I admit I did not head over to
https://trac.osgeo.org/gdal/wiki/ConfigOptions from where it is implied
that it can be much higher than 2047MB.

Can't r.in.gdal deal with memory=4096 for example (will try)? If yes,
can we update the manual(s)?

Also related?  GTIFF_DIRECT_IO, GTIFF_VIRTUAL_MEM_IO


>finishes in 28 minutes.

Impressive!

>you could try gdal 2.1.3, maybe 2.1.3 has a more efficient cache regarding
>block-wise reading than gdal 1.11.4

Yes, I have to.

Kudos, Nikos