[GRASS-user] Slow import of GHSL

[Markus Metz]

On Sat, Mar 11, 2017 at 8:53 AM, Nikos Alexandris <nik at nikosalexandris.net>
wrote:
>
> Nikos Alexandris
>
>>> Why does (attempting to) import a 38m pixel resolution GHSL [0] GeoTIFF
>>> layer, ie GHS_BUILT_LDS1990_GLOBE_R2016A_3857_38_v1_0_p1.tif, in GRASS'
>>> db progress slow?

because it is a very large raster map: Size is 507904, 647168
>
>
> (Apologies for cross-posting to gdal-dev)
>
> Markus Neteler:
>
>> Can you elaborate a bit more? I have downloaded and checked:
>>
>> That is 9835059101  bytes in 19885 files or I downloaded the wrong one
>> (please post an URL).
>
> For example <http://ghsl.jrc.ec.europa.eu/ghs_bu.php>,
>
> see
>
> GHS_BUILT_LDS1975_GLOBE_R2016A_3857_38 (768MB)
GHS_BUILT_LDS1990_GLOBE_R2016A_3857_38 (854MB)
GHS_BUILT_LDS2000_GLOBE_R2016A_3857_38 (892MB)
GHS_BUILT_LDS2014_GLOBE_R2016A_3857_38 (900MB)
>
> "3857" is the EPSG code.  They are split in two GeoTIFFs (p1, p2) and
> there is a VRT along with overviews for it.  No overviews for the TIFFs.
>
> For example:
>
> GHSL_data_access_v1.3.pdf
> GHS_BUILT_LDS1990_GLOBE_R2016A_3857_38_v1_0.clr
> GHS_BUILT_LDS1990_GLOBE_R2016A_3857_38_v1_0.vrt
> GHS_BUILT_LDS1990_GLOBE_R2016A_3857_38_v1_0.vrt.ovr
> GHS_BUILT_LDS1990_GLOBE_R2016A_3857_38_v1_0_p1.tif
> GHS_BUILT_LDS1990_GLOBE_R2016A_3857_38_v1_0_p2.tif
>
>
> Even trying to clip, with gdal_translate, might create file(s) of
> hundreds of GBs. This might be due to missing compression.

then use compression. The source tiffs use LZW with blocks of 4096x4096
cells.

>
> The import of p1 or p2 or of the VRT file in GRASS' data base, via
> r.in.gdal/r.import, does not progress at all.

Importing GHS_BUILT_LDS1990_GLOBE_R2016A_3857_38_v1_0_p1.tif with r.in.gdal
took 1:31 hours on a laptop with SSD. The resultant cell file was 1.5 GB.

Recompressing with BZIP2 took 2:20 hours and the size of the cell file was
reduced to a mere 143 MB.

> (
> Side question: why is max 2047?  What if there is a lot more of RAM?
> )
To avoid integer overflow because 2047 is converted to bytes with 2047 *
1024 * 1024.

Markus M
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-user/attachments/20170311/c56fa3f6/attachment-0001.html>