[GRASS-user] Slow import of GHSL

[Markus Metz]

On Wed, Mar 22, 2017 at 9:52 PM, Markus Neteler <neteler at osgeo.org> wrote:
>
> On Wed, Mar 22, 2017 at 9:28 PM, Markus Metz
> <markus.metz.giswork at gmail.com> wrote:
> > On Wed, Mar 22, 2017 at 8:12 PM, Markus Neteler <neteler at osgeo.org>
wrote:
> ...
> >> Nikos, for an even bigger map try
> >>
> >> Global Surface Water (2000-2012, 30 m, Data coverage is from 80° north
> >> to 60° south):
> >> http://landcover.usgs.gov/glc/WaterDescriptionAndDownloads.php
> >> by USGS. 1.6GB in size.
> >>
> >> Using gdalbuildvrt I created a VRT from the 504 GeoTIFF files.
> >>
> >> After import into GRASS GIS, here the timings:
> >>
> >> # final map size:
> >> g.region -p
> >> ...
> >> rows:       493200
> >> cols:       1296001
> >> cells:      639187693200
> >>
> >> (handling only works in GRASS GIS 7.3.svn since Markus Metz's recent
> >> improvements on global data import are needed).
> >
> > (my changes were bug fixes, not improvements)
> >
> >>
> >> Benchmarks:
> >> - Import took 2h while reading the data from a CIFS mounted storage
> >> box (slow) and writing on SSD.
> >> - Displaying the entire map (639 giga-pixel) in GRASS GIS' display
> >> (d.mon) took ~15 sec over a ssh tunnel from my laptop to the server,
> >> since I am at a conference.
> >>
> >> Fair deal I would say :-)
> >
> > A bit more information would help to compare:
> >  - what is your GDAL version?
>
> GDAL 2.1.2
>
> >  - are 504 GeoTIFF files compressed? If yes, which method?
>
> Yes, COMPRESSION=LZW
>
> >  - what are the block dimensions of the input GeoTIFFs?
>
> Size is 36001, 36001  - Block=36001x1

This is row by row compression as in GRASS. That could help import with
r.in.gdal which also reads and writes row by row.

> Type=Byte
>
> >  - what kind of GRASS compression did you use?
>
> Default raster + NULL compression enabled. I.e.,
>
> r.compress -p watermask2010
> <watermask2010> is compressed (method 2: ZLIB). Data type: CELL

You might save disk space at the cost of longer reading times with BZIP2.

> <watermask2010> has a compressed NULL file
>
> Again, the fact that I had to read from an attached storage box likely
> slowed down the import.
> Just thought to post these numbers here.

Impressive that such a large raster can be imported at all, and relatively
fast!

Reading about 1.6 GB (also from an attached storage box) should not take 2
hours, therefore I think the limit is software input decompression and
output compression.

Markus M
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-user/attachments/20170322/0ae9b675/attachment.html>