[GRASS-user] Slow import of GHSL

[Markus Metz]

On Wed, Mar 22, 2017 at 8:12 PM, Markus Neteler <neteler at osgeo.org> wrote:
>
> On Sat, Mar 11, 2017 at 7:01 PM, Markus Metz
> <markus.metz.giswork at gmail.com> wrote:
> > On Sat, Mar 11, 2017 at 8:53 AM, Nikos Alexandris <
nik at nikosalexandris.net>
> > wrote:
> >>
> >> Nikos Alexandris
> >>
> >>>> Why does (attempting to) import a 38m pixel resolution GHSL [0]
GeoTIFF
> >>>> layer, ie GHS_BUILT_LDS1990_GLOBE_R2016A_3857_38_v1_0_p1.tif, in
GRASS'
> >>>> db progress slow?
> >
> > because it is a very large raster map: Size is 507904, 647168
>
> Nikos, for an even bigger map try
>
> Global Surface Water (2000-2012, 30 m, Data coverage is from 80° north
> to 60° south):
http://landcover.usgs.gov/glc/WaterDescriptionAndDownloads.php
> by USGS. 1.6GB in size.
>
> Using gdalbuildvrt I created a VRT from the 504 GeoTIFF files.
>
> After import into GRASS GIS, here the timings:
>
> # final map size:
> g.region -p
> ...
> rows:       493200
> cols:       1296001
> cells:      639187693200
>
> (handling only works in GRASS GIS 7.3.svn since Markus Metz's recent
> improvements on global data import are needed).

(my changes were bug fixes, not improvements)

>
> Benchmarks:
> - Import took 2h while reading the data from a CIFS mounted storage
> box (slow) and writing on SSD.
> - Displaying the entire map (639 giga-pixel) in GRASS GIS' display
> (d.mon) took ~15 sec over a ssh tunnel from my laptop to the server,
> since I am at a conference.
>
> Fair deal I would say :-)

A bit more information would help to compare:
 - what is your GDAL version?
 - are 504 GeoTIFF files compressed? If yes, which method?
 - what are the block dimensions of the input GeoTIFFs?
 - what kind of GRASS compression did you use?

Markus M
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-user/attachments/20170322/77ca97f6/attachment.html>