[GRASS-dev] Cloud optimized GeoTIFF support

[Even Rouault]

On dimanche 15 octobre 2017 21:28:24 CEST Markus Neteler wrote:
> On Sun, Oct 15, 2017 at 7:47 PM, Markus Metz
> 
> <markus.metz.giswork at gmail.com> wrote:
> > What do you mean with "a more direct GRASS GIS integration" regarding
> > cloud
> > storage and/or Cloud Optimized GeoTIFF?
> 
> Well, I tought of r.external and r.external.out.
> 
> Markus Metz wrote:
> > Have you tried GDAL's virtual network based file systems [0]?
> 
> ... I didn't yet since traveling all the time. Maybe simple and solved
> but I have no stable connections at time to try out.
> 

Also note that the performance of network accesses depends a lot of how close 
the machine is from the server. For example if you use the VMs of a cloud 
provider in the same region as the bucket you access, the timing measuremetns 
will be 10 times or more better than if you try from a consumer ADSL 
connection.

For Linux & Mac, there are also various projects that use FUSE to mount 
buckets in the file system
https://github.com/kahing/goofys
https://github.com/googlecloudplatform/gcsfuse

I tried quickly goofys and it worked pretty well. It tends to have a more 
aggressive read-ahead strategy than GDAL /vsis3/, which makes reading a lot of 
sequential data faster than with /vsis3/. In latest GDAL trunk, the GeoTIFF 
driver is aware of the network nature of the files and thus can better 
optimize windowed access, so this makes it a bit faster in cases where you 
only access part of the imagery (for example with MapServer that must satisfy 
a WMS request on a BBOX, which was the use case for this recent round of 
optimization, or if you have a processing spread over several VMs with a 
spatial subdivision)

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com