[GRASS-dev] Cloud optimized GeoTIFF support

[Markus Metz]

On Mon, Oct 16, 2017 at 9:10 AM, Markus Neteler <neteler at osgeo.org> wrote:
>
> On Oct 15, 2017 9:50 PM, "Markus Metz" <markus.metz.giswork at gmail.com>
wrote:
> > On Sun, Oct 15, 2017 at 9:28 PM, Markus Neteler <neteler at osgeo.org>
wrote:
> > >
> > > On Sun, Oct 15, 2017 at 7:47 PM, Markus Metz
> > > <markus.metz.giswork at gmail.com> wrote:
> > > > What do you mean with "a more direct GRASS GIS integration"
regarding cloud
> > > > storage and/or Cloud Optimized GeoTIFF?
> > >
> > > Well, I tought of r.external and r.external.out.
> >
> > OK, rephrasing my question:
> >
> > What do you mean with "a more direct GRASS GIS integration" regarding
cloud storage and/or Cloud Optimized GeoTIFF, and regarding reading and
writing such data?
>
> So, my need is to process EO data closest possible where they are stored.

That sounds like you don't need a more direct GRASS GIS integration of
cloud storage, but instead are more direct integration of a GRASS
installation into cloud storage.

> Imagine a huge object storage or bucket or the like which is surrounded
by cloud compute instances. Given the data size between some Terabytes up
to Petabytes (e.g. Sentinel) I need to minimize whatever data transfer or
extra (format) conversion.

Assuming a local GRASS db, data transfer increases with remote input to
r.external or a remote directory as output for r.external.out.
If you want to reduce data transfer, use r.in.gdal/r.out.gdal.
If you want to reduce format conversions, use r.external/r.external.out.

>
> Even's suggestion to use FUSE to mount
> buckets in the file system will be something to try.
>
> However, the location concept is also causing some friction here:
Sentinel data are stored in many different zones i.e. would result in many
different locations.

Would gdalwarp help?

Markus M

> I have no solution to that yet but this issue  plus the need to minimize
bandwidth usage is the key for a successful cloud storage based processing.
>
> So I started with thinking about how to connect to current EO storages.
>
> Best.
> markusN
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-dev/attachments/20171016/42834632/attachment.html>