[COG] COG performance best practices?

[Kirill Kouzoubov]

I like the idea of profiles Chris is proposing.

To me there are two major use-cases for COGs with very different needs and
constraints

1. Interactive use by a human analyst
2. Processing by algorithms

Both of these benefit from main concepts of COG: constrained file
structure, use
of tiles. Other features I feel are more geared towards interactive use
case:
embedded overviews, streaming-friendly overview structure.

With so much data being generated by satellites, majority of image
consumption
will be happening by algorithms, not human operators. People will still
look at
various summaries of course, some of those will be in image form, but others
will be a video (think change detection), or a graph (e.g. change in
vegetation
over time for a given region)

While interactive use case is happening over the Internet (limited
bandwidth,
really high network latency, high cost of bandwidth), processing by
algorithms
is in the same data center as the imagery (high bandwidth, much lower
latency,
but still much higher than disk, zero cost bandwidth). These are very
different
environments. Even if you are doing machine learning training on your own
GPUs,
you are most likely better off spinning up an instance to assemble your
training
set in the cloud, massage the data into usable form and download that as one
massive tar file rather than fetching individual files/tiles over the
Internet.

In the second scenario having overviews might not be a cost-effective
option,
you pay for them forever in storage costs. If you ever operate on scales
other
than native, it might be more cost-effective to recompute lower resolution
imagery. If downscaling computation becomes a significant burden you can add
caching, if and when you need it.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/cog/attachments/20180613/50d9b966/attachment-0001.html>