[COG] COG performance best practices?

[Seth Fitzsimmons]

For OpenAerialMap and various contract projects (including work with DEMs),
I've formalized my COG generation settings in
https://github.com/mojodna/marblecutter-tools.
https://github.com/mojodna/marblecutter-tools/blob/master/bin/transcode.sh does
the bulk of the heavy lifting, so if you're curious, that's the script to
check out. It's a total work in progress (yes, it needs a README ;-), so if
anything jumps out at anyone as an obvious improvement, please let me know!

To summarize:

* block size (internal tiling) is 512x512
* sources are kept in their original CRS
* 4-band, 8-bit sources are assumed to be RGBA; band 4 (if present) is
assumed to be an alpha channel and is extracted as the mask. RGB bands are
converted to the YCbCr colorspace and JPEG-compressed w/ default quality
settings. Pixel interleaving is used.
* floating-point sources are compressed using DEFLATE with a floating point
predictor (3 for GDAL)
* all other sources are compressed using DEFLATE with a horizontal
predictor (2 for GDAL)
* internal overviews are generated with the same settings and produced for
factors of 2 until the overview is smaller than the block size (readers
shouldn't gain much benefit beyond that even when rendering at very low
resolutions)

BigTIFFs should be generated when necessary; since compression is in play,
`BIGTIFF=IF_SAFER` is used (we added this recently after encountering a
source within the threshold that GDAL's default heuristic didn't pick
BigTIFF).

External masks (.msk sidecar) are currently created; this isn't ideal, but
there were a couple driving reasons for doing this in the past (which may
no longer be accurate, particularly after some recent improvements to
rasterio):

* when read as masks by rasterio, [mask] overviews weren't used
* masks should be resampled using nearest-neighbor to preserve edge
crispness, even when other resampling methods (lanczos, bicubic, etc) are
used for imagery bands

I've compared data from Planet with re-transcoded sources using this
pipeline; Planet's compression ratios are better, potentially because they
use LZW rather than DEFLATE (and potentially a higher ZLEVEL; though I just
changed that in marblecutter-tools).

seth

On Tue, Jun 12, 2018 at 9:26 AM Chris Holmes <cholmes at planet.com> wrote:

> So one thing that many people new to COG wonder about is how well the
> different options perform. The spec leaves quite a bit open - compression,
> tile size, exact number of overviews, projection, etc. I know there's a lot
> of experience on this list, so I was wondering if people can share what is
> working well for them, including advanced optimizations that may be more
> specific to individual use cases, like serving web tiles or accessing large
> temporal stacks.
>
> My hope is that from sharing these best practices we can start to get
> towards an idea of 'profiles' for COG's, or at least a set of solid
> recommendations for different use cases.
>
> There's been some nice work done looking in to performance, in the original
>  COG wiki page
> <https://trac.osgeo.org/gdal/wiki/CloudOptimizedGeoTIFF#Performancetesting>,
> and I also just came across this nice report
> <https://github.com/opendatacube/benchmark-rio-s3/blob/master/report.md> from
> some s3 gdal benchmarking tools
> <https://github.com/opendatacube/benchmark-rio-s3> from opendatacube. If
> anyone has other good tools or reports please share them here, or just
> write up your experiences.
>
> I'm also interested in comparisons to other formats (jp2k / mrf / ecw),
> along with tooling for performance testing (which we might gather in a
> repo), but I think I'll start another thread on this.
>
> _______________________________________________
> COG mailing list
> COG at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/cog
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/cog/attachments/20180612/866e92fa/attachment.html>