[Landsat-pds] COG Format discussion

[Flasher, Joe]

I’m interested in joining this BOF if it happens!

On May 13, 2018, at 11:05 PM, Chris Holmes <cholmes at planet.com<mailto:cholmes at planet.com>> wrote:

Two semi-related things:

- It could be good for us to set up a COG format list? Does anyone have osgeo list creation powers? If not I can just make a google group. We've been playing with -stats in COG's to give hints to tile services about analytic data, would be good to discuss adding it as a cog 'recommendation', and indeed perhaps making more recommendations / possibly making 'tiers' of COG for some of the more extreme optimizations.

- Anyone up for a cog 'birds of feather' session at foss4g? I'll try to sign up tomorrow, curious if anyone has preferences for monday vs tuesday.



On Tue, Apr 17, 2018 at 12:54 AM, Even Rouault <even.rouault at spatialys.com<mailto:even.rouault at spatialys.com>> wrote:
> I'd be interested in whether webp is believed to have good enough
> performance that it would be a substantial improvement over the Deflate
> support already available in libtiff.  I'm also interested in how it
> compares to the new zstd support (
> https://github.com/OSGeo/gdal/commit/1c60366a193e67ee90856e1008e3c17cb8524f6
> 0#diff-ce45424050585add924746240ffc2761). I'm willing to support new codecs
> in libtiff if they add significant value, but I don't want to just add
> every compression format known.

You can only compare webp vs deflate or zstd for its lossless profile.
The WebP website provides this comparison against PNG:
https://developers.google.com/speed/webp/docs/webp_lossless_alpha_study
so claiming a 42% size improvement over PNG

For lossy support, if you believe Mozilla (pushing for MozJpeg)
https://research.mozilla.org/2014/07/15/mozilla-advances-jpeg-encoding-with-mozjpeg-2-0/
rather than Google
https://developers.google.com/speed/webp/docs/webp_study
"We consider this study to be inconclusive when it comes to the question of whether
"WebP and/or JPEG XR outperform JPEG by any significant margin"

One potential advantage of webp is that lossy webp for imagery with lossless alpha would
be naturally supported (in comparison to the second point below).
One drawback of webp is that it is really RGB[A] only (would likely be unwise to use
it for 3 or 4 band image with other photometric interpretations)

>
> > - Improve mask storing inside COG. For some technical reason, when you
> > create a COG with internal masking, the mask is appended to the end of the
> > IFD. Some of improvement could be to append mask TILE data just after the
> > imagery tile data.

>
> I assume you are suggesting a file with the "nodata" handled as a distinct
> IFD like this (our internal serving format):

That was I understood from previous discussion with Vincent's team.

> That is interesting.  Even, can you comment on what it would take to ensure
> extra mask IFDs are located near their corresponding imagery as part of
> GDAL and the implications for COG?

Such a layout of blocks would apply only for Planar Configuration==single image plane
That's certainly doable by the GDAL GTiff driver, but would require the copying
of imagery to be done in a custom fashion, to properly interleave imagery blocks with
mask blocks, instead of using GDALDatasetCopyWholeRaster() (for imagery) and
GDALRasterBandCopyWholeRaster() (for masks). Actually, implementation wise,
that could be a new option of GDALDatasetCopyWholeRaster (INTERLEAVE_MASK=YES).
For very large images (dimensions > 100,000 pixels) where the size of the
TileOffsets and TileByteCounts tags is big this constant back and forth between IFD
could be rather costly, as they are reloaded/flushed each time you change the active
IFD in libtiff.

The COG definition would have to be updated for that use case (probably as an
allowed extra optimization, rather than forcing people to use it), and the validation
script as well.

On the GDAL read side, the GTiff driver would also have to be updated to detect this
layout and when reading a block of imagery, it should issue a GET range request that
is big enough to fetch the imagery block and its mask block at once (the base logic to
optimize GET requests for a given IRasterIO() request is already in place)

Even


--
Spatialys - Geospatial professional services
http://www.spatialys.com<http://www.spatialys.com/>
_______________________________________________
Landsat-pds mailing list
Landsat-pds at lists.osgeo.org<mailto:Landsat-pds at lists.osgeo.org>
https://lists.osgeo.org/mailman/listinfo/landsat-pds

_______________________________________________
Landsat-pds mailing list
Landsat-pds at lists.osgeo.org<mailto:Landsat-pds at lists.osgeo.org>
https://lists.osgeo.org/mailman/listinfo/landsat-pds

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/landsat-pds/attachments/20180514/c892b541/attachment-0001.html>