[Landsat-pds] Including Overviews

[Sundwall, Jed]

I’ve done the math, and we can certainly afford to take on an additional 13% of data volume over the next two years, but I’d like to take some more time to do research on demand for this. The approach we’ve taken so far has been to touch the data as little as possible, and beyond the jpg thumbnails we produce, we aren’t creating any derivative products from the raw scenes.

In the meantime, any customer could produce these overlays themselves if they wanted to, but if we hear from multiple customers that they’re doing redundant work to do this, we'll consider adding overviews to this data set.

Open to others’ thoughts on this.

Jed.

On Feb 5, 2015, at 4:41 PM, Peter Becker <pbecker at esri.com<mailto:pbecker at esri.com>> wrote:

Currently the scenes do not include any overviews. This forces applications to read the imagery at full resolution even if accessing at a small scale.
It’s common to include overviews/pyramids/reduced resolution datasets with such imagery.
I would like to recommend that such overviews are included with the imagery

Typically this would increase the file size by approx 33%, which is quite considerable.
The size of the overviews can be reduced by skipping a level (eg use level  4 8 16 32) vs 2 4 8 16 32, alternatively using a factor of 3 ie 3 9 27 81
I tested this out on some images using Deflate with predictor 2
2 4 8 16 32  adds about 35%
4 8 16 32  adds about 9%
3 9 27 81 adds about 13%
My recommendation is to go with the 3 9 27 81 factors. This keeps the factor constant between all levels.
Average sampling would be used, except for the BQA band for which Nearest (else Majority) should be used.

Implementation would only require adding GDALADDO to the existing script

We would also need to determine if we use internal or external overviews.
Internal overviews would increase the size of the files, but not change the directory structure.
Using external overviews (.tif.ovr) would add additional files to the directory, is more specific to GDAL.

As I expect many users to be using GDAL to access these files, I would recommend that we use the external overviews. I don’t see the additional files as a burden.

One other potential concern (and I’m hoping Frank can answer this) is the tile size of the overviews. GDAL appear to always use a tile size (BLOCKSIZE) of 128. It would be more optimum if the file size could be set to 512.

_Peter

_______________________________________________
Landsat-pds mailing list
Landsat-pds at lists.osgeo.org<mailto:Landsat-pds at lists.osgeo.org>
http://lists.osgeo.org/cgi-bin/mailman/listinfo/landsat-pds

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/landsat-pds/attachments/20150206/fdb5e097/attachment.html>