[gdal-dev] Outdated external overviews

[Even Rouault]

Hi Julien,
> 
> I recently came accross an issue in our software which uses overviews to
> speed-up navigation in image : the image can change after an external
> overview has been generated (for instance it has been re-generated by an
> upstream processing chain with different parameters), this can lead to
> display bugs or even crash in client code (if for instance the number of
> bands has changed)

Are the crashes you mention occuring in GDAL ? If so that should be fixed. From 
a quick try, RasterIO() requests that involve overviews trigger a proper error 
if the .ovr has less bands than the full resolution dataset. Of course if 
client code directly uses GetOverview(), it must be careful to check for NULL 
pointer.

> . Of course this is not a problem for the user that
> knows what she is doing : simply removing and generating the overview
> again fixes the problem. But for software that offers overview
> generation to the end user, this might become an issue, as the software
> has no clue wether the ovr file is outdated or not.
> 
> Possible mecanism to prevent this would be :
> 
> - Check file last modification time between external overview and image
> file. If image file is newer than overview files, it is probably outdated.

"Probably" yes :-) But I can imagine workflows where the overviews are not 
necessarily a subsampling of the full resolution dataset, in which case such 
behaviour wouldn't be desirable. I'm also wondering if that wouldn't cause 
problems when datasets are copied. Hopefully timestamps should be preserved 
most of the time, but I guess you could find situations where they are not and 
where the overview could be copied before the main dataset.

> 
> - Encode image checksum in ovr file, and compare it upon loading (might
> be a bit intensive).

We definitely don't want to do that on gigabyte sized datasets...

> 
> With those checks, gdal could detect that the ovr file is outdated and
> simply ignore it. Then client code could be aware that there are
> actually no overviews for this image and take actions to generate new ones.

The client code can also use the GDALDataset::GetFileList() API and see if 
there's a .ovr file listed in there. And thus decide to apply a timestamp based 
logic if it whishes. 

Or one could imagine to put that logic into GDAL itself, but I think that 
should be an option explicitly set.

There would be a subtelty also. Imagine that the external overview is 
outdated, and you want to update it, but not recreate it from scratch. For 
example if you know which area has been updated in full resolution dataset. 
Then you'd want the overview to be still accessible. So hiding the overview 
should only apply for datasets opened in read-only mode.

> 
> PS: This of course does not apply if overviews are internal

You probably meant that detecting that overviews are outdated is even less 
doable than for the external case, but that situation can still happen.

Even


-- 
Spatialys - Geospatial professional services
http://www.spatialys.com