[gdal-dev] Outdated external overviews

[Julien Michel]

Hi Even,

Thanks for your response. I understand all your points, and that Gdal 
requirements usually fall far beyond our limited use cases ... We can of 
course do a lot in the client code (starting with comparing timestamps), 
but this is not specific to our software, I bet Qgis for instance will 
happily use outdated overviews without notice. This is why I am 
wondering if it could not be handled more cleanly at the Gdal level. I 
understand though that this is not an easy problem to fix.

Regards,

Julien

Le 07/10/2016 à 16:46, Even Rouault a écrit :
> Hi Julien,
>> I recently came accross an issue in our software which uses overviews to
>> speed-up navigation in image : the image can change after an external
>> overview has been generated (for instance it has been re-generated by an
>> upstream processing chain with different parameters), this can lead to
>> display bugs or even crash in client code (if for instance the number of
>> bands has changed)
> Are the crashes you mention occuring in GDAL ? If so that should be fixed. From
> a quick try, RasterIO() requests that involve overviews trigger a proper error
> if the .ovr has less bands than the full resolution dataset. Of course if
> client code directly uses GetOverview(), it must be careful to check for NULL
> pointer.
>
>> . Of course this is not a problem for the user that
>> knows what she is doing : simply removing and generating the overview
>> again fixes the problem. But for software that offers overview
>> generation to the end user, this might become an issue, as the software
>> has no clue wether the ovr file is outdated or not.
>>
>> Possible mecanism to prevent this would be :
>>
>> - Check file last modification time between external overview and image
>> file. If image file is newer than overview files, it is probably outdated.
> "Probably" yes :-) But I can imagine workflows where the overviews are not
> necessarily a subsampling of the full resolution dataset, in which case such
> behaviour wouldn't be desirable. I'm also wondering if that wouldn't cause
> problems when datasets are copied. Hopefully timestamps should be preserved
> most of the time, but I guess you could find situations where they are not and
> where the overview could be copied before the main dataset.
>
>> - Encode image checksum in ovr file, and compare it upon loading (might
>> be a bit intensive).
> We definitely don't want to do that on gigabyte sized datasets...
>
>> With those checks, gdal could detect that the ovr file is outdated and
>> simply ignore it. Then client code could be aware that there are
>> actually no overviews for this image and take actions to generate new ones.
> The client code can also use the GDALDataset::GetFileList() API and see if
> there's a .ovr file listed in there. And thus decide to apply a timestamp based
> logic if it whishes.
>
> Or one could imagine to put that logic into GDAL itself, but I think that
> should be an option explicitly set.
>
> There would be a subtelty also. Imagine that the external overview is
> outdated, and you want to update it, but not recreate it from scratch. For
> example if you know which area has been updated in full resolution dataset.
> Then you'd want the overview to be still accessible. So hiding the overview
> should only apply for datasets opened in read-only mode.
>
>> PS: This of course does not apply if overviews are internal
> You probably meant that detecting that overviews are outdated is even less
> doable than for the external case, but that situation can still happen.
>
> Even
>
>


-- 
Julien MICHEL
CNES - DCT/SI/AP - BPI 1219
18, avenue Edouard Belin
31401 Toulouse Cedex 09 - France
Tel: +33 561 282 894 - Fax: +33 561 283 109