[gdal-dev] Cannot read overviews if VSI cache is on

[Radim Blazek]

Hi Even,

thanks for quick response and exhaustive information.

On Fri, Jul 26, 2013 at 2:37 PM, Even Rouault
<even.rouault at mines-paris.org> wrote:
> Selon Radim Blazek <radim.blazek at gmail.com>:
>
>> GDAL 1.9.2 fails to read (large?) overview files if
>>     VSI_CACHE=TRUE
>>     VSI_CACHE_SIZE=1000000
>> The problem is described here: http://hub.qgis.org/issues/8356.
>> I just guess, that overview file keeps overview headers scattered over
>> the file and VSI fails to set offset, 17891712530 in this case, over
>> some limit.
>>
>> Could it be some hardcoded limit in VSI subsystem? Or just int used
>> for offset instead of long? Is it easy to fix? Should I create a new
>> ticket?
>
> Hi Radim,
>
> yes, you likely spotted a bug in the VSI Cache mechanism. Please file a GDAL
> ticket.
>
> I've just done some quick code inspection in port/cpl_vsil_cache.cpp and I
> suspect that the error comes from the signature of
>
> int VSICachedFile::LoadBlocks( size_t nStartBlock, size_t nBlockCount,
>                                void *pBuffer, size_t nBufferSize );
>
> and a few other uses of size_t in VSICachedFile::Read().
>
> The size_t nStartBlock should likely be a vsi_l_offset. I guess the issue is met
> on a 32bit GDAL build ? On 64 bit, the error must not be hit as size_t will be
> 64 bit.
>
>
>>
>> Until it gets fixed in GDAL, how can we easily enable the VSI cache
>> for OGR and disable it for GDAL?
>
> Well, you could try that at each place where you GDALOpen():
>
> /* yes strdup it since SetThreadLocalConfigOption() might invalidate the value
> returned by CPLGetConfigOption() */
> char* pszOldVal = CPLStrdup(CPLGetConfigOption("VSI_CACHE", "FALSE"));
> CPLSetThreadLocalConfigOption("VSI_CACHE", "FALSE");
> GDALDataset hDS = GDALOpen(xxxx);
> CPLSetThreadLocalConfigOption("VSI_CACHE", pszOldVal);
> CPLFree(pszOldVal);
>
> I'd suggest doing a wrapper function for the above, in case GDALOpen() is used
> several times in QGIS code base...
>
> However.... in some cases, the opening of GDAL datasets is differed. This is the
> case for a VRT. If you open a VRT, GDAL opens the .vrt file, but doesn't open
> immediately the tiles that compose it. So the above might not work reliably. Or
> you might need to wrap GDALRasterIO() similarly, and potentially other GDAL API
> that can indirectly lead to a GDALOpen() (statistics, histograms, ...)

But the problem only happens with OVR file which is too big, the tiles
are OK. I believe that OVR is opened together with VRT, (GDALOpen() is
called form GDALOpen()?) so it should work just with GDALOpen()
wrapper?

Radim

> Best regards,
>
> Even
>
>