[gdal-dev] Cannot read overviews if VSI cache is on

[Even Rouault]

Selon Radim Blazek <radim.blazek at gmail.com>:

> GDAL 1.9.2 fails to read (large?) overview files if
>     VSI_CACHE=TRUE
>     VSI_CACHE_SIZE=1000000
> The problem is described here: http://hub.qgis.org/issues/8356.
> I just guess, that overview file keeps overview headers scattered over
> the file and VSI fails to set offset, 17891712530 in this case, over
> some limit.
>
> Could it be some hardcoded limit in VSI subsystem? Or just int used
> for offset instead of long? Is it easy to fix? Should I create a new
> ticket?

Hi Radim,

yes, you likely spotted a bug in the VSI Cache mechanism. Please file a GDAL
ticket.

I've just done some quick code inspection in port/cpl_vsil_cache.cpp and I
suspect that the error comes from the signature of

int VSICachedFile::LoadBlocks( size_t nStartBlock, size_t nBlockCount,
                               void *pBuffer, size_t nBufferSize );

and a few other uses of size_t in VSICachedFile::Read().

The size_t nStartBlock should likely be a vsi_l_offset. I guess the issue is met
on a 32bit GDAL build ? On 64 bit, the error must not be hit as size_t will be
64 bit.


>
> Until it gets fixed in GDAL, how can we easily enable the VSI cache
> for OGR and disable it for GDAL?

Well, you could try that at each place where you GDALOpen():

/* yes strdup it since SetThreadLocalConfigOption() might invalidate the value
returned by CPLGetConfigOption() */
char* pszOldVal = CPLStrdup(CPLGetConfigOption("VSI_CACHE", "FALSE"));
CPLSetThreadLocalConfigOption("VSI_CACHE", "FALSE");
GDALDataset hDS = GDALOpen(xxxx);
CPLSetThreadLocalConfigOption("VSI_CACHE", pszOldVal);
CPLFree(pszOldVal);

I'd suggest doing a wrapper function for the above, in case GDALOpen() is used
several times in QGIS code base...

However.... in some cases, the opening of GDAL datasets is differed. This is the
case for a VRT. If you open a VRT, GDAL opens the .vrt file, but doesn't open
immediately the tiles that compose it. So the above might not work reliably. Or
you might need to wrap GDALRasterIO() similarly, and potentially other GDAL API
that can indirectly lead to a GDALOpen() (statistics, histograms, ...)

Best regards,

Even