[gdal-dev] OGR slow with network files on Windows

[Radim Blazek]

On Sun, Apr 28, 2013 at 9:26 PM, Even Rouault
<even.rouault at mines-paris.org> wrote:
> Le dimanche 28 avril 2013 20:40:12, Radim Blazek a écrit :
>> Hello,
>> I am working on #6448 QGIS issue - Extremely slow shapefile reading
>> over network. You can find my latest observations in
>> http://hub.qgis.org/issues/6448#note-19
>>
>> Long story short, OGR does not buffer I/O and Windows does not always
>> cache network files.
>>
>> I have few questions for OGR developers:
>> 1. Are there my OGR related notes in the issue comment correct?
>
> Yes. One observation about MITAB not benefiting from VSI_CACHE is that MITAB
> uses the standard VSI API (which is just a thin wrapper around C standard
> FILE* API), whereas the Shapefile driver uses the VSI Virtual File API. And
> VSICache is a virtual file system handled by the VSI Virtual File API.
>
>> 2. Would it be possible to implement buffering in VSI?
>
> For the sake of exhaustivity, I should mention there's a
> VSIBufferedReaderHandle object that implements *backwards* buffering. It is
> currently used by the /vsigzip/ layer to allow backward reading of small
> amounts (gzip streams aren't naturaly seekable. /vsigzip/ has some extension
> support for seeking, but this is still rather slow, hence
> VSIBufferedReaderHandle ) .
>
>> 3. Is it VSI cache (if VSI_CACHE_SIZE is set to some small number)
>> suitable to be used as buffer?
>
> Yes, I think so.

Do you prefer VSI cache for buffering to implementation of another
buffering mechanism as long term solution?

>> 4. Would it be possible to implement buffering or to extend VSI cache
>> for writing (to allow editing)?
>
> Yes, I don't see any reason why VSICache couldn't be extended for writing. It
> should be just a matter of coding it.
>
>> 5. Have you considered to increase CHUNK_SIZE (currently 32768)? Where
>> does that size comes from? I saw that XP is using 32768 but from what
>> I have read, SMB1 block size limit was 64K but SMB2 should not have
>> that limit. E.g. gvSIG 2.0.0 is using via its BigByteBuffer2 class
>> block size 61440.
>
> I have not a particular informed opinion on this. Did you try experimenting
> increasing CHUNK_SIZE to see if this improves performance ?

No, CHUNK_SIZE cannot be set as environment variable and I am using
precompiled osgeo4w. Maybe it is not that important. However, it could
be useful to change it be read from environment variable so that it
would be easy to experiment or to force higher value if necessary.

Radim

> Even