[gdal-dev] OGR slow with network files on Windows

[Even Rouault]

Le dimanche 28 avril 2013 20:40:12, Radim Blazek a écrit :
> Hello,
> I am working on #6448 QGIS issue - Extremely slow shapefile reading
> over network. You can find my latest observations in
> http://hub.qgis.org/issues/6448#note-19
> 
> Long story short, OGR does not buffer I/O and Windows does not always
> cache network files.
> 
> I have few questions for OGR developers:
> 1. Are there my OGR related notes in the issue comment correct?

Yes. One observation about MITAB not benefiting from VSI_CACHE is that MITAB 
uses the standard VSI API (which is just a thin wrapper around C standard 
FILE* API), whereas the Shapefile driver uses the VSI Virtual File API. And 
VSICache is a virtual file system handled by the VSI Virtual File API.

> 2. Would it be possible to implement buffering in VSI?

For the sake of exhaustivity, I should mention there's a 
VSIBufferedReaderHandle object that implements *backwards* buffering. It is 
currently used by the /vsigzip/ layer to allow backward reading of small 
amounts (gzip streams aren't naturaly seekable. /vsigzip/ has some extension 
support for seeking, but this is still rather slow, hence 
VSIBufferedReaderHandle ) .

> 3. Is it VSI cache (if VSI_CACHE_SIZE is set to some small number)
> suitable to be used as buffer?

Yes, I think so.

> 4. Would it be possible to implement buffering or to extend VSI cache
> for writing (to allow editing)?

Yes, I don't see any reason why VSICache couldn't be extended for writing. It 
should be just a matter of coding it.

> 5. Have you considered to increase CHUNK_SIZE (currently 32768)? Where
> does that size comes from? I saw that XP is using 32768 but from what
> I have read, SMB1 block size limit was 64K but SMB2 should not have
> that limit. E.g. gvSIG 2.0.0 is using via its BigByteBuffer2 class
> block size 61440.

I have not a particular informed opinion on this. Did you try experimenting 
increasing CHUNK_SIZE to see if this improves performance ?

Even