[gdal-dev] Can OGR WFS driver take advantage of WFS Server streaming?

[Even Rouault]

Le jeudi 17 mai 2012 00:25:52, Yingqi Tang a écrit :
> Thanks for confirming it, Even. I have two follow up questions though:
> 
> 1. Does the GML driver (reading local big GML file) also load the whole GML
> in memory too?

No, it will read it by small chunks. If no matching XML schema is available 
(or that the parser of the GML driver cannot manage to understand it because 
it is too complex (when the GML data is not limited to the Simple Feature 
convention)), there will be an initial full scan of the file to determine the 
schema of features.

> 2. Does the GML driver take url pointing to a GML?

It can work with /vsicurl/http://something, provided that the server allows 
partial downloading of arbitrary ranges of bytes within the file, which is 
rarely the case with GetFeature() requests.

In theory the WFS driver could be improved to use the capability of the GML 
driver to read chunk by chunk, but there is some work to make that happen. 
This would probably require a /vsicurl_streaming/ virtual file system where 
basically only Read() requests would be available, thus not requiring the 
server to have download range capabilities.

> 
> Thanks
> 
> On Wed, May 16, 2012 at 3:20 PM, Even Rouault
> 
> <even.rouault at mines-paris.org>wrote:
> > Le mercredi 16 mai 2012 23:58:29, Yingqi Tang a écrit :
> > > Hi all,
> > > 
> > > I am communicating against a WFS server which supports data (GML)
> > > streaming, which means that once I send a GetFeature quest for a big
> > > GML (2.0GB+), server will take only seconds to start streaming GML
> > > back to client while at the same time it's still preparing the rest of
> > > the big
> > 
> > GML.
> > 
> > > Now I wonder if GDAL/OGR WFS client can take advantage of that? Will I
> > > be able to access the first few features while the streaming is still
> > > going on? It seems to me that it doesn't because it always waiting the
> > > whole
> > 
> > big
> > 
> > > GML response is loaded in memory.
> > 
> > Yes indeed, the OGR WFS driver waits for the whole response to be
> > downloaded
> > before parsing it. It could be desirable to have on-the-fly parsing
> > indeed, but
> > we currently lack the capability of doing it with the way we use the Curl
> > library.
> > 
> > > Thanks