[gdal-dev] Strange reset OGR_INTERLEAVED_READING when using Python bindings to read OSM PBF file

[Even Rouault]

Le mercredi 02 avril 2014 21:21:14, Dražen Odobašić a écrit :
> On 02.04.2014 20:29, Even Rouault wrote:
> > Le mardi 01 avril 2014 19:26:09, Dražen Odobašić a écrit :
> >> Hi all,
> >> 
> >> I've been successfully reading OSM PBF files using GDAL Python bindings,
> >> even files larger than 5Gb. I did have to use
> >> OGR_INTERLEAVED_READING=YES and OSM_USE_CUSTOM_INDEXING=NO, but
> >> eventually OSM PBF is read and I can extract data.
> > 
> > Generally, you shouldn't have to mess with OSM_USE_CUSTOM_INDEXING.
> > 
> > As far as OGR_INTERLEAVED_READING=YES is concerned, it is not sufficient
> > to define the environmenent variable. I invite you to read again
> > "Interleaved reading" in http://gdal.org/ogr/drv_osm.html. The sample
> > C++ code (ported to Python) must be used *exactly*.  I really mean it.
> > That's what ogr2ogr does with OSM data.
> 
> Hi Even,
> 
> this is the Python code that implements interleaved reading, which is more
> or less copied from the c++ example in the docs:
> 
> """
> # prepare for the layer/feature iteration
> nLayerCount = ds.GetLayerCount()
> thereIsDataInLayer = True
> 
> while thereIsDataInLayer:
>     thereIsDataInLayer = False
> 
>     # read data from layers using OGR_INTERLEAVED_READING method
>     for iLayer in xrange(nLayerCount):
>         lyr = ds.GetLayer(iLayer)
> 
>         # read next Feature
>         feat = lyr.GetNextFeature()
>         while(feat is not None):
>             # continue reading features from this layer
>             thereIsDataInLayer = True
> 
>             # do something with the feature
>             # save_feature(feat)
> 
>             # force feature removal - required for INTERLEAVED_READING
>             feat.Destroy()
>             # get the next feature
>             feat = lyr.GetNextFeature()
> """
> 

That looks fine.

> I had no issues when reading large PBF files using this method. For the
> large files, without OSM_USE_CUSTOM_INDEXING=NO I got lots of "Cannot read
> node ..." error messages, and I'm not sure why. Basically I stumbled onto
> OSM_USE_CUSTOM_INDEXING=NO which seems to help. However, in this case
> custom indexing has no effect, script still fails to extract data.

Hum, there might be a bug in custom indexing that you've hit. What is the PBF 
file you are converting ?

And to be sure to understand: when setting OSM_USE_CUSTOM_INDEXING=NO, the 
conversion works ?

> 
> Is there anything else I could try ?
> 
> Dražen

-- 
Geospatial professional services
http://even.rouault.free.fr/services.html