[gdal-dev] Strange reset OGR_INTERLEAVED_READING when using Python bindings to read OSM PBF file

[Dražen Odobašić]

On 02.04.2014 20:29, Even Rouault wrote:
> Le mardi 01 avril 2014 19:26:09, Dražen Odobašić a écrit :
>> Hi all,
>>
>> I've been successfully reading OSM PBF files using GDAL Python bindings,
>> even files larger than 5Gb. I did have to use OGR_INTERLEAVED_READING=YES
>> and OSM_USE_CUSTOM_INDEXING=NO, but eventually OSM PBF is read and I can
>> extract data.
> 
> Generally, you shouldn't have to mess with OSM_USE_CUSTOM_INDEXING.
> 
> As far as OGR_INTERLEAVED_READING=YES is concerned, it is not sufficient to 
> define the environmenent variable. I invite you to read again "Interleaved 
> reading" in http://gdal.org/ogr/drv_osm.html. The sample C++ code (ported to 
> Python) must be used *exactly*.  I really mean it. That's what ogr2ogr does 
> with OSM data.
> 

Hi Even,

this is the Python code that implements interleaved reading, which is more or
less copied from the c++ example in the docs:

"""
# prepare for the layer/feature iteration
nLayerCount = ds.GetLayerCount()
thereIsDataInLayer = True

while thereIsDataInLayer:
    thereIsDataInLayer = False

    # read data from layers using OGR_INTERLEAVED_READING method
    for iLayer in xrange(nLayerCount):
        lyr = ds.GetLayer(iLayer)

        # read next Feature
        feat = lyr.GetNextFeature()
        while(feat is not None):
            # continue reading features from this layer
            thereIsDataInLayer = True

            # do something with the feature
            # save_feature(feat)

            # force feature removal - required for INTERLEAVED_READING
            feat.Destroy()
            # get the next feature
            feat = lyr.GetNextFeature()
"""

I had no issues when reading large PBF files using this method. For the large
files, without OSM_USE_CUSTOM_INDEXING=NO I got lots of "Cannot read node ..."
error messages, and I'm not sure why. Basically I stumbled onto
OSM_USE_CUSTOM_INDEXING=NO which seems to help. However, in this case custom
indexing has no effect, script still fails to extract data.

Is there anything else I could try ?

Dražen