[gdal-dev] Strange reset OGR_INTERLEAVED_READING when using Python bindings to read OSM PBF file
Even Rouault
even.rouault at mines-paris.org
Wed Apr 2 12:38:28 PDT 2014
Le mercredi 02 avril 2014 21:21:14, Dražen Odobašić a écrit :
> On 02.04.2014 20:29, Even Rouault wrote:
> > Le mardi 01 avril 2014 19:26:09, Dražen Odobašić a écrit :
> >> Hi all,
> >>
> >> I've been successfully reading OSM PBF files using GDAL Python bindings,
> >> even files larger than 5Gb. I did have to use
> >> OGR_INTERLEAVED_READING=YES and OSM_USE_CUSTOM_INDEXING=NO, but
> >> eventually OSM PBF is read and I can extract data.
> >
> > Generally, you shouldn't have to mess with OSM_USE_CUSTOM_INDEXING.
> >
> > As far as OGR_INTERLEAVED_READING=YES is concerned, it is not sufficient
> > to define the environmenent variable. I invite you to read again
> > "Interleaved reading" in http://gdal.org/ogr/drv_osm.html. The sample
> > C++ code (ported to Python) must be used *exactly*. I really mean it.
> > That's what ogr2ogr does with OSM data.
>
> Hi Even,
>
> this is the Python code that implements interleaved reading, which is more
> or less copied from the c++ example in the docs:
>
> """
> # prepare for the layer/feature iteration
> nLayerCount = ds.GetLayerCount()
> thereIsDataInLayer = True
>
> while thereIsDataInLayer:
> thereIsDataInLayer = False
>
> # read data from layers using OGR_INTERLEAVED_READING method
> for iLayer in xrange(nLayerCount):
> lyr = ds.GetLayer(iLayer)
>
> # read next Feature
> feat = lyr.GetNextFeature()
> while(feat is not None):
> # continue reading features from this layer
> thereIsDataInLayer = True
>
> # do something with the feature
> # save_feature(feat)
>
> # force feature removal - required for INTERLEAVED_READING
> feat.Destroy()
> # get the next feature
> feat = lyr.GetNextFeature()
> """
>
That looks fine.
> I had no issues when reading large PBF files using this method. For the
> large files, without OSM_USE_CUSTOM_INDEXING=NO I got lots of "Cannot read
> node ..." error messages, and I'm not sure why. Basically I stumbled onto
> OSM_USE_CUSTOM_INDEXING=NO which seems to help. However, in this case
> custom indexing has no effect, script still fails to extract data.
Hum, there might be a bug in custom indexing that you've hit. What is the PBF
file you are converting ?
And to be sure to understand: when setting OSM_USE_CUSTOM_INDEXING=NO, the
conversion works ?
>
> Is there anything else I could try ?
>
> Dražen
--
Geospatial professional services
http://even.rouault.free.fr/services.html
More information about the gdal-dev
mailing list