[gdal-dev] Strange reset OGR_INTERLEAVED_READING when using Python bindings to read OSM PBF file

[Even Rouault]

Le lundi 07 avril 2014 10:03:02, Dražen Odobašić a écrit :
> On 05.04.2014 11:23, Even Rouault wrote:
> > Le mercredi 02 avril 2014 21:45:23, Dražen Odobašić a écrit :
> >> On 02.04.2014 21:38, Even Rouault wrote:
> >>>> I had no issues when reading large PBF files using this method. For
> >>>> the large files, without OSM_USE_CUSTOM_INDEXING=NO I got lots of
> >>>> "Cannot read node ..." error messages, and I'm not sure why.
> >>>> Basically I stumbled onto OSM_USE_CUSTOM_INDEXING=NO which seems to
> >>>> help. However, in this case custom indexing has no effect, script
> >>>> still fails to extract data.
> >>> 
> >>> Hum, there might be a bug in custom indexing that you've hit. What is
> >>> the PBF file you are converting ?
> >>> 
> >>> And to be sure to understand: when setting OSM_USE_CUSTOM_INDEXING=NO,
> >>> the conversion works ?
> >> 
> >> Yes, converting
> >> http://download.geofabrik.de/north-america-latest.osm.pbf (5.9G), works
> >> only with manually overridden OSM_USE_CUSTOM_INDEXING=NO
> > 
> > Dražen,
> > 
> > I've successfully converted the north-america-latest.osm.pbf with GDAL
> > 1.11beta1 (Linux 64bit system) without having to override
> > OSM_USE_CUSTOM_INDEXING. But the code of the OSM driver hasn't changed
> > significantly w.r.t. 1.10 so I'm not sure why you get an error...
> > 
> > Even
> 
> Hi Even,
> 
> I've tested north-america-latest.osm.pbf (on 1.10 and 1.11), and it works
> without having to override OSM_USE_CUSTOM_INDEXING. I have not been really
> methodical during testing, so there might have been something else that
> triggered that behaviour. At the moment it works as advertised.
> 
> Also, the original problem is triggered only with a PBF file that was
> generated with 'osmconvert' utility, and using GDAL Python bindings to
> read all layers. ogr2ogr CLI utility will correctly finish converting
> osm->shp. However, if I use a tool like 'osmosis' to convert .osm -> .pbf
> in the final data preparation step, Python script also executes correctly.
> 
> It's possible that osmconvert creates a strange PBF file, which is somehow
> misinterpreted when using Python bindings. Anyway, the file is available at
> http://open.geof.hr/~dodobas/admin_level_BAD.pbf (455Mb) if you want to
> test using Python script (http://open.geof.hr/~dodobas/osm_extract.zip)

Hi Dražen,

I've just tested your pbf file and it showed a limitation of the driver caused 
by the huge PBF blocks produced by osmconvert, that I've fixed with 
http://trac.osgeo.org/gdal/changeset/27138 .
But in my testing, the issue appeared also with ogr2ogr when not specifying 
filters. Your script and ogr2ogr should behave similarly, all other things 
(filters,etc.) remaining equal.

About your script, to test if a field exists, you can use 
lyr.GetLayerDefn().GetFieldIndex('admin_level') >= 0
And for efficiency purpose do not put it in inner loop.
In fact, in your case you could just test that the lyr.GetName() == 
"multipolygons"

Even

-- 
Geospatial professional services
http://even.rouault.free.fr/services.html