[gdal-dev] Strange reset OGR_INTERLEAVED_READING when using Python bindings to read OSM PBF file

[Dražen Odobašić]

On 05.04.2014 11:23, Even Rouault wrote:
> Le mercredi 02 avril 2014 21:45:23, Dražen Odobašić a écrit :
>> On 02.04.2014 21:38, Even Rouault wrote:
>>>> I had no issues when reading large PBF files using this method. For the
>>>> large files, without OSM_USE_CUSTOM_INDEXING=NO I got lots of "Cannot
>>>> read node ..." error messages, and I'm not sure why. Basically I
>>>> stumbled onto OSM_USE_CUSTOM_INDEXING=NO which seems to help. However,
>>>> in this case custom indexing has no effect, script still fails to
>>>> extract data.
>>>
>>> Hum, there might be a bug in custom indexing that you've hit. What is the
>>> PBF file you are converting ?
>>>
>>> And to be sure to understand: when setting OSM_USE_CUSTOM_INDEXING=NO,
>>> the conversion works ?
>>
>> Yes, converting http://download.geofabrik.de/north-america-latest.osm.pbf
>> (5.9G), works only with manually overridden OSM_USE_CUSTOM_INDEXING=NO
> 
> Dražen,
> 
> I've successfully converted the north-america-latest.osm.pbf with GDAL 
> 1.11beta1 (Linux 64bit system) without having to override 
> OSM_USE_CUSTOM_INDEXING. But the code of the OSM driver hasn't changed 
> significantly w.r.t. 1.10 so I'm not sure why you get an error...
> 
> Even
> 


Hi Even,

I've tested north-america-latest.osm.pbf (on 1.10 and 1.11), and it works
without having to override OSM_USE_CUSTOM_INDEXING. I have not been really
methodical during testing, so there might have been something else that
triggered that behaviour. At the moment it works as advertised.

Also, the original problem is triggered only with a PBF file that was generated
with 'osmconvert' utility, and using GDAL Python bindings to read all layers.
ogr2ogr CLI utility will correctly finish converting osm->shp. However, if I use
a tool like 'osmosis' to convert .osm -> .pbf in the final data preparation
step, Python script also executes correctly.

It's possible that osmconvert creates a strange PBF file, which is somehow
misinterpreted when using Python bindings. Anyway, the file is available at
http://open.geof.hr/~dodobas/admin_level_BAD.pbf (455Mb) if you want to test
using Python script (http://open.geof.hr/~dodobas/osm_extract.zip)

Dražen