[gdal-dev] Strange reset OGR_INTERLEAVED_READING when using Python bindings to read OSM PBF file

[Dražen Odobašić]

Hi all,

I've been successfully reading OSM PBF files using GDAL Python bindings, even
files larger than 5Gb. I did have to use OGR_INTERLEAVED_READING=YES and
OSM_USE_CUSTOM_INDEXING=NO, but eventually OSM PBF is read and I can extract data.

If I use osmconvert [0] and osmfilter [1] to preprocess data, Python script
complains that OSM_USE_CUSTOM_INDEXING is not used. In the debug output, it will
report that OSM_USE_CUSTOM_INDEXING is used...

"""
Tue Apr  1 19:04:56 2014: Shape: DBF Codepage = UTF-8 for /tmp/out/world1.shp
Tue Apr  1 19:04:56 2014: Shape: Treating as encoding 'UTF-8'.
Tue Apr  1 19:04:57 2014: OSM: Using SQLite indexing for points
Tue Apr  1 19:04:57 2014: OGR: OGROpen(/go_video/admin-levels.pbf/0x1d4a570)
succeeded as OSM.
Tue Apr  1 19:04:57 2014: CPLError: 'admin_level' not recognised as an available
field.
ERROR 1: 'admin_level' not recognised as an available field.
Tue Apr  1 19:04:57 2014: CPLError: 'admin_level' not recognised as an available
field.
ERROR 1: 'admin_level' not recognised as an available field.
Tue Apr  1 19:04:57 2014: CPLError: 'admin_level' not recognised as an available
field.
ERROR 1: 'admin_level' not recognised as an available field.
Tue Apr  1 19:04:57 2014: OSM: OGR_INTERLEAVED_READING = 1
Tue Apr  1 19:04:57 2014: CPLError: 'admin_level' not recognised as an available
field.
ERROR 1: 'admin_level' not recognised as an available field.
Tue Apr  1 19:04:57 2014: OSM: Got bounds : minx=-180.000000, miny=-90.000000,
maxx=180.000000, maxy=90.000000
Tue Apr  1 19:04:57 2014: OSM: For layer points, new max size is 128
Tue Apr  1 19:04:57 2014: OSM: For layer points, new max size is 320
Tue Apr  1 19:04:57 2014: OSM: For layer points, new max size is 608
Tue Apr  1 19:04:57 2014: OSM: For layer points, new max size is 1040
Tue Apr  1 19:04:58 2014: OSM: For layer points, new max size is 1688
Tue Apr  1 19:04:58 2014: OSM: For layer points, new max size is 2660
Tue Apr  1 19:04:58 2014: OSM: For layer points, new max size is 4118
Tue Apr  1 19:04:58 2014: OSM: For layer points, new max size is 6305
Tue Apr  1 19:04:58 2014: OSM: For layer points, new max size is 9585
Tue Apr  1 19:04:58 2014: OSM: For layer points, new max size is 14505
Tue Apr  1 19:04:58 2014: OSM: For layer points, new max size is 21885
Tue Apr  1 19:04:58 2014: OSM: For layer points, new max size is 32955
Tue Apr  1 19:05:01 2014: OSM: all_tags/other_tags field truncated for feature
249399339
Tue Apr  1 19:05:01 2014: OSM: For layer points, new max size is 49560
Tue Apr  1 19:05:03 2014: OSM: For layer points, new max size is 74468
Tue Apr  1 19:05:06 2014: OSM: /vsimem/osm_importer/osm_temp_0x1d4a570.sqlite
too big for RAM. Transfering it onto disk in /tmp/osm_tmp1067730688_0
Tue Apr  1 19:05:20 2014: OSM: For layer points, new max size is 111830
Tue Apr  1 19:05:26 2014: CPLError: Too many features have accumulated in points
layer. Use OGR_INTERLEAVED_READING=YES mode
ERROR 1: Too many features have accumulated in points layer. Use
OGR_INTERLEAVED_READING=YES mode
Tue Apr  1 19:05:26 2014: OGR:
ReleaseDataSource(/go_video/admin-levels.pbf/0x1d4a570) on unshared datasource!
Deleting directly.
Tue Apr  1 19:05:26 2014: OSM: Number of bytes read in file : 143840051
Tue Apr  1 19:05:26 2014: OGR: ReleaseDataSource(/tmp/out/world1.shp/0x19a93d0)
on unshared datasource!
Deleting directly.
Tue Apr  1 19:05:26 2014: GDAL: In GDALDestroy - unloading GDAL shared library.
"""

However, using ogr2ogr utility on the command line works like a charm. If I
convert PBF->OSM, script will work, but exported data is missing some of the
features. I can provide test data and the script if required.

Thank you,

Dražen


[0] https://wiki.openstreetmap.org/wiki/Osmconvert
[1] https://wiki.openstreetmap.org/wiki/Osmfilter