[gdal-dev] Strange reset OGR_INTERLEAVED_READING when using Python bindings to read OSM PBF file

[Even Rouault]

Le mardi 01 avril 2014 19:26:09, Dražen Odobašić a écrit :
> Hi all,
> 
> I've been successfully reading OSM PBF files using GDAL Python bindings,
> even files larger than 5Gb. I did have to use OGR_INTERLEAVED_READING=YES
> and OSM_USE_CUSTOM_INDEXING=NO, but eventually OSM PBF is read and I can
> extract data.

Generally, you shouldn't have to mess with OSM_USE_CUSTOM_INDEXING.

As far as OGR_INTERLEAVED_READING=YES is concerned, it is not sufficient to 
define the environmenent variable. I invite you to read again "Interleaved 
reading" in http://gdal.org/ogr/drv_osm.html. The sample C++ code (ported to 
Python) must be used *exactly*.  I really mean it. That's what ogr2ogr does 
with OSM data.

> 
> If I use osmconvert [0] and osmfilter [1] to preprocess data, Python script
> complains that OSM_USE_CUSTOM_INDEXING is not used. In the debug output, it
> will report that OSM_USE_CUSTOM_INDEXING is used...
> 
> """
> Tue Apr  1 19:04:56 2014: Shape: DBF Codepage = UTF-8 for
> /tmp/out/world1.shp Tue Apr  1 19:04:56 2014: Shape: Treating as encoding
> 'UTF-8'.
> Tue Apr  1 19:04:57 2014: OSM: Using SQLite indexing for points
> Tue Apr  1 19:04:57 2014: OGR:
> OGROpen(/go_video/admin-levels.pbf/0x1d4a570) succeeded as OSM.
> Tue Apr  1 19:04:57 2014: CPLError: 'admin_level' not recognised as an
> available field.
> ERROR 1: 'admin_level' not recognised as an available field.
> Tue Apr  1 19:04:57 2014: CPLError: 'admin_level' not recognised as an
> available field.
> ERROR 1: 'admin_level' not recognised as an available field.
> Tue Apr  1 19:04:57 2014: CPLError: 'admin_level' not recognised as an
> available field.
> ERROR 1: 'admin_level' not recognised as an available field.
> Tue Apr  1 19:04:57 2014: OSM: OGR_INTERLEAVED_READING = 1
> Tue Apr  1 19:04:57 2014: CPLError: 'admin_level' not recognised as an
> available field.
> ERROR 1: 'admin_level' not recognised as an available field.
> Tue Apr  1 19:04:57 2014: OSM: Got bounds : minx=-180.000000,
> miny=-90.000000, maxx=180.000000, maxy=90.000000
> Tue Apr  1 19:04:57 2014: OSM: For layer points, new max size is 128
> Tue Apr  1 19:04:57 2014: OSM: For layer points, new max size is 320
> Tue Apr  1 19:04:57 2014: OSM: For layer points, new max size is 608
> Tue Apr  1 19:04:57 2014: OSM: For layer points, new max size is 1040
> Tue Apr  1 19:04:58 2014: OSM: For layer points, new max size is 1688
> Tue Apr  1 19:04:58 2014: OSM: For layer points, new max size is 2660
> Tue Apr  1 19:04:58 2014: OSM: For layer points, new max size is 4118
> Tue Apr  1 19:04:58 2014: OSM: For layer points, new max size is 6305
> Tue Apr  1 19:04:58 2014: OSM: For layer points, new max size is 9585
> Tue Apr  1 19:04:58 2014: OSM: For layer points, new max size is 14505
> Tue Apr  1 19:04:58 2014: OSM: For layer points, new max size is 21885
> Tue Apr  1 19:04:58 2014: OSM: For layer points, new max size is 32955
> Tue Apr  1 19:05:01 2014: OSM: all_tags/other_tags field truncated for
> feature 249399339
> Tue Apr  1 19:05:01 2014: OSM: For layer points, new max size is 49560
> Tue Apr  1 19:05:03 2014: OSM: For layer points, new max size is 74468
> Tue Apr  1 19:05:06 2014: OSM:
> /vsimem/osm_importer/osm_temp_0x1d4a570.sqlite too big for RAM.
> Transfering it onto disk in /tmp/osm_tmp1067730688_0 Tue Apr  1 19:05:20
> 2014: OSM: For layer points, new max size is 111830 Tue Apr  1 19:05:26
> 2014: CPLError: Too many features have accumulated in points layer. Use
> OGR_INTERLEAVED_READING=YES mode
> ERROR 1: Too many features have accumulated in points layer. Use
> OGR_INTERLEAVED_READING=YES mode
> Tue Apr  1 19:05:26 2014: OGR:
> ReleaseDataSource(/go_video/admin-levels.pbf/0x1d4a570) on unshared
> datasource! Deleting directly.
> Tue Apr  1 19:05:26 2014: OSM: Number of bytes read in file : 143840051
> Tue Apr  1 19:05:26 2014: OGR:
> ReleaseDataSource(/tmp/out/world1.shp/0x19a93d0) on unshared datasource!
> Deleting directly.
> Tue Apr  1 19:05:26 2014: GDAL: In GDALDestroy - unloading GDAL shared
> library. """
> 
> However, using ogr2ogr utility on the command line works like a charm. If I
> convert PBF->OSM, script will work, but exported data is missing some of
> the features. I can provide test data and the script if required.
> 
> Thank you,
> 
> Dražen
> 
> 
> [0] https://wiki.openstreetmap.org/wiki/Osmconvert
> [1] https://wiki.openstreetmap.org/wiki/Osmfilter
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev

-- 
Geospatial professional services
http://even.rouault.free.fr/services.html