[gdal-dev] OSM Driver and World Planet file (pbf format)

[Jukka Rahkonen]

Even Rouault <even.rouault <at> mines-paris.org> writes:


> 
> However, the performance on germany.osm.pbf seemed very slow on your PC, but 
> after testing on mine it takes ~9 hours, which seemed too slow since a 
> conversion for the full planet-latest.osm.pbf (17 GB)  into "null" (this is a 
> debug output driver, not compiled by default, that doesn't write anything) has 
> taken ~ 30h (which, while looking at 
> http://wiki.openstreetmap.org/wiki/Osm2pgsql/benchmarks, isn't particularly 
> bad)
> 
> After investigations, most of the slowdown it is due to the building of the 
> spatial index of the output spatialite DB. When the spatial index is created 
> at DB initialization, and updated at each feature insertion, the performance 
> is clearly affected. For example, when adding  -lco SPATIAL_INDEX=NO to the 
> command line, the conversion of germany only takes 2 hours. Adding manually 
> the spatial index at the end with ogrinfo the.db -sql "SELECT 
> CreateSpatialIndex('points', 'GEOMETRY')" (and the same for lines, polygons, 
> multilines, multipolygons, other_relations) takes ~ 22 minutes, so overall,  
> this is 4 times faster.
> 
> In r24715, I've implemented defered spatial index creation. And indeed the 
> whole process takes now ~ 2h20.

Hi,

I borrowed my son's computer and made one more test. Important numbers about the
computer:
Windows 7, 64 -bit
Four-core Intel i5 2500k @3,3 GHz
SSD disk

Timings for germany.osm.pbf by using  -lco SPATIAL_INDEX=NO. Times are total
times from the beginning of the test

70% progress  - 5 minutes (I suppose resolving ways begins about at this phase)
100 % progress - 17 minutes
manual index creation for all the layers ready - 45 minutes

Results are interesting. Import to Spatialite without indexing was six times
faster for me and I suppose it is mostly because of the SSD drive. But creating
indexes took me 30 minutes while you timing was 22 minutes. Perhaps there is
something sub-optimal in combination of Windows/Spatialite/Create spatial index.

Anyway, the score for germany.osm.pbf is now 45 minutes. Someone with Linux and
SSD drive is perhaps the one to beat the record.

For comparison, converting finland.osm.pbf took 2 min 40 sec. Converting Germany
took 20 times more time and I believe that the relation is OK now.

-Jukka Rahkonen-