[gdal-dev] OSM Driver and World Planet file (pbf format)

[Jukka Rahkonen]

Jukka Rahkonen <jukka.rahkonen <at> mmmtike.fi> writes:


> I borrowed my son's computer and made one more test. Important
 numbers about the
> computer:
> Windows 7, 64 -bit
> Four-core Intel i5 2500k @3,3 GHz
> SSD disk
> 
> Timings for germany.osm.pbf by using  -lco SPATIAL_INDEX=NO. Times are 
total
> times from the beginning of the test
> 
> 70% progress  - 5 minutes (I suppose resolving ways begins about at this phase)
> 100 % progress - 17 minutes
> manual index creation for all the layers ready - 45 minutes
> 
> Results are interesting. Import to Spatialite without indexing was 
six times
> faster for me and I suppose it is mostly because of the SSD drive.
 But creating
> indexes took me 30 minutes while you timing was 22 minutes. 
Perhaps there is
> something sub-optimal in combination of 
Windows/Spatialite/Create spatial index.
> 
> Anyway, the score for germany.osm.pbf is now 45 minutes. Someone
 with Linux and
> SSD drive is perhaps the one to beat the record.
> 
> For comparison, converting finland.osm.pbf took 2 min 40 sec.
 Converting Germany
> took 20 times more time and I believe that the relation is OK now.

Another set of tests with a brand new and quite powerful laptop.
 Specs for the
computer:
Intel i7-2760QM @2.4 GHz processor (8 threads)
Hitachi Travelstar Z7K320 7200 rpm SATA disk
8 GB of memory
Windows 7, 64-bit

GDAL-version r24717, Win64 build from gisinternals.com

Timings for germany.osm.pbf (1.3 GB)
====================================

A) Default settings with command
ogr2ogr -f sqlite -dsco spatialite=yes germany.sqlite 
germany.osm.pbf -gt 20000 -progress --config OGR_SQLITE_SYNCHRONOUS OFF

- reading the data               67 minutes
- creating spatial indexes       38 minutes
- total                         105 minutes

B) Using in-memory Spatialite db for the first step by giving
SET OSM_MAX_TMPFILE_SIZE=7000

- reading the data              16 minutes
- creating spatial indexes      38 minutes
- total                         54 minutes

Peak memory usage during this conversion was 4.4 GB.

Conclusions
===========
* The initial reading of data is heavily i/o bound. This phase 
is really fast if there is enough memory for keeping the OSM 
tempfile in memory but SSD disk seems to offer equally good 
performance.
* Creating spatial indexes for the Spatialite tables is also 
i/o bound. The hardware sets the speed limit and there are 
no other tricks for improving the performance. Multi-core 
CPU is quite idle during this phase with 10-15% load.
* If user does not plan to do spatial queries then then it 
may be handy to save some time and create the Spatialite db 
without spatial indexes by using -lco SPATIAL_INDEX=NO option.
* Windows disk i/o may be a limiting factor.

I consider that for small OSM datasets the speed starts to be 
good enough. For me it is about the same if converting the 
Finnish OSM data (137 MB in .pbf format) takes 160 or 140 
seconds when using the default settings or in-memory temporary 
database, respectively.  

-Jukka Rahkonen-