[gdal-dev] Slow convertion from OSM to PG with -skipfailures

[Rahkonen Jukka]

Even Rouault wrote:

> Selon Jukka Rahkonen <jukka.rahkonen at mmmtike.fi>:

>> Hi,
>>
>> I have measured a huge speed difference by running the same conversion from
>> OSM data file into PostGIS either with or without using the -skipfailures
>> parameter.
>>
>> Without -skipfailures conversion takes about two minutes but if I add the
>> parameter it takes at least two hours. The command I used is this
>>
>> ogr2ogr -f PostgreSQL PG:"dbname='gis' host='server' port='4326' user='user'
>> password='passwd'" finland.osm.pbf -gt 20000 -progress
>> --config OSM_COMPRESS_NODES YES  -lco DIM=2 -lco geometry_name=geoloc
>> -lco fid=fid -skipfailures  --config PG_USE_COPY YES
>>
>> I wonder if it really needs to be so slow. My guess is that -skipfailures
>> somehow invalidates my -gt 20000 parameter.

> Yes, it does. In -skipfailures mode, the transaction must be per feature, so
> that a failure on a feature doesn't impact other features.

Don't you feel that the price is rather high? In this example like 120 minutes vs. 2 minutes. Could you imagine a Speedy Sanitizer (TM) option which would use a small super fast in-memory container for collecting something like one thousand valid features before flushing them into the database? 

-Jukka-