[gdal-dev] Slow convertion from OSM to PG with -skipfailures

[Stephen Woodbridge]

Even,

Would it make sense to create and load a stored procedure that you use 
to load the feature with, then you can trap exceptions and ignore them. 
I would think that this would be much faster. Well the stored procedure 
is run in a transaction bout you could do something like:

begin;
select loadfeatureskipfailures(<args>) from features where gid between 
gid_from and gid_to;
commit;

And run this in a loop incrementing gid_from and gid_to over your range 
of features.

I would think that something along these lines would be faster than 
doing every insert in a separate transaction.

-Steve W

On 5/27/2013 11:12 AM, Even Rouault wrote:
> Selon Jukka Rahkonen <jukka.rahkonen at mmmtike.fi>:
>
>> Hi,
>>
>> I have measured a huge speed difference by running the same conversion from
>> OSM data file into PostGIS either with or without using the -skipfailures
>> parameter.
>>
>> Without -skipfailures conversion takes about two minutes but if I add the
>> parameter it takes at least two hours. The command I used is this
>>
>> ogr2ogr -f PostgreSQL PG:"dbname='gis' host='server' port='4326' user='user'
>> password='passwd'" finland.osm.pbf -gt 20000 -progress
>> --config OSM_COMPRESS_NODES YES  -lco DIM=2 -lco geometry_name=geoloc
>> -lco fid=fid -skipfailures  --config PG_USE_COPY YES
>>
>> I wonder if it really needs to be so slow. My guess is that -skipfailures
>> somehow invalidates my -gt 20000 parameter.
>
> Yes, it does. In -skipfailures mode, the transaction must be per feature, so
> that a failure on a feature doesn't impact other features.
>
>>
>> -Jukka Rahkonen-
>>
>> _______________________________________________
>> gdal-dev mailing list
>> gdal-dev at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>>
>
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>