[gdal-dev] Ogr2ogr and default value of -gt

[Jukka Rahkonen]

Saulteau Don <sault.don <at> gmail.com> writes:

> 
> On 2/17/2014 11:39 AM, Even Rouault wrote:
> > Le lundi 17 février 2014 16:31:55, Jukka Rahkonen a écrit :
> >> Jukka Rahkonen <jukka.rahkonen <at> mmmtike.fi> writes:
> >>
> >> I took some timings:
> >>
> >> ogr2ogr -f GPKG germany.gpkg germany.osm.pbf -gt 20000 -progress
> >> --config OSM_COMPRESS_NODES YES
> >> Timing: 18 minutes
> >>
> >> ogr2ogr -f GPKG germany_2.gpkg germany.osm.pbf -progress
> >> --config OSM_COMPRESS_NODES YES
> >> Timing: 2 hours and 11 minutes
> >
> > Jukka,
> > If anyone has
> > an objection to raise against 20000 instead of 200, please speak up.
> >
> > Even
> >
> >>
> >> -Jukka-
> >>
> >
> 
> Does -gt 65536, which is a high value mentioned in the docs have a 
> performance penalty for other RDBMS? I use it on Spatialite and 
> PostgreSQL db's without issues.
> 
> Donovan

I do not believe it is so much about performance but perhaps at some time
RDBMS whith is accessed through a slow network would get a timeout and
decides to do a rollback. And RDBMS must also reserve resources for being
able to rollback the whole transaction. The -gt option is still there and
20000 is hundred times more than 200 so it feels like a good new default.

Actually I have been thinking that rollbacks with ogr2ogr are rather
useless. If RDBMS rollbacks just one block user must still try again with 
the whole dataset because there is no way to know which transaction failed
and having 999800 rows out of million written into DB is as good as having
no rows at all. Thus it could be a good strategy to try to do the whole
conversion into database as a one single transaction. Then the rollback
would behave in a logically correct way with ogr2ogr "either all or
nothing". The cost was discussed already: reserved DB resources for being
able to do rollback. But perhaps this is not enough strong argument for
having an ogr2ogr option "-gt auto" for single transactions.

-Jukka Rahkonen-