[GRASS-dev] how to increase performance of v.out.ogr to geopackage ?

[Moritz Lennert]

On 20/12/18 20:10, Markus Metz wrote:
> 
> 
> On Thu, Dec 20, 2018 at 10:34 AM Markus Metz 
> <markus.metz.giswork at gmail.com <mailto:markus.metz.giswork at gmail.com>> 
> wrote:
>  >
>  >
>  >
>  > On Thu, Nov 29, 2018 at 3:43 PM Moritz Lennert 
> <mlennert at club.worldonline.be <mailto:mlennert at club.worldonline.be>> wrote:
>  > >
>  > > Hi,
>  > >
>  > > I have large amount of vector maps (~ 400) which each has hundreds of
>  > > thousands of areas and over 150 attribute columns. I'm looking for the
>  > > most efficient way to export these to a format I can share with
>  > > non-GRASS GIS users. My favourite would probably be Geopackage, amongst
>  > > others because the attribute column names are sometimes fairly long,
>  > > thus making export to a DBF based format not easily feasible.
>  > >
>  > > I've seen several sources of information about possible performance
>  > > increases, either in the SQLite world (as a Geopackage is based on
>  > > SQLite) or in the OGR SQLite driver page.
>  > >
>  > > For OGR [1], there is the -gt option to ogr2ogr which allows increasing
>  > > the number of features per transaction (or even running the entire
>  > > conversion in one single transaction).
> [...]
>  > > How about the
>  > > transaction option ?
>  >
>  > I'm afraid the transaction option (equivalent to ogr2ogr -gt) is out 
> of control of v.out.ogr because it is format-specific and v.out.ogr does 
> not use the output format drivers directly, instead it uses the generic 
> OGR interface. However, we could try 
> GDALDatasetStartTransaction()/GDALDatasetCommitTransaction() in v.out.ogr.
> 
> v.out.ogr already uses the OGR interface to start and commit 
> transactions when possible.
> 
> Another reason for slow export can be a buggy NFS implementation if you 
> export to a NFS mount: check if v.out.ogr is running at 100%. If not you 
> could export to a local temp dir and then copy the GeoPackages to the 
> final destination.

Thanks for all the hints. Using

OGR_SQLITE_CACHE=1024 OGR_SQLITE_SYNCHRONOUS=OFF v.out.ogr ...

should already help.

In this particular case export was happening to local disk, so NFS was 
not an issue.

Moritz