[GRASS-dev] how to increase performance of v.out.ogr to geopackage ?

[Moritz Lennert]

Hi,

I have large amount of vector maps (~ 400) which each has hundreds of 
thousands of areas and over 150 attribute columns. I'm looking for the 
most efficient way to export these to a format I can share with 
non-GRASS GIS users. My favourite would probably be Geopackage, amongst 
others because the attribute column names are sometimes fairly long, 
thus making export to a DBF based format not easily feasible.

I've seen several sources of information about possible performance 
increases, either in the SQLite world (as a Geopackage is based on 
SQLite) or in the OGR SQLite driver page.

For OGR [1], there is the -gt option to ogr2ogr which allows increasing 
the number of features per transaction (or even running the entire 
conversion in one single transaction). There is also mention of the 
OGR_SQLITE_CACHE and OGR_SQLITE_SYNCHRONOUS configuration options. 
However, I don't seem to understand if, and if yes, how I can set any of 
these when using v.out.ogr. In r73117, MarkusM added config and dataset 
open options to r.in.gdal and v.in.ogr. Would we need something like 
this to be able to set the above config options ? How about the 
transaction option ?

On the SQLite side, there are some pragma statements that allow 
increasing performance (generally at the cost of less data security, but 
in some cases this might be acceptable), such as journal_mode [2] or 
synchronous [3] (which I suppose is what OGR_SQLITE_SYNCHRONOUS sets as 
well), or possibly locking_mode (I don't know how this is handled 
internally in OGR).

Again, my question would be how these could be set when exporting data 
with v.out.ogr.

Other interesting options (which would also influence performance) 
might be a flag in v.out.ogr allowing exporting without attribute data 
(although a simple v.db.connect -d before export would probably be 
enough) and allowing selecting a subset of columns to export. But that's 
something else.

Moritz




[1] https://gdal.org/drv_sqlite.html
[2] https://sqlite.org/pragma.html#pragma_journal_mode
[3] https://sqlite.org/pragma.html#pragma_synchronous