[GRASS-dev] how to increase performance of v.out.ogr to geopackage ?

[Markus Metz]

On Thu, Nov 29, 2018 at 3:43 PM Moritz Lennert <mlennert at club.worldonline.be>
wrote:
>
> Hi,
>
> I have large amount of vector maps (~ 400) which each has hundreds of
> thousands of areas and over 150 attribute columns. I'm looking for the
> most efficient way to export these to a format I can share with
> non-GRASS GIS users. My favourite would probably be Geopackage, amongst
> others because the attribute column names are sometimes fairly long,
> thus making export to a DBF based format not easily feasible.
>
> I've seen several sources of information about possible performance
> increases, either in the SQLite world (as a Geopackage is based on
> SQLite) or in the OGR SQLite driver page.
>
> For OGR [1], there is the -gt option to ogr2ogr which allows increasing
> the number of features per transaction (or even running the entire
> conversion in one single transaction). There is also mention of the
> OGR_SQLITE_CACHE and OGR_SQLITE_SYNCHRONOUS configuration options.
> However, I don't seem to understand if, and if yes, how I can set any of
> these when using v.out.ogr. In r73117, MarkusM added config and dataset
> open options to r.in.gdal and v.in.ogr. Would we need something like
> this to be able to set the above config options ? How about the
> transaction option ?

For GDAL/OGR, you can set configuration options in various ways [0], e.g.

OGR_SQLITE_CACHE=1024 OGR_SQLITE_SYNCHRONOUS=OFF v.out.ogr ...

I'm afraid the transaction option (equivalent to ogr2ogr -gt) is out of
control of v.out.ogr because it is format-specific and v.out.ogr does not
use the output format drivers directly, instead it uses the generic OGR
interface. However, we could try
GDALDatasetStartTransaction()/GDALDatasetCommitTransaction() in v.out.ogr.

Markus M

[0] https://trac.osgeo.org/gdal/wiki/ConfigOptions

>
> On the SQLite side, there are some pragma statements that allow
> increasing performance (generally at the cost of less data security, but
> in some cases this might be acceptable), such as journal_mode [2] or
> synchronous [3] (which I suppose is what OGR_SQLITE_SYNCHRONOUS sets as
> well), or possibly locking_mode (I don't know how this is handled
> internally in OGR).
>
> Again, my question would be how these could be set when exporting data
> with v.out.ogr.
>
> Other interesting options (which would also influence performance)
> might be a flag in v.out.ogr allowing exporting without attribute data
> (although a simple v.db.connect -d before export would probably be
> enough) and allowing selecting a subset of columns to export. But that's
> something else.
>
> Moritz
>
>
>
>
> [1] https://gdal.org/drv_sqlite.html
> [2] https://sqlite.org/pragma.html#pragma_journal_mode
> [3] https://sqlite.org/pragma.html#pragma_synchronous
> _______________________________________________
> grass-dev mailing list
> grass-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/grass-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-dev/attachments/20181220/d1037a74/attachment.html>