[gdal-dev] Tips on Speeding up OGR2OGR GeoPackage Creation

[Dennis Burgess]

Ya like 10 seconds, most things are under 2 minutes, so I can live with that..


[LTI-Full_175px]
Dennis Burgess

Author of "Learn RouterOS- Second Edition”
Link Technologies, Inc -- Mikrotik & WISP Support Services
Office: 314-735-0270  Website: http://www.linktechs.net<http://www.linktechs.net/>
Create Wireless Coverage’s with www.towercoverage.com
Need MikroTik Cloud Management: https://cloud.linktechs.net

From: Laurențiu Nicola <lnicola at dend.ro>
Sent: Friday, October 21, 2022 7:21 AM
To: Dennis Burgess <dmburgess at linktechs.net>
Cc: gdal-dev at lists.osgeo.org
Subject: Re: [gdal-dev] Tips on Speeding up OGR2OGR GeoPackage Creation

It is single-threaded (AFAIK), but as you said, loading the data doesn't take too long, so you won't win more than that by pipelining it. And SQLite doesn't support concurrent writes, not even in WAL mode, so there's not much use for threads there.

I would have expected OGR_SQLITE_SYNCHRONOUS=NO to make a slight difference, but probably less than disabling the spatial index.


Laurentiu

On Fri, Oct 21, 2022, at 15:14, Dennis Burgess wrote:

Yep, tried all of those items it does help if I don’t create the index and then index it later, but we are not talking a huge amount of time.  Guess it’s a single threaded app?





[LTI-Full_175px]

Dennis Burgess

Author of "Learn RouterOS- Second Edition”

Link Technologies, Inc -- Mikrotik & WISP Support Services

Office: 314-735-0270  Website: http://www.linktechs.net<http://www.linktechs.net/>

Create Wireless Coverage’s with www.towercoverage.com<http://www.towercoverage.com>

Need MikroTik Cloud Management: https://cloud.linktechs.net


From: gdal-dev <gdal-dev-bounces at lists.osgeo.org<mailto:gdal-dev-bounces at lists.osgeo.org>> On Behalf Of Lauren?iu Nicola via gdal-dev
Sent: Thursday, October 20, 2022 8:09 AM
To: gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org>
Subject: Re: [gdal-dev] Tips on Speeding up OGR2OGR GeoPackage Creation



Hi,



https://gdal.org/drivers/vector/gpkg.html#configuration-options has a couple of options that can help with performance. See also https://gdal.org/drivers/vector/sqlite.html#target-drivers-vector-sqlite-performance-hints.



But I don't recall them helping that much. YMMV.



Regards,

Laurentiu



On Thu, Oct 20, 2022, at 15:06, Dennis Burgess wrote:

We have a large table in MSSQL, this table we extract data from and drop it into a GeoPackage for further processing.  However, the extraction can take some time, its only using 5% of the CPU available, and pulling the data is fairly quick in debug mode, its really its generation and index creation that is slow.  Any tips on speeding this up?  We have SSDs and NVME drives, 10gig networking etc..  So.  It should be as quick as it can be, but the 5% CPU is what I think the killer is.  On this server 24 cores, so 5% would be right at one core..  Is there no multi-threading or other options here?





[LTI-Full_175px]

Dennis Burgess, Mikrotik Certified Trainer
MTCNA, MTCRE, MTCWE, MTCTCE, MTCINE, MTCSE, HE IPv6 Sage, Cambium ePMP Certified

Author of "Learn RouterOS- Second Edition”

Link Technologies, Inc -- Mikrotik & WISP Support Services

Office: 314-735-0270  Website: http://www.linktechs.net<http://www.linktechs.net/>

Need to Automate MikroTik Backups:  https://cloud.linktechs.net

Create Wireless Coverage’s with www.towercoverage.com<http://www.towercoverage.com>



_______________________________________________

gdal-dev mailing list

gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org>

https://lists.osgeo.org/mailman/listinfo/gdal-dev




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20221021/651e1d2c/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 4424 bytes
Desc: image001.png
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20221021/651e1d2c/attachment-0001.png>