[gdal-dev] Tips on Speeding up OGR2OGR GeoPackage Creation

[Laurențiu Nicola]

It is single-threaded (AFAIK), but as you said, loading the data doesn't take too long, so you won't win more than that by pipelining it. And SQLite doesn't support concurrent writes, not even in WAL mode, so there's not much use for threads there.

I would have expected OGR_SQLITE_SYNCHRONOUS=NO to make a slight difference, but probably less than disabling the spatial index.


Laurentiu

On Fri, Oct 21, 2022, at 15:14, Dennis Burgess wrote:
> Yep, tried all of those items it does help if I don’t create the index and then index it later, but we are not talking a huge amount of time.  Guess it’s a single threaded app?
>  
>  
> *LTI-Full_175px*
> *Dennis Burgess*
> *
*Author of "Learn RouterOS- Second Edition” 
> *Link Technologies, Inc* -- Mikrotik & WISP Support Services 
> *Office*: 314-735-0270  Website: http://www.linktechs.net 
> Create Wireless Coverage’s with www.towercoverage.com
> Need MikroTik Cloud Management: https://cloud.linktechs.net
>  
> *From:* gdal-dev <gdal-dev-bounces at lists.osgeo.org> * On Behalf Of *Lauren?iu Nicola via gdal-dev
> *Sent:* Thursday, October 20, 2022 8:09 AM
> *To:* gdal-dev at lists.osgeo.org
> *Subject:* Re: [gdal-dev] Tips on Speeding up OGR2OGR GeoPackage Creation
> 
>  
> Hi,
>  
> https://gdal.org/drivers/vector/gpkg.html#configuration-options has a couple of options that can help with performance. See also https://gdal.org/drivers/vector/sqlite.html#target-drivers-vector-sqlite-performance-hints.
>  
> But I don't recall them helping that much. YMMV.
>  
> Regards,
> Laurentiu
>  
> On Thu, Oct 20, 2022, at 15:06, Dennis Burgess wrote:
>> We have a large table in MSSQL, this table we extract data from and drop it into a GeoPackage for further processing.  However, the extraction can take some time, its only using 5% of the CPU available, and pulling the data is fairly quick in debug mode, its really its generation and index creation that is slow.  Any tips on speeding this up?  We have SSDs and NVME drives, 10gig networking etc..  So.  It should be as quick as it can be, but the 5% CPU is what I think the killer is.  On this server 24 cores, so 5% would be right at one core..  Is there no multi-threading or other options here? 
>> 
>>  
>> 
>>  
>> 
>> *LTI-Full_175px*
>> 
>> *Dennis Burgess, Mikrotik Certified Trainer**
>> MTCNA, MTCRE, MTCWE, MTCTCE, MTCINE, MTCSE, HE IPv6 Sage, Cambium ePMP Certified*
>> 
>> Author of "Learn RouterOS- Second Edition”
>> 
>> *Link Technologies, Inc* -- Mikrotik & WISP Support Services
>> 
>> *Office*: 314-735-0270  Website: http://www.linktechs.net
>> 
>> Need to Automate MikroTik Backups:  https://cloud.linktechs.net
>> 
>> Create Wireless Coverage’s with www.towercoverage.com
>> 
>>  
>> 
>> _______________________________________________
>> gdal-dev mailing list
>> gdal-dev at lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>  
>  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20221021/286c42e1/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 4424 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20221021/286c42e1/attachment-0001.png>