[gdal-dev] FW: GDAL MVT Multithreading

[Szymon Piskula (C)]

Hello,

I am trying to speed up my MVT generation with GDAL. The GDAL documentation says that some of the work can be multi-threaded by use of the GDAL_NUM_THREADS parameter. From my understanding by default there should be as many threads in the pool as many cores there are on the machine. Since I run on Azure VM, I tried scaling up and down between 4 and 16 core but my MVT test data always processes in 10minutes, regardless of the VM spec. I tried running ogr2ogr both with setting --config GDAL_NUM_THREADS 2 (to try force slow down) or help speed up with --config GDAL_NUM_THREADS 12 by hinting to use more threads. Still my processing time is nearly fixed and does not change. I also tried setting the environmental variable GDAL_NUM_THREADS 12 but no change with performance. Does anyone know please how well the GDAL MVT actually multi-threads? Can this help speed things up in any significant way? Or is there a different way to control the multi-threading ? The full command I run (on Windows) is :

ogr2ogr --debug ON --config GDAL_NUM_THREADS 12 -f MVT F:\Task_3 -dsco MINZOOM=16 -dsco MAXZOOM=17 -dsco MAX_SIZE=1000000 -dsco EXTENT=8192 -dsco COMPRESS=NO PG:<constring> tablea, tableb

My GDAL Version is:
GDAL 3.4.1, released 2021/12/27

I can see that the tiles are written to the .temp.db SQLite, then are written to the local disk folder, so I wonder if it's the SQLite bit that is the bottleneck ?

Thank you


This email and any attachments are intended only for the intended recipient and may contain sensitive information. If you are not the intended recipient, please immediately delete this email and inform the sender.

OS email communications may be monitored to ensure the secure and effective operation of our systems and for other lawful purposes. Subject to contract: No rights are to be derived from any proposal contained in this email until a written agreement containing all necessary terms is executed between the relevant parties.

Thank you for your cooperation.

Ordnance Survey Limited (Company Registration number 09121572)
Registered Office: Explorer House
Adanac Drive
Southampton SO16 0AS
Tel: 03456 050505
http://www.os.uk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20220302/14492d11/attachment-0001.html>