[gdal-dev] FW: GDAL MVT Multithreading

[Momtchil Momtchev]

Can you check the CPU usage of your process? Normally, if you don't 
specify /GDAL_NUM_THREADS/, GDAL will use as many threads as there are 
CPU cores. Also try reducing to =1 to see if it slows down.


On 02/03/2022 08:21, Szymon Piskula (C) via gdal-dev wrote:
>
> Hello,
>
> I am trying to speed up my MVT generation with GDAL. The GDAL 
> documentation says that some of the work can be multi-threaded by use 
> of the GDAL_NUM_THREADS parameter. From my understanding by default 
> there should be as many threads in the pool as many cores there are on 
> the machine. Since I run on Azure VM, I tried scaling up and down 
> between 4 and 16 core but my MVT test data always processes in 
> 10minutes, regardless of the VM spec. I tried running ogr2ogr both 
> with setting --config GDAL_NUM_THREADS 2 (to try force slow down) or 
> help speed up with --config GDAL_NUM_THREADS 12 by hinting to use more 
> threads. Still my processing time is nearly fixed and does not change. 
> I also tried setting the environmental variable GDAL_NUM_THREADS 12 
> but no change with performance. Does anyone know please how well the 
> GDAL MVT actually multi-threads? Can this help speed things up in any 
> significant way? Or is there a different way to control the 
> multi-threading ? The full command I run (on Windows) is :
>
> ogr2ogr --debug ON --config GDAL_NUM_THREADS 12 -f MVT F:\Task_3 -dsco 
> MINZOOM=16 -dsco MAXZOOM=17 -dsco MAX_SIZE=1000000 -dsco EXTENT=8192 
> -dsco COMPRESS=NO PG:<constring> tablea, tableb
>
> My GDAL Version is:
>
> GDAL 3.4.1, released 2021/12/27
>
> I can see that the tiles are written to the .temp.db SQLite, then are 
> written to the local disk folder, so I wonder if it’s the SQLite bit 
> that is the bottleneck ?
>
> Thank you
>
>
>
> This email and any attachments are intended only for the intended 
> recipient and may contain sensitive information. If you are not the 
> intended recipient, please immediately delete this email and inform 
> the sender.
>
> OS email communications may be monitored to ensure the secure and 
> effective operation of our systems and for other lawful purposes. 
> Subject to contract: No rights are to be derived from any proposal 
> contained in this email until a written agreement containing all 
> necessary terms is executed between the relevant parties.
>
> Thank you for your cooperation.
>
> Ordnance Survey Limited (Company Registration number 09121572)
> Registered Office: Explorer House
> Adanac Drive
> Southampton SO16 0AS
> Tel: 03456 050505
> http://www.os.uk
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev

-- 
Momtchil Momtchev<momtchil at momtchev.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20220302/607073b6/attachment.html>