<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-ligatures:standardcontextual;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
mso-ligatures:standardcontextual;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle20
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">ogr2ogr -f gpkg geom_multi_38.gpkg "MSSQL:server=xxxxxx;database=xxxxxxxxx;uid=xxxxxxxxx;pwd=xxxxxxxx;MultipleActiveResultSets=true;" -sql "EXEC xxxxxx 42039, 38" -overwrite -gt 1000000 -lco SPATIAL_INDEX=yes OGR_SQLITE_CACHE=10240MB
OGR_SQLITE_SYNCHRONOUS=OFF <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">This one takes around 30 min to export to gpkg file, if I run the same exec command on sql it takes less than 40 seconds to pull down .. these are geometry lines. Around 940 lines.
<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">The second one<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">"c:\program files\qgis 3.26.2\apps\qgis\bin\qgis_process" run native:dissolve --INPUT=geom_multi_55096.gpkg --OUTPUT=geom_multi_diss_38.gpkg
<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">This would e one of them. Last job took 49 minutes, orginal input is around 1.2 gig.
<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Dedicated platform for processing: 72 cores, 32 gig of RAM, running on NVME disk.<o:p></o:p></p>
<p class="MsoNormal">SQL 2019 Enterprise: 80 cores, 512gig ram, NVME disks.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="mso-ligatures:none">From:</span></b><span style="mso-ligatures:none"> Johannes Kröger (WhereGroup) <johannes.kroeger@wheregroup.com>
<br>
<b>Sent:</b> Tuesday, February 6, 2024 2:01 AM<br>
<b>To:</b> Dennis Burgess <dmburgess@linktechs.net>; QGIS-User@lists.osgeo.org<br>
<b>Subject:</b> Re: [Qgis-user] Optimize QGIS Command_Line<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p>Hey Dennis,<o:p></o:p></p>
<p>can you share those command lines (maybe with sensitive information stripped)? Otherwise it is hard to guess what the issue might be.<o:p></o:p></p>
<p>What does your CPU, RAM and I/O utilisation look like during those long running processes? Same for the MSSQL database server host system.<o:p></o:p></p>
<p>Cheers, Hannes<o:p></o:p></p>
<div>
<p class="MsoNormal">On 05.02.24 14:46, Dennis Burgess via QGIS-User wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">I have two command lines, one reads from my MSSQL database and creates a GeoPackage on the local disk. This takes WAY too long, just to create it , it uses 5% or less CPU, and is SUPER slow. Looking for methods to speed this up. The
data is around 1.2 to 1.4 gig of pop, but it should not take 50 min to create.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Also I am doing a dissolve on this data, once its on the disk, this also takes quite a while, even though when I do it from QGIS it takes like 10 min vs an hour or more on the computer with the command line.
<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">I’m sure I am doing something wrong on both of these. Just trying to understand what it would be.
<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Thanks for any assistance !<o:p></o:p></p>
<p class="MsoNormal"><br>
Dennis<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span style="mso-ligatures:none"><br>
<br>
<o:p></o:p></span></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>QGIS-User mailing list<o:p></o:p></pre>
<pre><a href="mailto:QGIS-User@lists.osgeo.org">QGIS-User@lists.osgeo.org</a><o:p></o:p></pre>
<pre>List info: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user">https://lists.osgeo.org/mailman/listinfo/qgis-user</a><o:p></o:p></pre>
<pre>Unsubscribe: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user">https://lists.osgeo.org/mailman/listinfo/qgis-user</a><o:p></o:p></pre>
</blockquote>
<pre>-- <o:p></o:p></pre>
<pre>Johannes Kröger / GIS-Entwickler/-Berater<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>---------------------------------------------<o:p></o:p></pre>
<pre>Aufwind durch Wissen!<o:p></o:p></pre>
<pre>Web-Seminare und Online-Schulungen<o:p></o:p></pre>
<pre>bei der <a href="https://imsva91-ctp.trendmicro.com:443/wis/clicktime/v1/query?url=http%3a%2f%2fwww.foss%2dacademy.com&umid=7036B80A-10B7-5306-BAB5-505120F413E8&auth=079c058f437b7c6303d36c6513e5e8848d0c5ac4-0b980f826f3d937e36ac4d15a89c87a459c73be8">www.foss-academy.com</a><o:p></o:p></pre>
<pre>---------------------------------------------<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>WhereGroup GmbH<o:p></o:p></pre>
<pre>c/o KK03 GmbH<o:p></o:p></pre>
<pre>Lange Reihe 29<o:p></o:p></pre>
<pre>20099 Hamburg<o:p></o:p></pre>
<pre>Germany<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Tel: +49 (0)228 / 90 90 38 - 36<o:p></o:p></pre>
<pre>Fax: +49 (0)228 / 90 90 38 - 11<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre><a href="mailto:johannes.kroeger@wheregroup.com">johannes.kroeger@wheregroup.com</a><o:p></o:p></pre>
<pre><a href="http://www.wheregroup.com">www.wheregroup.com</a><o:p></o:p></pre>
<pre>Geschäftsführer:<o:p></o:p></pre>
<pre>Olaf Knopp, Peter Stamm <o:p></o:p></pre>
<pre>Amtsgericht Bonn, HRB 9885<o:p></o:p></pre>
<pre>-------------------------------<o:p></o:p></pre>
</div>
</body>
</html>