<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Aptos;
panose-1:2 11 0 4 2 2 2 2 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:12.0pt;
font-family:"Calibri",sans-serif;
mso-ligatures:standardcontextual;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Consolas",serif;
mso-ligatures:standardcontextual;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style></head><body lang=EN-US link=blue vlink=purple style='word-wrap:break-word'><div class=WordSection1><p class=MsoNormal>It was had to really test the ADBC connection as it would crash my cloud instance due to memory / cpu consumption. Doing two map queries at the same time would kill it.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><div><div><p class=MsoNormal><span style='font-family:"Aptos",sans-serif;mso-ligatures:none'>-- <o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Aptos",sans-serif;mso-ligatures:none'>Michael Smith<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Aptos",sans-serif;mso-ligatures:none'>RSGIS Center – ERDC CRREL NH<o:p></o:p></span></p></div></div><p class=MsoNormal><span style='font-family:"Aptos",sans-serif;mso-ligatures:none'>US Army Corps</span><o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='color:black'>From: </span></b><span style='color:black'>Even Rouault <even.rouault@spatialys.com><br><b>Date: </b>Monday, July 21, 2025 at 10:13 AM<br><b>To: </b>Michael Smith <michael.smith.erdc@gmail.com>, <gdal-dev@lists.osgeo.org><br><b>Subject: </b>Re: [gdal-dev] Understanding parquet vs adbc performance and SORT_BY_BBOX=YES</span><span style='color:black;mso-ligatures:none'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Aptos",sans-serif'><o:p> </o:p></span></p></div><div><p class=MsoNormal>Le 20/07/2025 à 13:27, Michael Smith via gdal-dev a écrit :<o:p></o:p></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal>Using GDAL 3.11.3:<o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal>I have a dataset Geometry: Point Feature Count: 15546949 in parquet format (written using gdal from oracle source). When doing a spatial query using the geoparquet driver, I see it accessing almost all the row groups of the dataset (PARQUET: 155/156 row groups selected) with a spatial filter fetching 12000 of the 15M points and it takes 0m18.794s. When accessing via ADBC and libduckdb, it takes 0m7.102s (but it also uses 7x CPU and about 10x memory (from looking at top). <o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal>I then rewrote the dataset using -lco SORT_BY_BBOX=YES. Then parquet driver accesses PARQUET: 9/238 row groups selected, and the time drops to 0m1.412s. Using ADBC and libduckdb, the performance doesn’t change. <o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal>For proper performance with gdal, is SORT_BY_BBOX=YES always needed?<o:p></o:p></p></blockquote><p class=MsoNormal><span style='font-family:"Aptos",sans-serif;mso-ligatures:none'>yes, unless your features are already spatially sorted. It is a bit strange that you don't see improvements with the ADBC driver as it does push the spatial filter bbox in the SQL request, so that's perhaps a limitation on how duckdb itself deals with such filters<br><br><o:p></o:p></span></p><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><div><div><p class=MsoNormal><span style='font-family:"Aptos",sans-serif;mso-ligatures:none'>-- </span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-family:"Aptos",sans-serif;mso-ligatures:none'>Michael Smith</span><o:p></o:p></p><p class=MsoNormal><span style='font-family:"Aptos",sans-serif;mso-ligatures:none'>RSGIS Center – ERDC CRREL NH</span><o:p></o:p></p><p class=MsoNormal><span style='font-family:"Aptos",sans-serif;mso-ligatures:none'>US Army Corps</span><o:p></o:p></p></div></div><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal><span style='font-family:"Aptos",sans-serif;mso-ligatures:none'><br><br><o:p></o:p></span></p><pre>_______________________________________________<o:p></o:p></pre><pre>gdal-dev mailing list<o:p></o:p></pre><pre><a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a><o:p></o:p></pre><pre><a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a><o:p></o:p></pre></blockquote><pre>-- <o:p></o:p></pre><pre><a href="http://www.spatialys.com">http://www.spatialys.com</a><o:p></o:p></pre><pre>My software is free, but my time generally not.<o:p></o:p></pre></div></body></html>