<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EstiloCorreo18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 3.0cm 70.85pt 3.0cm;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="CA" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span lang="ES" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Hi Even,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="ES" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="ES" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">you’re right. We also know that.
</span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">When programming the driver I took it in consideration. Our solution is not rely on windows to make a good job with memory and we try
to reuse as memory as possible instead of use calloc/free freely.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">For instance, in the driver, for each feature I have to get or write the coordinates. I could do it every time I have
to, so lots of times: create memory for reading, and then put them on the feature, and then free... so many times. What I do? When opening the layer I create some memory blocs of 250 Mb (due to the format itself) and I use that created memory to manage whatever
I need. And when closing, I free it.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">While doing that I observed that sometimes I have to use GDAL code that doesn’t take it in consideration (</span><span style="font-size:9.5pt;font-family:Consolas;color:#6F008A;mso-fareast-language:EN-US">CPLRecode()</span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">
for instance). Perhaps it could be improves as well.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Thanks for noticing that.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span lang="ES" style="font-size:11.0pt;font-family:"Calibri",sans-serif">De:</span></b><span lang="ES" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> gdal-dev <gdal-dev-bounces@lists.osgeo.org>
<b>En nombre de </b>Javier Jimenez Shaw via gdal-dev<br>
<b>Enviado el:</b> dijous, 21 de març de 2024 8:27<br>
<b>Para:</b> Even Rouault <even.rouault@spatialys.com><br>
<b>CC:</b> gdal dev <gdal-dev@lists.osgeo.org><br>
<b>Asunto:</b> Re: [gdal-dev] Experience with slowness of free() on Windows with lots of allocations?<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">In my company we confirmed that "Windows heap allocation mechanism sucks."<o:p></o:p></p>
<div>
<p class="MsoNormal">Closing the application after using gtiff driver can take many seconds due to memory deallocations.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">One workaround was to use tcmalloc. I will ask my colleagues more details next week.<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Thu, 21 Mar 2024, 01:55 Even Rouault via gdal-dev, <<a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal">Hi,<br>
<br>
while investigating <br>
<a href="https://github.com/OSGeo/gdal/issues/9510#issuecomment-2010950408" target="_blank">https://github.com/OSGeo/gdal/issues/9510#issuecomment-2010950408</a>, I've
<br>
come to the conclusion that the Windows heap allocation mechanism sucks. <br>
Basically if you allocate a lot of heap regions of modest size with <br>
malloc()/new[], the time spent when freeing them all with corresponding <br>
free()/delete[] is excruciatingly slow (like ~ 10 seconds for ~ 80,000 <br>
allocations). The slowness is clearly quadratic with the number of <br>
allocations. You only start noticing it with ~ 30,000 allocations. And <br>
interestingly, another condition for that slowness is that each <br>
individual allocation much be strictly greater than 4096 * 4 bytes. At <br>
exactly that value, perf is acceptable, but add one extra byte, and it <br>
suddenly drops. I suspect that there must be a threshold from which <br>
malloc() starts using VirtualAlloc() instead of the heap, which must <br>
involve slow system calls, instead of a user-land allocation mechanism.<br>
<br>
Anyone has already hit that and found solutions? The only potential idea <br>
I found until now would be to use a private heap with HeapCreate() with <br>
a fixed maximum size, which is a bit problematic to adopt by default, <br>
basically that would mean that the size of GDAL_CACHEMAX would be <br>
consumed as soon as one use the block cache.<br>
<br>
Even<br>
<br>
-- <br>
<a href="http://www.spatialys.com" target="_blank">http://www.spatialys.com</a><br>
My software is free, but my time generally not.<br>
<br>
_______________________________________________<br>
gdal-dev mailing list<br>
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a><br>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a><o:p></o:p></p>
</blockquote>
</div>
</div>
</body>
</html>