<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Aptos;
panose-1:2 11 0 4 2 2 2 2 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Aptos",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:70.85pt 85.05pt 70.85pt 85.05pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Aptos",sans-serif">+1. We use a variety of hand-rolled VirtualAlloc based (for basic tasks, a simple pointer bump, and for more elaborate needs, a ‘buddy’) allocators, some of which try to be
smart about memory usage via de-committing regions. In our work, we tend to disable the GDAL cache entirely and rely on the file system’s file cache instead, which is a simplification we can make but is surely untenable in general here.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Aptos",sans-serif"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span lang="CA" style="font-family:"Calibri",sans-serif;color:black">From:
</span></b><span lang="CA" style="font-family:"Calibri",sans-serif;color:black">gdal-dev <gdal-dev-bounces@lists.osgeo.org> on behalf of Abel Pau via gdal-dev <gdal-dev@lists.osgeo.org><br>
<b>Reply-To: </b>Abel Pau <a.pau@creaf.uab.cat><br>
<b>Date: </b>Thursday, March 21, 2024 at 4:51 AM<br>
<b>To: </b>"gdal-dev@lists.osgeo.org" <gdal-dev@lists.osgeo.org><br>
<b>Subject: </b>[EXTERNAL] [BULK] Re: [gdal-dev] Experience with slowness of free() on Windows with lots of allocations?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="CA" style="font-family:"Aptos",sans-serif"><o:p> </o:p></span></p>
</div>
<table class="MsoNormalTable" border="1" cellspacing="0" cellpadding="0" align="left" style="border:solid black 1.5pt">
<tbody>
<tr>
<td width="100%" style="width:100.0%;border:none;background:#FFEB9C;padding:3.75pt 3.75pt 3.75pt 3.75pt">
<p class="MsoNormal" style="mso-element:frame;mso-element-frame-hspace:2.25pt;mso-element-wrap:around;mso-element-anchor-vertical:paragraph;mso-element-anchor-horizontal:column;mso-height-rule:exactly">
<b><span style="font-size:10.0pt;font-family:"Aptos",sans-serif;color:black">CAUTION:</span></b><span style="font-family:"Aptos",sans-serif;color:black">
</span><span style="font-size:10.0pt;font-family:"Aptos",sans-serif;color:black">This email originated from outside of NASA. Please take care when clicking links or opening attachments. Use the "Report Message" button to report suspicious messages to the
NASA SOC.</span><span style="font-family:"Aptos",sans-serif;color:black"> </span>
<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="CA" style="font-family:"Aptos",sans-serif"><br>
<br>
<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span lang="ES" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hi Even,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="ES" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="ES" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">you’re right. We also know that.
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">When programming the driver I took it in consideration. Our solution is not rely on windows to make a good job with memory and we try to reuse as memory as possible instead
of use calloc/free freely.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">For instance, in the driver, for each feature I have to get or write the coordinates. I could do it every time I have to, so lots of times: create memory for
reading, and then put them on the feature, and then free... so many times. What I do? When opening the layer I create some memory blocs of 250 Mb (due to the format itself) and I use that created memory to manage whatever I need. And when closing, I free it.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">While doing that I observed that sometimes I have to use GDAL code that doesn’t take it in consideration (</span><span lang="CA" style="font-size:9.5pt;font-family:Consolas;color:#6F008A">CPLRecode()</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">
for instance). Perhaps it could be improves as well.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Thanks for noticing that.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span lang="ES" style="font-size:11.0pt;font-family:"Calibri",sans-serif">De:</span></b><span lang="ES" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> gdal-dev <gdal-dev-bounces@lists.osgeo.org>
<b>En nombre de </b>Javier Jimenez Shaw via gdal-dev<br>
<b>Enviado el:</b> dijous, 21 de març de 2024 8:27<br>
<b>Para:</b> Even Rouault <even.rouault@spatialys.com><br>
<b>CC:</b> gdal dev <gdal-dev@lists.osgeo.org><br>
<b>Asunto:</b> Re: [gdal-dev] Experience with slowness of free() on Windows with lots of allocations?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="CA"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span lang="CA">In my company we confirmed that "Windows heap allocation mechanism sucks."<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span lang="CA">Closing the application after using gtiff driver can take many seconds due to memory deallocations.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="CA"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="CA">One workaround was to use tcmalloc. I will ask my colleagues more details next week.<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="CA"><o:p> </o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="CA">On Thu, 21 Mar 2024, 01:55 Even Rouault via gdal-dev, <<a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a>> wrote:<o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<p class="MsoNormal"><span lang="CA">Hi,<br>
<br>
while investigating <br>
<a href="https://github.com/OSGeo/gdal/issues/9510#issuecomment-2010950408" target="_blank">https://github.com/OSGeo/gdal/issues/9510#issuecomment-2010950408</a>, I've
<br>
come to the conclusion that the Windows heap allocation mechanism sucks. <br>
Basically if you allocate a lot of heap regions of modest size with <br>
malloc()/new[], the time spent when freeing them all with corresponding <br>
free()/delete[] is excruciatingly slow (like ~ 10 seconds for ~ 80,000 <br>
allocations). The slowness is clearly quadratic with the number of <br>
allocations. You only start noticing it with ~ 30,000 allocations. And <br>
interestingly, another condition for that slowness is that each <br>
individual allocation much be strictly greater than 4096 * 4 bytes. At <br>
exactly that value, perf is acceptable, but add one extra byte, and it <br>
suddenly drops. I suspect that there must be a threshold from which <br>
malloc() starts using VirtualAlloc() instead of the heap, which must <br>
involve slow system calls, instead of a user-land allocation mechanism.<br>
<br>
Anyone has already hit that and found solutions? The only potential idea <br>
I found until now would be to use a private heap with HeapCreate() with <br>
a fixed maximum size, which is a bit problematic to adopt by default, <br>
basically that would mean that the size of GDAL_CACHEMAX would be <br>
consumed as soon as one use the block cache.<br>
<br>
Even<br>
<br>
-- <br>
<a href="http://www.spatialys.com/" target="_blank">http://www.spatialys.com</a><br>
My software is free, but my time generally not.<br>
<br>
_______________________________________________<br>
gdal-dev mailing list<br>
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a><br>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a><o:p></o:p></span></p>
</blockquote>
</div>
</div>
</div>
</body>
</html>