<div dir="ltr">Hi Even,<div><br></div><div>Sorry for the slow response. I was out yesterday and this morning. Testing your fix now.</div><div><br></div><div>Simon</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Nov 15, 2021 at 4:59 AM Even Rouault <<a href="mailto:even.rouault@spatialys.com" target="_blank">even.rouault@spatialys.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Simon,</p>
<p>unfortunately there are a number of places in the degrib library
which aren't thread-safe, and you just spotted that the
errSprintf() routine was one of them. I've queued a fix for that
in <a href="https://github.com/OSGeo/gdal/pull/4830" target="_blank">https://github.com/OSGeo/gdal/pull/4830</a>. Could you try it ?</p>
<p>Regarding gdal_translate performance, this is related to
something I mentioned recently (not sure to whom), but you'll get
much better performance if you add -co INTERLEAVE=BAND, so that
the output GeoTIFF is written band after band, to match the most
performant access pattern for reading GRIB files.<br>
</p>
<p>Even<br>
</p>
<div>Le 15/11/2021 à 01:32, Simon Eves a
écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">We have recently implemented
a geo raster importer, and all seems fine, except that we
hit an issue with a particular GRIB2 file from the NOAA
website, where we get an inconsistent crash inside GDAL
after a few hundred scanlines.<br>
</div>
<div dir="ltr">
<div><br>
</div>
<div>We have seen two different crashes inside GDAL, and a
third in one of our code paths, but given that there is a
memory corruption, the latter is perhaps unsurprising.</div>
<div><br>
</div>
<div>All crashes report "double free or corruption
(fasttop)".<br>
</div>
<div><br>
</div>
<div>We are multi-threading the reading, but using a
OGRDataSource per thread. The child threads are only
calling GetRasterBand(), GetRasterDataType() and
RasterIO() and only one one band at a time.</div>
<div><br>
</div>
<div>The GRIB2 file is 103MB with 73 Float64 bands, but only
2345x1597 "pixels".</div>
<div><br>
</div>
<div>We tried converting the file to GeoTIFF with
gdal_translate (no options, just in and out) and it took
28 minutes (on a ~2017 i7 quad 4.2), which is surprising,
as we have other GRIB2 files (between 2 and 12MB) which
convert "instantly". The resulting GeoTIFF is much bigger
(21x) but seems to import reliably, has basically the same
schema (as reported by gdalinfo) and results in the same
data when imported into our system.</div>
<div><br>
</div>
<div>We only get the crash occasionally, and have only been
able to trap it in the Debugger a couple of times, with
nothing obviously wrong.</div>
<div><br>
</div>
<div>Here is a link to the GRIB2 file in question:</div>
<div><br>
</div>
<div><a href="https://drive.google.com/file/d/12Fo6jnIhxzCvnSsup9n0kHVKy9lrHD2l/view?usp=sharing" target="_blank">https://drive.google.com/file/d/12Fo6jnIhxzCvnSsup9n0kHVKy9lrHD2l/view?usp=sharing</a><br>
</div>
<div><br>
</div>
<div>Attached is the most common stack trace.</div>
<div><br>
</div>
<div>With a DEBUG build of GDAL, looks like it's crashing
trying to do a realloc() on "buffer" which is NULL,
although that is supposedly a copy of "&errBuffer" at
the frame above which seems fine.</div>
<div><br>
</div>
<div>Gonna try robustifying that code and see what
happens...</div>
<div><br>
</div>
<div>This is all with GDAL 3.2.2 on Ubuntu 20.04 LTS with
GCC 9.</div>
<div><br>
</div>
</div>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div style="margin:0px;padding:0px 0px 20px;width:2544px;font-family:Roboto,RobotoDraft,Helvetica,Arial,sans-serif;font-size:medium">
<div>
<div style="font-size:12.8px;margin:8px 0px 0px;padding:0px">
<div>
<div dir="ltr"><span><font color="#888888">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<table style="font-family:Times;width:2544px" cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td style="vertical-align:top;font-size:0px" align="left">
<table cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td style="padding:0px 15px 0px 0px;vertical-align:middle" align="left"><font size="2" face="arial,
helvetica,
sans-serif"><a href="http://www.omnisci.com/" target="_blank"><img src="http://www2.omnisci.com/l/298412/2018-09-18/3sqpg/298412/61753/OmniSci_Email_Header2.png"></a><br>
</font></td>
<td style="padding:0px 0px 0px 15px;vertical-align:top" align="left">
<table style="width:215px" cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td style="vertical-align:top" align="left"><span style="white-space:nowrap;color:rgb(0,0,1)"><span style="color:rgb(14,76,144);font-weight:700"><font size="2" face="arial,
helvetica,
sans-serif">Simon
Eves</font></span></span></td>
</tr>
<tr>
<td style="vertical-align:top" align="left">
<table cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td style="vertical-align:top" align="left"><span style="white-space:nowrap;color:rgb(0,0,1)"><font size="2" face="arial,
helvetica,
sans-serif">Senior
Graphics
Engineer,
Rendering
Group<br>
100 Montgomery
St (5th
Floor), San
Francisco, CA
94104, USA<br>
</font></span></td>
</tr>
</tbody>
</table>
</td>
</tr>
<tr>
<td style="vertical-align:top" align="left">
<table cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td style="padding:0px;vertical-align:top" align="left"><br>
</td>
<td style="padding:0px;vertical-align:top" align="left"><br>
</td>
</tr>
<tr>
<td style="padding:0px;vertical-align:top" align="left"><span style="white-space:nowrap;color:rgb(0,0,1)"><font size="2" face="arial,
helvetica,
sans-serif">Email: <a href="mailto:simon.eves@omnisci.com" target="_blank">simon.eves@omnisci.com</a> |
Cell: </font></span></td>
<td style="padding:0px;vertical-align:top" align="left"><span style="white-space:nowrap;color:rgb(0,0,1)"><font size="2" face="arial,
helvetica,
sans-serif">+1
(415) 902-1996</font></span></td>
</tr>
</tbody>
</table>
<br>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</font></span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
gdal-dev mailing list
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a>
</pre>
</blockquote>
<pre cols="72">--
<a href="http://www.spatialys.com" target="_blank">http://www.spatialys.com</a>
My software is free, but my time generally not.</pre>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div style="margin:0px;padding:0px 0px 20px;width:2544px;font-family:Roboto,RobotoDraft,Helvetica,Arial,sans-serif;font-size:medium"><div><div style="font-size:12.8px;margin:8px 0px 0px;padding:0px"><div><div dir="ltr"><span><font color="#888888"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><table cellpadding="0" cellspacing="0" border="0" style="font-family:Times;width:2544px"><tbody><tr><td align="left" style="vertical-align:top;font-size:0px"><table cellpadding="0" cellspacing="0" border="0"><tbody><tr><td align="left" style="padding:0px 15px 0px 0px;vertical-align:middle"><font face="arial, helvetica, sans-serif" size="2"><a href="http://www.omnisci.com/" target="_blank"><img src="http://www2.omnisci.com/l/298412/2018-09-18/3sqpg/298412/61753/OmniSci_Email_Header2.png"></a><br></font></td><td align="left" style="padding:0px 0px 0px 15px;vertical-align:top"><table cellpadding="0" cellspacing="0" border="0" style="width:215px"><tbody><tr><td align="left" style="vertical-align:top"><span style="white-space:nowrap;color:rgb(0,0,1)"><span style="color:rgb(14,76,144);font-weight:700"><font face="arial, helvetica, sans-serif" size="2">Simon Eves</font></span></span></td></tr><tr><td align="left" style="vertical-align:top"><table cellpadding="0" cellspacing="0" border="0"><tbody><tr><td align="left" style="vertical-align:top"><span style="white-space:nowrap;color:rgb(0,0,1)"><font face="arial, helvetica, sans-serif" size="2">Senior Graphics Engineer, Rendering Group<br>100 Montgomery St (5th Floor), San Francisco, CA 94104, USA<br></font></span></td></tr></tbody></table></td></tr><tr><td align="left" style="vertical-align:top"><table cellpadding="0" cellspacing="0" border="0"><tbody><tr><td align="left" style="padding:0px;vertical-align:top"><br></td><td align="left" style="padding:0px;vertical-align:top"><br></td></tr><tr><td align="left" style="padding:0px;vertical-align:top"><span style="white-space:nowrap;color:rgb(0,0,1)"><font face="arial, helvetica, sans-serif" size="2">Email: <a href="mailto:simon.eves@omnisci.com" target="_blank">simon.eves@omnisci.com</a> | Cell: </font></span></td><td align="left" style="padding:0px;vertical-align:top"><span style="white-space:nowrap;color:rgb(0,0,1)"><font face="arial, helvetica, sans-serif" size="2">+1 (415) 902-1996</font></span></td></tr></tbody></table><br></td></tr></tbody></table></td></tr></tbody></table></td></tr></tbody></table><br></div></div></div></div></div></div></div></div></div></div></font></span></div><div></div></div></div><div></div></div></div></div></div></div></div></div></div></div></div>