<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Simon,</p>
<p>unfortunately there are a number of places in the degrib library
which aren't thread-safe, and you just spotted that the
errSprintf() routine was one of them. I've queued a fix for that
in <a class="moz-txt-link-freetext" href="https://github.com/OSGeo/gdal/pull/4830">https://github.com/OSGeo/gdal/pull/4830</a>. Could you try it ?</p>
<p>Regarding gdal_translate performance, this is related to
something I mentioned recently (not sure to whom), but you'll get
much better performance if you add -co INTERLEAVE=BAND, so that
the output GeoTIFF is written band after band, to match the most
performant access pattern for reading GRIB files.<br>
</p>
<p>Even<br>
</p>
<div class="moz-cite-prefix">Le 15/11/2021 à 01:32, Simon Eves a
écrit :<br>
</div>
<blockquote type="cite"
cite="mid:CAJf0KTQk2yuqQ_sDvDmuStCWWELf-0Wm-SDSiBVd4iXOJZ68Sw@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">We have recently implemented
a geo raster importer, and all seems fine, except that we
hit an issue with a particular GRIB2 file from the NOAA
website, where we get an inconsistent crash inside GDAL
after a few hundred scanlines.<br>
</div>
<div dir="ltr">
<div><br>
</div>
<div>We have seen two different crashes inside GDAL, and a
third in one of our code paths, but given that there is a
memory corruption, the latter is perhaps unsurprising.</div>
<div><br>
</div>
<div>All crashes report "double free or corruption
(fasttop)".<br>
</div>
<div><br>
</div>
<div>We are multi-threading the reading, but using a
OGRDataSource per thread. The child threads are only
calling GetRasterBand(), GetRasterDataType() and
RasterIO() and only one one band at a time.</div>
<div><br>
</div>
<div>The GRIB2 file is 103MB with 73 Float64 bands, but only
2345x1597 "pixels".</div>
<div><br>
</div>
<div>We tried converting the file to GeoTIFF with
gdal_translate (no options, just in and out) and it took
28 minutes (on a ~2017 i7 quad 4.2), which is surprising,
as we have other GRIB2 files (between 2 and 12MB) which
convert "instantly". The resulting GeoTIFF is much bigger
(21x) but seems to import reliably, has basically the same
schema (as reported by gdalinfo) and results in the same
data when imported into our system.</div>
<div><br>
</div>
<div>We only get the crash occasionally, and have only been
able to trap it in the Debugger a couple of times, with
nothing obviously wrong.</div>
<div><br>
</div>
<div>Here is a link to the GRIB2 file in question:</div>
<div><br>
</div>
<div><a
href="https://drive.google.com/file/d/12Fo6jnIhxzCvnSsup9n0kHVKy9lrHD2l/view?usp=sharing"
target="_blank" moz-do-not-send="true">https://drive.google.com/file/d/12Fo6jnIhxzCvnSsup9n0kHVKy9lrHD2l/view?usp=sharing</a><br>
</div>
<div><br>
</div>
<div>Attached is the most common stack trace.</div>
<div><br>
</div>
<div>With a DEBUG build of GDAL, looks like it's crashing
trying to do a realloc() on "buffer" which is NULL,
although that is supposedly a copy of "&errBuffer" at
the frame above which seems fine.</div>
<div><br>
</div>
<div>Gonna try robustifying that code and see what
happens...</div>
<div><br>
</div>
<div>This is all with GDAL 3.2.2 on Ubuntu 20.04 LTS with
GCC 9.</div>
<div><br>
</div>
</div>
</div>
-- <br>
<div dir="ltr" class="gmail_signature"
data-smartmail="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div style="margin:0px;padding:0px 0px
20px;width:2544px;font-family:Roboto,RobotoDraft,Helvetica,Arial,sans-serif;font-size:medium">
<div>
<div style="font-size:12.8px;margin:8px 0px
0px;padding:0px">
<div>
<div dir="ltr"><span><font
color="#888888">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<table
style="font-family:Times;width:2544px"
cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td
style="vertical-align:top;font-size:0px"
align="left">
<table
cellspacing="0"
cellpadding="0" border="0">
<tbody>
<tr>
<td
style="padding:0px
15px 0px
0px;vertical-align:middle"
align="left"><font
size="2"
face="arial,
helvetica,
sans-serif"><a
href="http://www.omnisci.com/" target="_blank" moz-do-not-send="true"><img
src="http://www2.omnisci.com/l/298412/2018-09-18/3sqpg/298412/61753/OmniSci_Email_Header2.png"
moz-do-not-send="true"></a><br>
</font></td>
<td
style="padding:0px
0px 0px
15px;vertical-align:top"
align="left">
<table
style="width:215px"
cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td
style="vertical-align:top"
align="left"><span
style="white-space:nowrap;color:rgb(0,0,1)"><span
style="color:rgb(14,76,144);font-weight:700"><font
size="2"
face="arial,
helvetica,
sans-serif">Simon
Eves</font></span></span></td>
</tr>
<tr>
<td
style="vertical-align:top"
align="left">
<table
cellspacing="0"
cellpadding="0" border="0">
<tbody>
<tr>
<td
style="vertical-align:top"
align="left"><span
style="white-space:nowrap;color:rgb(0,0,1)"><font size="2" face="arial,
helvetica,
sans-serif">Senior
Graphics
Engineer,
Rendering
Group<br>
100 Montgomery
St (5th
Floor), San
Francisco, CA
94104, USA<br>
</font></span></td>
</tr>
</tbody>
</table>
</td>
</tr>
<tr>
<td
style="vertical-align:top"
align="left">
<table
cellspacing="0"
cellpadding="0" border="0">
<tbody>
<tr>
<td
style="padding:0px;vertical-align:top"
align="left"><br>
</td>
<td
style="padding:0px;vertical-align:top"
align="left"><br>
</td>
</tr>
<tr>
<td
style="padding:0px;vertical-align:top"
align="left"><span
style="white-space:nowrap;color:rgb(0,0,1)"><font size="2" face="arial,
helvetica,
sans-serif">Email: <a
href="mailto:simon.eves@omnisci.com" target="_blank"
moz-do-not-send="true">simon.eves@omnisci.com</a> |
Cell: </font></span></td>
<td
style="padding:0px;vertical-align:top"
align="left"><span
style="white-space:nowrap;color:rgb(0,0,1)"><font size="2" face="arial,
helvetica,
sans-serif">+1
(415) 902-1996</font></span></td>
</tr>
</tbody>
</table>
<br>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</font></span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
gdal-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a>
<a class="moz-txt-link-freetext" href="https://lists.osgeo.org/mailman/listinfo/gdal-dev">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a>
</pre>
</blockquote>
<pre class="moz-signature" cols="72">--
<a class="moz-txt-link-freetext" href="http://www.spatialys.com">http://www.spatialys.com</a>
My software is free, but my time generally not.</pre>
</body>
</html>