<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p><br>
</p>
<div class="moz-cite-prefix">Le 17/11/2021 à 00:21, Simon Eves a
écrit :<br>
</div>
<blockquote type="cite"
cite="mid:CAJf0KTRJ9T-V+O9Ro8BCWn+am5AsB+UC+XENLjW0OMyfLV0KJw@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">Hi Even,
<div><br>
</div>
<div>Thank you very much for the quick and plausible fix. It
certainly seems to resolve the issue on my machine. I'm
sending a build to my colleague to test it on his
Threadripper, as he was the first one to hit it.</div>
<div><br>
</div>
<div>Looks like you have merged this into master and into the
3.4 release branch. Can we assume, therefore, that it will be
in 3.4.1, which appears to be scheduled for the end of
December?</div>
</div>
</blockquote>
your inference is correct :-)<br>
<blockquote type="cite"
cite="mid:CAJf0KTRJ9T-V+O9Ro8BCWn+am5AsB+UC+XENLjW0OMyfLV0KJw@mail.gmail.com">
<div dir="ltr">
<div><br>
</div>
<div>Simon</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, Nov 16, 2021 at 12:57
PM Simon Eves <<a href="mailto:simon.eves@omnisci.com"
moz-do-not-send="true">simon.eves@omnisci.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Hi Even,
<div><br>
</div>
<div>Sorry for the slow response. I was out yesterday and
this morning. Testing your fix now.</div>
<div><br>
</div>
<div>Simon</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, Nov 15, 2021 at
4:59 AM Even Rouault <<a
href="mailto:even.rouault@spatialys.com" target="_blank"
moz-do-not-send="true">even.rouault@spatialys.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<p>Simon,</p>
<p>unfortunately there are a number of places in the
degrib library which aren't thread-safe, and you just
spotted that the errSprintf() routine was one of them.
I've queued a fix for that in <a
href="https://github.com/OSGeo/gdal/pull/4830"
target="_blank" moz-do-not-send="true">https://github.com/OSGeo/gdal/pull/4830</a>.
Could you try it ?</p>
<p>Regarding gdal_translate performance, this is related
to something I mentioned recently (not sure to whom),
but you'll get much better performance if you add -co
INTERLEAVE=BAND, so that the output GeoTIFF is written
band after band, to match the most performant access
pattern for reading GRIB files.<br>
</p>
<p>Even<br>
</p>
<div>Le 15/11/2021 à 01:32, Simon Eves a écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">We have recently
implemented a geo raster importer, and all seems
fine, except that we hit an issue with a
particular GRIB2 file from the NOAA website,
where we get an inconsistent crash inside GDAL
after a few hundred scanlines.<br>
</div>
<div dir="ltr">
<div><br>
</div>
<div>We have seen two different crashes inside
GDAL, and a third in one of our code paths,
but given that there is a memory corruption,
the latter is perhaps unsurprising.</div>
<div><br>
</div>
<div>All crashes report "double free or
corruption (fasttop)".<br>
</div>
<div><br>
</div>
<div>We are multi-threading the reading, but
using a OGRDataSource per thread. The child
threads are only calling GetRasterBand(),
GetRasterDataType() and RasterIO() and only
one one band at a time.</div>
<div><br>
</div>
<div>The GRIB2 file is 103MB with 73 Float64
bands, but only 2345x1597 "pixels".</div>
<div><br>
</div>
<div>We tried converting the file to GeoTIFF
with gdal_translate (no options, just in and
out) and it took 28 minutes (on a ~2017 i7
quad 4.2), which is surprising, as we have
other GRIB2 files (between 2 and 12MB) which
convert "instantly". The resulting GeoTIFF is
much bigger (21x) but seems to import
reliably, has basically the same schema (as
reported by gdalinfo) and results in the same
data when imported into our system.</div>
<div><br>
</div>
<div>We only get the crash occasionally, and
have only been able to trap it in the Debugger
a couple of times, with nothing obviously
wrong.</div>
<div><br>
</div>
<div>Here is a link to the GRIB2 file in
question:</div>
<div><br>
</div>
<div><a
href="https://drive.google.com/file/d/12Fo6jnIhxzCvnSsup9n0kHVKy9lrHD2l/view?usp=sharing"
target="_blank" moz-do-not-send="true">https://drive.google.com/file/d/12Fo6jnIhxzCvnSsup9n0kHVKy9lrHD2l/view?usp=sharing</a><br>
</div>
<div><br>
</div>
<div>Attached is the most common stack trace.</div>
<div><br>
</div>
<div>With a DEBUG build of GDAL, looks like it's
crashing trying to do a realloc() on "buffer"
which is NULL, although that is supposedly a
copy of "&errBuffer" at the frame above
which seems fine.</div>
<div><br>
</div>
<div>Gonna try robustifying that code and see
what happens...</div>
<div><br>
</div>
<div>This is all with GDAL 3.2.2 on Ubuntu 20.04
LTS with GCC 9.</div>
<div><br>
</div>
</div>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div style="margin:0px;padding:0px
0px
20px;width:2544px;font-family:Roboto,RobotoDraft,Helvetica,Arial,sans-serif;font-size:medium">
<div>
<div
style="font-size:12.8px;margin:8px
0px 0px;padding:0px">
<div>
<div dir="ltr"><span><font
color="#888888">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<table
style="font-family:Times;width:2544px"
cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td
style="vertical-align:top;font-size:0px"
align="left">
<table
cellspacing="0"
cellpadding="0" border="0">
<tbody>
<tr>
<td
style="padding:0px
15px 0px
0px;vertical-align:middle"
align="left"><font
size="2"
face="arial,
helvetica,
sans-serif"><a
href="http://www.omnisci.com/" target="_blank" moz-do-not-send="true"><img
src="http://www2.omnisci.com/l/298412/2018-09-18/3sqpg/298412/61753/OmniSci_Email_Header2.png"
moz-do-not-send="true"></a><br>
</font></td>
<td
style="padding:0px
0px 0px
15px;vertical-align:top"
align="left">
<table
style="width:215px"
cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td
style="vertical-align:top"
align="left"><span
style="white-space:nowrap;color:rgb(0,0,1)"><span
style="color:rgb(14,76,144);font-weight:700"><font
size="2"
face="arial,
helvetica,
sans-serif">Simon
Eves</font></span></span></td>
</tr>
<tr>
<td
style="vertical-align:top"
align="left">
<table
cellspacing="0"
cellpadding="0" border="0">
<tbody>
<tr>
<td
style="vertical-align:top"
align="left"><span
style="white-space:nowrap;color:rgb(0,0,1)"><font size="2" face="arial,
helvetica,
sans-serif">Senior
Graphics
Engineer,
Rendering
Group<br>
100 Montgomery
St (5th
Floor), San
Francisco, CA
94104, USA<br>
</font></span></td>
</tr>
</tbody>
</table>
</td>
</tr>
<tr>
<td
style="vertical-align:top"
align="left">
<table
cellspacing="0"
cellpadding="0" border="0">
<tbody>
<tr>
<td
style="padding:0px;vertical-align:top"
align="left"><br>
</td>
<td
style="padding:0px;vertical-align:top"
align="left"><br>
</td>
</tr>
<tr>
<td
style="padding:0px;vertical-align:top"
align="left"><span
style="white-space:nowrap;color:rgb(0,0,1)"><font size="2" face="arial,
helvetica,
sans-serif">Email: <a
href="mailto:simon.eves@omnisci.com" target="_blank"
moz-do-not-send="true">simon.eves@omnisci.com</a> |
Cell: </font></span></td>
<td
style="padding:0px;vertical-align:top"
align="left"><span
style="white-space:nowrap;color:rgb(0,0,1)"><font size="2" face="arial,
helvetica,
sans-serif">+1
(415) 902-1996</font></span></td>
</tr>
</tbody>
</table>
<br>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</font></span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
gdal-dev mailing list
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank" moz-do-not-send="true">gdal-dev@lists.osgeo.org</a>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank" moz-do-not-send="true">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a>
</pre>
</blockquote>
<pre cols="72">--
<a href="http://www.spatialys.com" target="_blank" moz-do-not-send="true">http://www.spatialys.com</a>
My software is free, but my time generally not.</pre>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div style="margin:0px;padding:0px 0px
20px;width:2544px;font-family:Roboto,RobotoDraft,Helvetica,Arial,sans-serif;font-size:medium">
<div>
<div style="font-size:12.8px;margin:8px
0px 0px;padding:0px">
<div>
<div dir="ltr"><span><font
color="#888888">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<table
style="font-family:Times;width:2544px"
cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td
style="vertical-align:top;font-size:0px"
align="left">
<table
cellspacing="0"
cellpadding="0" border="0">
<tbody>
<tr>
<td
style="padding:0px
15px 0px
0px;vertical-align:middle"
align="left"><font
size="2"
face="arial,
helvetica,
sans-serif"><a
href="http://www.omnisci.com/" target="_blank" moz-do-not-send="true"><img
src="http://www2.omnisci.com/l/298412/2018-09-18/3sqpg/298412/61753/OmniSci_Email_Header2.png"
moz-do-not-send="true"></a><br>
</font></td>
<td
style="padding:0px
0px 0px
15px;vertical-align:top"
align="left">
<table
style="width:215px"
cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td
style="vertical-align:top"
align="left"><span
style="white-space:nowrap;color:rgb(0,0,1)"><span
style="color:rgb(14,76,144);font-weight:700"><font
size="2"
face="arial,
helvetica,
sans-serif">Simon
Eves</font></span></span></td>
</tr>
<tr>
<td
style="vertical-align:top"
align="left">
<table
cellspacing="0"
cellpadding="0" border="0">
<tbody>
<tr>
<td
style="vertical-align:top"
align="left"><span
style="white-space:nowrap;color:rgb(0,0,1)"><font size="2" face="arial,
helvetica,
sans-serif">Senior
Graphics
Engineer,
Rendering
Group<br>
100 Montgomery
St (5th
Floor), San
Francisco, CA
94104, USA<br>
</font></span></td>
</tr>
</tbody>
</table>
</td>
</tr>
<tr>
<td
style="vertical-align:top"
align="left">
<table
cellspacing="0"
cellpadding="0" border="0">
<tbody>
<tr>
<td
style="padding:0px;vertical-align:top"
align="left"><br>
</td>
<td
style="padding:0px;vertical-align:top"
align="left"><br>
</td>
</tr>
<tr>
<td
style="padding:0px;vertical-align:top"
align="left"><span
style="white-space:nowrap;color:rgb(0,0,1)"><font size="2" face="arial,
helvetica,
sans-serif">Email: <a
href="mailto:simon.eves@omnisci.com" target="_blank"
moz-do-not-send="true">simon.eves@omnisci.com</a> |
Cell: </font></span></td>
<td
style="padding:0px;vertical-align:top"
align="left"><span
style="white-space:nowrap;color:rgb(0,0,1)"><font size="2" face="arial,
helvetica,
sans-serif">+1
(415) 902-1996</font></span></td>
</tr>
</tbody>
</table>
<br>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</font></span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div style="margin:0px;padding:0px 0px
20px;width:2544px;font-family:Roboto,RobotoDraft,Helvetica,Arial,sans-serif;font-size:medium">
<div>
<div style="font-size:12.8px;margin:8px 0px
0px;padding:0px">
<div>
<div dir="ltr"><span><font color="#888888">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<table
style="font-family:Times;width:2544px"
cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td
style="vertical-align:top;font-size:0px"
align="left">
<table
cellspacing="0"
cellpadding="0" border="0">
<tbody>
<tr>
<td
style="padding:0px
15px 0px
0px;vertical-align:middle"
align="left"><font
size="2"
face="arial,
helvetica,
sans-serif"><a
href="http://www.omnisci.com/" target="_blank" moz-do-not-send="true"><img
src="http://www2.omnisci.com/l/298412/2018-09-18/3sqpg/298412/61753/OmniSci_Email_Header2.png"
moz-do-not-send="true"></a><br>
</font></td>
<td
style="padding:0px
0px 0px
15px;vertical-align:top"
align="left">
<table
style="width:215px"
cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td
style="vertical-align:top"
align="left"><span
style="white-space:nowrap;color:rgb(0,0,1)"><span
style="color:rgb(14,76,144);font-weight:700"><font
size="2"
face="arial,
helvetica,
sans-serif">Simon
Eves</font></span></span></td>
</tr>
<tr>
<td
style="vertical-align:top"
align="left">
<table
cellspacing="0"
cellpadding="0" border="0">
<tbody>
<tr>
<td
style="vertical-align:top"
align="left"><span
style="white-space:nowrap;color:rgb(0,0,1)"><font size="2" face="arial,
helvetica,
sans-serif">Senior
Graphics
Engineer,
Rendering
Group<br>
100 Montgomery
St (5th
Floor), San
Francisco, CA
94104, USA<br>
</font></span></td>
</tr>
</tbody>
</table>
</td>
</tr>
<tr>
<td
style="vertical-align:top"
align="left">
<table
cellspacing="0"
cellpadding="0" border="0">
<tbody>
<tr>
<td
style="padding:0px;vertical-align:top"
align="left"><br>
</td>
<td
style="padding:0px;vertical-align:top"
align="left"><br>
</td>
</tr>
<tr>
<td
style="padding:0px;vertical-align:top"
align="left"><span
style="white-space:nowrap;color:rgb(0,0,1)"><font size="2" face="arial,
helvetica,
sans-serif">Email: <a
href="mailto:simon.eves@omnisci.com" target="_blank"
moz-do-not-send="true">simon.eves@omnisci.com</a> |
Cell: </font></span></td>
<td
style="padding:0px;vertical-align:top"
align="left"><span
style="white-space:nowrap;color:rgb(0,0,1)"><font size="2" face="arial,
helvetica,
sans-serif">+1
(415) 902-1996</font></span></td>
</tr>
</tbody>
</table>
<br>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</font></span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
<a class="moz-txt-link-freetext" href="http://www.spatialys.com">http://www.spatialys.com</a>
My software is free, but my time generally not.</pre>
</body>
</html>