<div dir="ltr">Hi Even,<div><br></div><div>Sorry for the slow response. I was out yesterday and this morning. Testing your fix now.</div><div><br></div><div>Simon</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Nov 15, 2021 at 4:59 AM Even Rouault <<a href="mailto:even.rouault@spatialys.com" target="_blank">even.rouault@spatialys.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
  
    
  
  <div>
    <p>Simon,</p>
    <p>unfortunately there are a number of places in the degrib library
      which aren't thread-safe, and you just spotted that the
      errSprintf() routine was one of them. I've queued a fix for that
      in <a href="https://github.com/OSGeo/gdal/pull/4830" target="_blank">https://github.com/OSGeo/gdal/pull/4830</a>. Could you try it ?</p>
    <p>Regarding gdal_translate performance, this is related to
      something I mentioned recently (not sure to whom), but you'll get
      much better performance if you add -co INTERLEAVE=BAND, so that
      the output GeoTIFF is written band after band, to match the most
      performant access pattern for reading GRIB files.<br>
    </p>
    <p>Even<br>
    </p>
    <div>Le 15/11/2021 à 01:32, Simon Eves a
      écrit :<br>
    </div>
    <blockquote type="cite">
      
      <div dir="ltr">
        <div class="gmail_quote">
          <div dir="ltr" class="gmail_attr">We have recently implemented
            a geo raster importer, and all seems fine, except that we
            hit an issue with a particular GRIB2 file from the NOAA
            website, where we get an inconsistent crash inside GDAL
            after a few hundred scanlines.<br>
          </div>
          <div dir="ltr">
            <div><br>
            </div>
            <div>We have seen two different crashes inside GDAL, and a
              third in one of our code paths, but given that there is a
              memory corruption, the latter is perhaps unsurprising.</div>
            <div><br>
            </div>
            <div>All crashes report "double free or corruption
              (fasttop)".<br>
            </div>
            <div><br>
            </div>
            <div>We are multi-threading the reading, but using a
              OGRDataSource per thread. The child threads are only
              calling GetRasterBand(), GetRasterDataType() and
              RasterIO() and only one one band at a time.</div>
            <div><br>
            </div>
            <div>The GRIB2 file is 103MB with 73 Float64 bands, but only
              2345x1597 "pixels".</div>
            <div><br>
            </div>
            <div>We tried converting the file to GeoTIFF with
              gdal_translate (no options, just in and out) and it took
              28 minutes (on a ~2017 i7 quad 4.2), which is surprising,
              as we have other GRIB2 files (between 2 and 12MB) which
              convert "instantly". The resulting GeoTIFF is much bigger
              (21x) but seems to import reliably, has basically the same
              schema (as reported by gdalinfo) and results in the same
              data when imported into our system.</div>
            <div><br>
            </div>
            <div>We only get the crash occasionally, and have only been
              able to trap it in the Debugger a couple of times, with
              nothing obviously wrong.</div>
            <div><br>
            </div>
            <div>Here is a link to the GRIB2 file in question:</div>
            <div><br>
            </div>
            <div><a href="https://drive.google.com/file/d/12Fo6jnIhxzCvnSsup9n0kHVKy9lrHD2l/view?usp=sharing" target="_blank">https://drive.google.com/file/d/12Fo6jnIhxzCvnSsup9n0kHVKy9lrHD2l/view?usp=sharing</a><br>
            </div>
            <div><br>
            </div>
            <div>Attached is the most common stack trace.</div>
            <div><br>
            </div>
            <div>With a DEBUG build of GDAL, looks like it's crashing
              trying to do a realloc() on "buffer" which is NULL,
              although that is supposedly a copy of "&errBuffer" at
              the frame above which seems fine.</div>
            <div><br>
            </div>
            <div>Gonna try robustifying that code and see what
              happens...</div>
            <div><br>
            </div>
            <div>This is all with GDAL 3.2.2 on Ubuntu 20.04 LTS with
              GCC 9.</div>
            <div><br>
            </div>
          </div>
        </div>
        -- <br>
        <div dir="ltr">
          <div dir="ltr">
            <div>
              <div dir="ltr">
                <div>
                  <div dir="ltr">
                    <div>
                      <div dir="ltr">
                        <div style="margin:0px;padding:0px 0px 20px;width:2544px;font-family:Roboto,RobotoDraft,Helvetica,Arial,sans-serif;font-size:medium">
                          <div>
                            <div style="font-size:12.8px;margin:8px 0px 0px;padding:0px">
                              <div>
                                <div dir="ltr"><span><font color="#888888">
                                      <div dir="ltr">
                                        <div dir="ltr">
                                          <div dir="ltr">
                                            <div dir="ltr">
                                              <div dir="ltr">
                                                <div dir="ltr">
                                                  <div dir="ltr">
                                                    <div dir="ltr">
                                                      <div dir="ltr">
                                                        <div dir="ltr">
                                                          <table style="font-family:Times;width:2544px" cellspacing="0" cellpadding="0" border="0">
                                                          <tbody>
                                                          <tr>
                                                          <td style="vertical-align:top;font-size:0px" align="left">
                                                          <table cellspacing="0" cellpadding="0" border="0">
                                                          <tbody>
                                                          <tr>
                                                          <td style="padding:0px 15px 0px 0px;vertical-align:middle" align="left"><font size="2" face="arial,
                                                          helvetica,
                                                          sans-serif"><a href="http://www.omnisci.com/" target="_blank"><img src="http://www2.omnisci.com/l/298412/2018-09-18/3sqpg/298412/61753/OmniSci_Email_Header2.png"></a><br>
                                                          </font></td>
                                                          <td style="padding:0px 0px 0px 15px;vertical-align:top" align="left">
                                                          <table style="width:215px" cellspacing="0" cellpadding="0" border="0">
                                                          <tbody>
                                                          <tr>
                                                          <td style="vertical-align:top" align="left"><span style="white-space:nowrap;color:rgb(0,0,1)"><span style="color:rgb(14,76,144);font-weight:700"><font size="2" face="arial,
                                                          helvetica,
                                                          sans-serif">Simon
                                                          Eves</font></span></span></td>
                                                          </tr>
                                                          <tr>
                                                          <td style="vertical-align:top" align="left">
                                                          <table cellspacing="0" cellpadding="0" border="0">
                                                          <tbody>
                                                          <tr>
                                                          <td style="vertical-align:top" align="left"><span style="white-space:nowrap;color:rgb(0,0,1)"><font size="2" face="arial,
                                                          helvetica,
                                                          sans-serif">Senior
                                                          Graphics
                                                          Engineer,
                                                          Rendering
                                                          Group<br>
                                                          100 Montgomery
                                                          St (5th
                                                          Floor), San
                                                          Francisco, CA
                                                          94104, USA<br>
                                                          </font></span></td>
                                                          </tr>
                                                          </tbody>
                                                          </table>
                                                          </td>
                                                          </tr>
                                                          <tr>
                                                          <td style="vertical-align:top" align="left">
                                                          <table cellspacing="0" cellpadding="0" border="0">
                                                          <tbody>
                                                          <tr>
                                                          <td style="padding:0px;vertical-align:top" align="left"><br>
                                                          </td>
                                                          <td style="padding:0px;vertical-align:top" align="left"><br>
                                                          </td>
                                                          </tr>
                                                          <tr>
                                                          <td style="padding:0px;vertical-align:top" align="left"><span style="white-space:nowrap;color:rgb(0,0,1)"><font size="2" face="arial,
                                                          helvetica,
                                                          sans-serif">Email: <a href="mailto:simon.eves@omnisci.com" target="_blank">simon.eves@omnisci.com</a> |
                                                          Cell: </font></span></td>
                                                          <td style="padding:0px;vertical-align:top" align="left"><span style="white-space:nowrap;color:rgb(0,0,1)"><font size="2" face="arial,
                                                          helvetica,
                                                          sans-serif">+1
                                                          (415) 902-1996</font></span></td>
                                                          </tr>
                                                          </tbody>
                                                          </table>
                                                          <br>
                                                          </td>
                                                          </tr>
                                                          </tbody>
                                                          </table>
                                                          </td>
                                                          </tr>
                                                          </tbody>
                                                          </table>
                                                          </td>
                                                          </tr>
                                                          </tbody>
                                                          </table>
                                                          <br>
                                                        </div>
                                                      </div>
                                                    </div>
                                                  </div>
                                                </div>
                                              </div>
                                            </div>
                                          </div>
                                        </div>
                                      </div>
                                    </font></span></div>
                              </div>
                            </div>
                          </div>
                        </div>
                      </div>
                    </div>
                  </div>
                </div>
              </div>
            </div>
          </div>
        </div>
      </div>
      <br>
      <fieldset></fieldset>
      <pre>_______________________________________________
gdal-dev mailing list
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a>
</pre>
    </blockquote>
    <pre cols="72">-- 
<a href="http://www.spatialys.com" target="_blank">http://www.spatialys.com</a>
My software is free, but my time generally not.</pre>
  </div>

</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div style="margin:0px;padding:0px 0px 20px;width:2544px;font-family:Roboto,RobotoDraft,Helvetica,Arial,sans-serif;font-size:medium"><div><div style="font-size:12.8px;margin:8px 0px 0px;padding:0px"><div><div dir="ltr"><span><font color="#888888"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><table cellpadding="0" cellspacing="0" border="0" style="font-family:Times;width:2544px"><tbody><tr><td align="left" style="vertical-align:top;font-size:0px"><table cellpadding="0" cellspacing="0" border="0"><tbody><tr><td align="left" style="padding:0px 15px 0px 0px;vertical-align:middle"><font face="arial, helvetica, sans-serif" size="2"><a href="http://www.omnisci.com/" target="_blank"><img src="http://www2.omnisci.com/l/298412/2018-09-18/3sqpg/298412/61753/OmniSci_Email_Header2.png"></a><br></font></td><td align="left" style="padding:0px 0px 0px 15px;vertical-align:top"><table cellpadding="0" cellspacing="0" border="0" style="width:215px"><tbody><tr><td align="left" style="vertical-align:top"><span style="white-space:nowrap;color:rgb(0,0,1)"><span style="color:rgb(14,76,144);font-weight:700"><font face="arial, helvetica, sans-serif" size="2">Simon Eves</font></span></span></td></tr><tr><td align="left" style="vertical-align:top"><table cellpadding="0" cellspacing="0" border="0"><tbody><tr><td align="left" style="vertical-align:top"><span style="white-space:nowrap;color:rgb(0,0,1)"><font face="arial, helvetica, sans-serif" size="2">Senior Graphics Engineer, Rendering Group<br>100 Montgomery St (5th Floor), San Francisco, CA 94104, USA<br></font></span></td></tr></tbody></table></td></tr><tr><td align="left" style="vertical-align:top"><table cellpadding="0" cellspacing="0" border="0"><tbody><tr><td align="left" style="padding:0px;vertical-align:top"><br></td><td align="left" style="padding:0px;vertical-align:top"><br></td></tr><tr><td align="left" style="padding:0px;vertical-align:top"><span style="white-space:nowrap;color:rgb(0,0,1)"><font face="arial, helvetica, sans-serif" size="2">Email: <a href="mailto:simon.eves@omnisci.com" target="_blank">simon.eves@omnisci.com</a> | Cell: </font></span></td><td align="left" style="padding:0px;vertical-align:top"><span style="white-space:nowrap;color:rgb(0,0,1)"><font face="arial, helvetica, sans-serif" size="2">+1 (415) 902-1996</font></span></td></tr></tbody></table><br></td></tr></tbody></table></td></tr></tbody></table></td></tr></tbody></table><br></div></div></div></div></div></div></div></div></div></div></font></span></div><div></div></div></div><div></div></div></div></div></div></div></div></div></div></div></div>