<div dir="ltr"><div><br class="gmail-Apple-interchange-newline">Wouldn't the the max size be limited by the number of bytes read? So in this case 4 bytes.</div><div><br></div><div><a href="http://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_sect5.shtml">http://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_sect5.shtml</a><br></div><div><br></div><div>Looking at netcdf's implementation they treat the value as a 32 bit signed int.</div><div><br></div><div><a href="https://github.com/Unidata/thredds/blob/5.0.0/grib/src/main/java/ucar/nc2/grib/grib2/Grib2SectionDataRepresentation.java#L69">https://github.com/Unidata/thredds/blob/5.0.0/grib/src/main/java/ucar/nc2/grib/grib2/Grib2SectionDataRepresentation.java#L69</a><br></div><div><br></div><div>I am dealing with proprietary grib2 files that do break the current limit of <span style="font-size:12.8px">33554432</span>.</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Nov 7, 2017 at 4:03 PM, Even Rouault <span dir="ltr"><<a href="mailto:even.rouault@spatialys.com" target="_blank">even.rouault@spatialys.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><u></u>
<div style="font-family:'Sans Serif';font-size:9pt;font-weight:400;font-style:normal"><span class="">
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">On mardi 7 novembre 2017 13:51:30 CET Kurt Schwehr wrote:</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> It's possible to cause massive allocations with a tiny corrupted grib file</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> causing an out-of-memory. I found that case with the llvm ASAN fuzzer. If</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> you have a specification that gives a more reasoned maximum or a better</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> overall check, I'm listening. I definitely think the sanity checking can</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> be improved. Mostly I just try to survive the g2clib code. It doesn't</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> come with tests and digging through GRIB specs to match up to g2clib source</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> isn't my favorite thing to do.</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> <a href="https://github.com/OSGeo/gdal/commit/ae92f7fb8e32381124a37588d27b9af695afce2" target="_blank">https://github.com/OSGeo/gdal/<wbr>commit/<wbr>ae92f7fb8e32381124a37588d27b9a<wbr>f695afce2</a></p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> 0</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
</span><p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">I guess that if Roarke is asking the question he might have a dataset that breaks this limit ? If so, we might consider reverting that change, or making it more robust (which can be very tricky I know. Perhaps some heuristics with the file size ?), or just using it in fuzzing mode and not in production for now. And a pointer to such a dataset would be much appreciated.</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">(By the way: 2<<24 is IMHO an usual way of writing a limit. I confused it with 2^24 initially. So 1 << 25 would perhaps be better. Or just in decimal form as it is completely arbitary and not related to a binary encoding)</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">Even</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">-- </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">Spatialys - Geospatial professional services</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"><a href="http://www.spatialys.com" target="_blank">http://www.spatialys.com</a></p></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div><span style="font-family:arial,helvetica,sans-serif;font-size:20px"><span style="color:rgb(64,64,65)"><strong><span style="font-size:14px">Roarke </span></strong></span><span style="font-size:14px"><span style="color:rgb(64,64,65)"><strong>Gaskill </strong></span><span style="color:rgb(105,105,105)"><span style="font-size:16px">|</span></span></span></span><span style="font-family:arial,helvetica,sans-serif;font-size:14px"><span style="color:rgb(64,64,65);margin-left:5px">Senior Software Engineer</span></span></div><span style="font-family:arial,helvetica,sans-serif"><div style="color:rgb(64,64,65);font-weight:bold;font-size:14px"><span style="color:rgb(105,105,105);font-weight:normal;font-size:12px"><strong>e:</strong> <a href="mailto:roarke.gaskill@weather.com" target="_blank">roarke.gaskill@weather.com</a></span></div></span><div><div> <a href="http://weather.com/apps" target="_blank"></a><a href="http://weather.com/apps" target="_blank"></a><a href="http://weather.com/apps" target="_blank"></a><a href="http://weather.com/apps" target="_blank"></a><a href="http://weather.com/apps" target="_blank"></a><a href="http://weather.com/apps" target="_blank"></a><a href="http://weather.com/apps" target="_blank"></a><a href="http://weather.com/apps" target="_blank"></a><a href="http://weather.com/apps" target="_blank"></a><a href="http://weather.com/apps" target="_blank"></a><a href="http://weather.com/apps" target="_blank"></a><a href="http://weather.com/apps" target="_blank"></a><a href="http://weather.com/apps" target="_blank"></a><a href="http://weather.com/apps" target="_blank"></a><a href="http://weather.com/apps" target="_blank"></a><a href="http://weather.com/apps" target="_blank"></a><a href="http://weather.com/apps" target="_blank"><img alt="" src="http://www.theweathercompany.com/sites/default/files/twco_emailsig_225x92_buffered_final.png" style="height:92px;width:225px"></a><br>
</div></div></div></div></div></div></div>
</div>