<div dir="ltr">Yeah, 1 << ## would have been better. Sigh.<div><br></div><div>But really, someone needs to work through the logic of this stuff and do something that actually works through what is reasonable. Code that is intelligent and explains why it is doing is far preferable. OOM is not okay in the world I work in, so I tried to pick a large number (basically at random) that should let almost all cases through until someone took more time.</div><div><br></div><div>allowing a full range number without checking to see if the size is reasonable is a *major* problem for me. An OOM is a fatal (as in no error message, your process is dead) situation in my world, so allowing larger without checking seems unpleasant.</div><div><br></div><div>Perhaps a config option to allow massive files through in the short term?</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Nov 7, 2017 at 2:15 PM, Roarke Gaskill <span dir="ltr"><<a href="mailto:roarke.gaskill@weather.com" target="_blank">roarke.gaskill@weather.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><br class="m_-1161601849180139309gmail-Apple-interchange-newline">Wouldn't the the max size be limited by the number of bytes read? So in this case 4 bytes.</div><div><br></div><div><a href="http://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_sect5.shtml" target="_blank">http://www.nco.ncep.noaa.gov/<wbr>pmb/docs/grib2/grib2_sect5.<wbr>shtml</a><br></div><div><br></div><div>Looking at netcdf's implementation they treat the value as a 32 bit signed int.</div><div><br></div><div><a href="https://github.com/Unidata/thredds/blob/5.0.0/grib/src/main/java/ucar/nc2/grib/grib2/Grib2SectionDataRepresentation.java#L69" target="_blank">https://github.com/Unidata/<wbr>thredds/blob/5.0.0/grib/src/<wbr>main/java/ucar/nc2/grib/grib2/<wbr>Grib2SectionDataRepresentation<wbr>.java#L69</a><br></div><div><br></div><div>I am dealing with proprietary grib2 files that do break the current limit of <span style="font-size:12.8px">33554432</span>.</div><div><br></div><div><br></div></div><div class="gmail_extra"><div><div class="h5"><br><div class="gmail_quote">On Tue, Nov 7, 2017 at 4:03 PM, Even Rouault <span dir="ltr"><<a href="mailto:even.rouault@spatialys.com" target="_blank">even.rouault@spatialys.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><u></u>
<div style="font-family:'Sans Serif';font-size:9pt;font-weight:400;font-style:normal"><span>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">On mardi 7 novembre 2017 13:51:30 CET Kurt Schwehr wrote:</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> It's possible to cause massive allocations with a tiny corrupted grib file</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> causing an out-of-memory. I found that case with the llvm ASAN fuzzer. If</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> you have a specification that gives a more reasoned maximum or a better</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> overall check, I'm listening. I definitely think the sanity checking can</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> be improved. Mostly I just try to survive the g2clib code. It doesn't</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> come with tests and digging through GRIB specs to match up to g2clib source</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> isn't my favorite thing to do.</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> <a href="https://github.com/OSGeo/gdal/commit/ae92f7fb8e32381124a37588d27b9af695afce2" target="_blank">https://github.com/OSGeo/gdal/<wbr>commit/ae92f7fb8e32381124a3758<wbr>8d27b9af695afce2</a></p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> 0</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
</span><p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">I guess that if Roarke is asking the question he might have a dataset that breaks this limit ? If so, we might consider reverting that change, or making it more robust (which can be very tricky I know. Perhaps some heuristics with the file size ?), or just using it in fuzzing mode and not in production for now. And a pointer to such a dataset would be much appreciated.</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">(By the way: 2<<24 is IMHO an usual way of writing a limit. I confused it with 2^24 initially. So 1 << 25 would perhaps be better. Or just in decimal form as it is completely arbitary and not related to a binary encoding)</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">Even</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">-- </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">Spatialys - Geospatial professional services</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"><a href="http://www.spatialys.com" target="_blank">http://www.spatialys.com</a></p></div></blockquote></div><br><br clear="all"><div><br></div></div></div><span class="HOEnZb"><font color="#888888">-- <br><div class="m_-1161601849180139309gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div><span style="font-family:arial,helvetica,sans-serif;font-size:20px"><span style="color:rgb(64,64,65)"><strong><span style="font-size:14px">Roarke </span></strong></span><span style="font-size:14px"><span style="color:rgb(64,64,65)"><strong>Gaskill </strong></span><span style="color:rgb(105,105,105)"><span style="font-size:16px">|</span></span></span></span><span style="font-family:arial,helvetica,sans-serif;font-size:14px"><span style="color:rgb(64,64,65);margin-left:5px">Senior Software Engineer</span></span></div><span style="font-family:arial,helvetica,sans-serif"><div style="color:rgb(64,64,65);font-weight:bold;font-size:14px"><span style="color:rgb(105,105,105);font-weight:normal;font-size:12px"><strong>e:</strong> <a href="mailto:roarke.gaskill@weather.com" target="_blank">roarke.gaskill@weather.com</a></span></div></span><div><div> <a href="http://weather.com/apps" target="_blank"></a><a href="http://weather.com/apps" target="_blank"></a><a href="http://weather.com/apps" target="_blank"></a><a href="http://weather.com/apps" target="_blank"></a><a href="http://weather.com/apps" target="_blank"></a><a href="http://weather.com/apps" target="_blank"></a><a href="http://weather.com/apps" target="_blank"></a><a href="http://weather.com/apps" target="_blank"></a><a href="http://weather.com/apps" target="_blank"></a><a href="http://weather.com/apps" target="_blank"></a><a href="http://weather.com/apps" target="_blank"></a><a href="http://weather.com/apps" target="_blank"></a><a href="http://weather.com/apps" target="_blank"></a><a href="http://weather.com/apps" target="_blank"></a><a href="http://weather.com/apps" target="_blank"></a><a href="http://weather.com/apps" target="_blank"></a><a href="http://weather.com/apps" target="_blank"><img alt="" src="http://www.theweathercompany.com/sites/default/files/twco_emailsig_225x92_buffered_final.png" style="height:92px;width:225px"></a><br>
</div></div></div></div></div></div></div>
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">--<div><a href="http://schwehr.org" target="_blank">http://schwehr.org</a></div></div>
</div>