<div dir="ltr">Why not something like this and just let me pick a small number?<div><br></div><div>#ifdef GRIB_MAX_ALLOC</div><div>const int knMaxAllloc = GRIB_MAX_ALLOC;</div><div>#else</div><div>const int knMaxAlloc = some massive number; </div><div>#endif</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Nov 7, 2017 at 3:18 PM, Even Rouault <span dir="ltr"><<a href="mailto:even.rouault@spatialys.com" target="_blank">even.rouault@spatialys.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><u></u>
<div style="font-family:'Sans Serif';font-size:9pt;font-weight:400;font-style:normal"><span class="">
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">On mardi 7 novembre 2017 14:21:27 CET Kurt Schwehr wrote:</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> Yeah, 1 << ## would have been better. Sigh.</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> But really, someone needs to work through the logic of this stuff and do</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> something that actually works through what is reasonable. Code that is</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> intelligent and explains why it is doing is far preferable. OOM is not</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> okay in the world I work in, so I tried to pick a large number (basically</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> at random) that should let almost all cases through until someone took more</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> time.</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
</span><p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">I'm afraid this is unlikely that "someone" pops up by magic to address that, given the time investment it would represent. For work related to adding a write side to the GRIB2 driver, I might try in the coming weeks to resync with newest version of g2clib & degrib, which by itself will be certainly a challenging exercice given all the patching we have done through the years, but adressing such issues of being robust with "hostile" files is definitely not in the scope of work.</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">I'm not sure if there are other C/C++ grib1+grib2 libs that would be better in that aspect, while having at least the same level of functionality, and a permissive license.</p><span class="">
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> allowing a full range number without checking to see if the size is</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> reasonable is a *major* problem for me. An OOM is a fatal (as in no error</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> message, your process is dead) situation in my world, so allowing larger</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> without checking seems unpleasant.</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> Perhaps a config option to allow massive files through in the short term?</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
</span><p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">I think the vast majority of users are likely less concerned with potential OOM situations on hostile/corrupted files, but more willing to have a working behaviour with real world datasets, so the reverse logic would seem better to me.</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">Or can we increase this threshold so that it works with datasets likes Roarke's one ? Kurt, what would be the maximum value you could bear ?</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">Roarke, can you add some breakpoint / debug message to see what value of *ndpts you get for your dataset ?</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">If we go with safe mode by default and a config option to disable it, then it should output a warning that explains which config option to enable to disable the limitation. + in the doc page of the driver.</p><span class="">
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">Even</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">-- </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">Spatialys - Geospatial professional services</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"><a href="http://www.spatialys.com" target="_blank">http://www.spatialys.com</a></p></span></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">--<div><a href="http://schwehr.org" target="_blank">http://schwehr.org</a></div></div>
</div>