[gdal-dev] Grib2 Question

[Even Rouault]

On mardi 7 novembre 2017 14:21:27 CET Kurt Schwehr wrote:
> Yeah, 1 << ## would have been better.  Sigh.
> 
> But really, someone needs to work through the logic of this stuff and do
> something that actually works through what is reasonable.  Code that is
> intelligent and explains why it is doing is far preferable.  OOM is not
> okay in the world I work in, so I tried to pick a large number (basically
> at random) that should let almost all cases through until someone took more
> time.

I'm afraid this is unlikely that "someone" pops up by magic to address that, given the time 
investment it would represent. For work related to adding a write side to the GRIB2 driver, I 
might try in the coming weeks to resync with newest version of g2clib & degrib, which by 
itself will be certainly a challenging exercice given all the patching we have done through the 
years, but adressing such issues of being robust with "hostile" files is definitely not in the 
scope of work.

I'm not sure if there are other C/C++ grib1+grib2 libs that would be better in that aspect, 
while having at least the same level of functionality, and a permissive license.

> 
> allowing a full range number without checking to see if the size is
> reasonable is a *major* problem for me.  An OOM is a fatal (as in no error
> message, your process is dead) situation in my world, so allowing larger
> without checking seems unpleasant.
> 
> Perhaps a config option to allow massive files through in the short term?

I think the vast majority of users are likely less concerned with potential OOM situations on 
hostile/corrupted files, but more willing to have a working behaviour with real world 
datasets, so the reverse logic would seem better to me.

Or can we increase this threshold so that it works with datasets likes Roarke's one ? Kurt, 
what would be the maximum value you could bear ?
Roarke, can you add some breakpoint / debug message to see what value of *ndpts you get 
for your dataset ?

If we go with safe mode by default and a config option to disable it, then it should output a 
warning that explains which config option to enable to disable the limitation. + in the doc 
page of the driver.

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20171108/34841fa6/attachment-0001.html>