[gdal-dev] Grib2 Question

[Roarke Gaskill]

Wouldn't the the max size be limited by the number of bytes read?  So in
this case 4 bytes.

http://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_sect5.shtml

Looking at netcdf's implementation they treat the value as a 32 bit signed
int.

https://github.com/Unidata/thredds/blob/5.0.0/grib/src/main/java/ucar/nc2/grib/grib2/Grib2SectionDataRepresentation.java#L69

I am dealing with proprietary grib2 files that do break the current limit
of 33554432.



On Tue, Nov 7, 2017 at 4:03 PM, Even Rouault <even.rouault at spatialys.com>
wrote:

> On mardi 7 novembre 2017 13:51:30 CET Kurt Schwehr wrote:
>
> > It's possible to cause massive allocations with a tiny corrupted grib
> file
>
> > causing an out-of-memory. I found that case with the llvm ASAN fuzzer. If
>
> > you have a specification that gives a more reasoned maximum or a better
>
> > overall check, I'm listening. I definitely think the sanity checking can
>
> > be improved. Mostly I just try to survive the g2clib code. It doesn't
>
> > come with tests and digging through GRIB specs to match up to g2clib
> source
>
> > isn't my favorite thing to do.
>
> >
>
> > https://github.com/OSGeo/gdal/commit/ae92f7fb8e32381124a37588d27b9a
> f695afce2
>
> > 0
>
>
>
> I guess that if Roarke is asking the question he might have a dataset that
> breaks this limit ? If so, we might consider reverting that change, or
> making it more robust (which can be very tricky I know. Perhaps some
> heuristics with the file size ?), or just using it in fuzzing mode and not
> in production for now. And a pointer to such a dataset would be much
> appreciated.
>
>
>
> (By the way: 2<<24 is IMHO an usual way of writing a limit. I confused it
> with 2^24 initially. So 1 << 25 would perhaps be better. Or just in decimal
> form as it is completely arbitary and not related to a binary encoding)
>
>
>
> Even
>
>
>
> --
>
> Spatialys - Geospatial professional services
>
> http://www.spatialys.com
>



-- 
*Roarke **Gaskill  *|Senior Software Engineer
*e:* roarke.gaskill at weather.com
<http://weather.com/apps> <http://weather.com/apps>
<http://weather.com/apps> <http://weather.com/apps>
<http://weather.com/apps> <http://weather.com/apps>
<http://weather.com/apps> <http://weather.com/apps>
<http://weather.com/apps> <http://weather.com/apps>
<http://weather.com/apps> <http://weather.com/apps>
<http://weather.com/apps> <http://weather.com/apps>
<http://weather.com/apps> <http://weather.com/apps>
<http://weather.com/apps>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20171107/fb5bc99c/attachment.html>