[gdal-dev] RE: GDALOpen crash on large GRIB files - workaround

[Sjur Kolberg]

Hi again,

This was obviously not a known problem. What we ended up with was to split the GRIB files up in smaller pieces using another software and importing bit by bit using GDAL. During the work we also had memory allocation problems in plain C++ code (myarray = new float[alotofcells]; gives "Out of memory" exception), so it seems that it is Windows that is not too happy with allocating large memory chunks. This amount of memory space, I guess, is most often needed by meteorological formats like grib, NetCDF or HDF, which are not too extensively used on Windows.

Does GDAL, or the drivers of often-large files, offer any possibility to restrict the subset of bands before calling GDALOpen() on the file?

Best regards,

Sjur K :-)


________________________________
From: gdal-dev-bounces at lists.osgeo.org [mailto:gdal-dev-bounces at lists.osgeo.org] On Behalf Of Sjur Kolberg
Sent: 10. desember 2009 00:48
To: gdal-dev at lists.osgeo.org
Cc: Kolbjørn Engeland
Subject: [gdal-dev] GDALOpen crash on large GRIB files


Hello, list:

I have problems reading some large GRIB files from ECMWF. I have several files, 29MB and 50MB works fine, 141 MB and above crashes. The crash is within the first GDALOpen() function, and I track it through GRIBDataset::Open() to GDALDataset::Setband(). The error arises in the last statement given below, and because papoBands==NULL. By the two breakpoints I can tell that papoBands is not NULL when entering the SetBand() function, so it seems to be the VSIRealloc function that returns NULL. The error doesn't occur for the same band each time I try, but typically runs to band number somewhere around 50000. (bandno is a combined index of forecast reference time, forecast lead time and variable).

I run GDAL 1.6.3 on Windows XP, compiled with Visual Studio 2008. The error appears both in debug and release version. The application uses about 144 MB when it crashes, but has been up to a peak memory usage of 283 MB before the crash. I have 3.5 GB memory on the PC, of which 2.5 appears to be available. Both GDAL and the application is compiled with extended memory sizes and /LARGEADDRESSAWARE, but I am not sure about how this works. The error message is:

Unhandled exception at 0x30c36acd (gdal16d-vc9.dll) in RegModel.exe: 0xC0000005: Access violation writing location 0x00036c74.

Has anyone had similar problems?
Best regards,
Sjur K :-)



void GDALDataset::SetBand( int nNewBand, GDALRasterBand * poBand )

{

/* -------------------------------------------------------------------- */

/* Do we need to grow the bands list? */

/* -------------------------------------------------------------------- */

if( nBands < nNewBand || papoBands == NULL ) {

    int i;

    if( papoBands == NULL )

        papoBands = (GDALRasterBand **)  /* Breakpoint before this statement */

            VSICalloc(sizeof(GDALRasterBand*), MAX(nNewBand,nBands));

    else

    papoBands = (GDALRasterBand **)

        VSIRealloc(papoBands, sizeof(GDALRasterBand*) *

            MAX(nNewBand,nBands));

    for( i = nBands; i < nNewBand; i++ )   /* Breakpoint before this statement; condition: papoBands=0 */

        papoBands[i] = NULL;    /* Offending statement, papoBands==NULL.



State of poBand at error:

-  poBand 0x357f4228 {start=172766438 subgNum=0 longFstLevel=0x746a3e28 "0[-] SFC (Ground or water surface)" ...} GDALRasterBand *
+  [GRIBRasterBand] {start=172766438 subgNum=0 longFstLevel=0x746a3e28 "0[-] SFC (Ground or water surface)" ...} GRIBRasterBand
+  GDALMajorObject {nFlags=49 sDescription={...} oMDMD={...} } GDALMajorObject
+  poDS 0x3424f4b0 {fp=0x3424fb08 pszProjection=0x35b3ce68 "GEOGCS["Coordinate System imported from GRIB file",DATUM["unknown",SPHEROID["Sphere",6367470,0]],PRIMEM["Greenwich",0],UNIT["degree",0.0174532925199433]]" pszDescription=0xcdcdcdcd <Bad Ptr> ...} GDALDataset *
  nBand 56094 int
  nRasterXSize 0 int
  nRasterYSize 0 int
  eDataType GDT_Float64 GDALDataType
  eAccess GA_ReadOnly GDALAccess
  nBlockXSize 45 int
  nBlockYSize 1 int
  nBlocksPerRow 0 int
  nBlocksPerColumn 0 int
  bSubBlockingActive 0 int
  nSubBlocksPerRow 0 int
  nSubBlocksPerColumn 0 int
+  papoBlocks 0x00000000 GDALRasterBlock * *
  nBlockReads 0 int
  bForceCachedIO 0 int
+  poMask 0x00000000 {poDS=??? nBand=??? nRasterXSize=??? ...} GDALRasterBand *
  bOwnMask false bool
  nMaskFlags 0 int

Status of this GDALDataset:

  i -858993460 int
  nBands 49093 int
  nNewBand 49094 int
+  papoBands 0x00000000 GDALRasterBand * *
-  this 0x3424f4b0 {fp=0x3424fab8 pszProjection=0x35b31048 "GEOGCS["Coordinate System imported from GRIB file",DATUM["unknown",SPHEROID["Sphere",6367470,0]],PRIMEM["Greenwich",0],UNIT["degree",0.0174532925199433]]" pszDescription=0xcdcdcdcd <Bad Ptr> ...} GDALDataset * const
+  [GRIBDataset] {fp=0x3424fab8 pszProjection=0x35b31048 "GEOGCS["Coordinate System imported from GRIB file",DATUM["unknown",SPHEROID["Sphere",6367470,0]],PRIMEM["Greenwich",0],UNIT["degree",0.0174532925199433]]" pszDescription=0xcdcdcdcd <Bad Ptr> ...} GRIBDataset
+  GDALMajorObject {nFlags=33 sDescription={...} oMDMD={...} } GDALMajorObject
+  poDriver 0x00000000 {pfnOpen=??? pfnCreate=??? pfnDelete=??? ...} GDALDriver *
  eAccess GA_ReadOnly GDALAccess
  nRasterXSize 45 int
  nRasterYSize 33 int
  nBands 49093 int
+  papoBands 0x00000000 GDALRasterBand * *
  bForceCachedIO 0 int
  nRefCount 1 int
  bShared 0 int
+  oOvManager {poDS=0x00000000 poODS=0x00000000 osOvrFilename={...} ...} GDALDefaultOverviews






-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/gdal-dev/attachments/20091211/3c860d99/attachment-0001.html