<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=Content-Type content="text/html; charset=iso-8859-1">
<META content="MSHTML 6.00.6000.16945" name=GENERATOR></HEAD>
<BODY>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff
size=2></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=027292912-11122009>Hi again,</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=027292912-11122009></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=027292912-11122009>This was obviously not a known problem. What we
ended up with was to split the GRIB files up in smaller pieces using another
software and importing bit by bit using GDAL. During the work we also had
memory allocation problems in plain C++ code (myarray = new float[alotofcells];
gives "Out of memory" exception), so it seems that it is Windows that is not too
happy with allocating large memory chunks. This amount of memory space, I guess,
is most often needed by meteorological formats like grib, NetCDF or HDF, which
are not too extensively used on Windows.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=027292912-11122009></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=027292912-11122009>Does GDAL, or the drivers of often-large files, offer
any possibility to restrict the subset of bands before calling GDALOpen()
on the file?</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=027292912-11122009></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><SPAN class=027292912-11122009><FONT face=Arial
color=#0000ff size=2>Best regards,</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=027292912-11122009><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=027292912-11122009><FONT face=Arial
color=#0000ff size=2>Sjur K :-)</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=027292912-11122009><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV><FONT face=Arial color=#0000ff
size=2></FONT><FONT face=Arial color=#0000ff size=2></FONT><FONT face=Arial
color=#0000ff size=2></FONT><BR>
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> gdal-dev-bounces@lists.osgeo.org
[mailto:gdal-dev-bounces@lists.osgeo.org] <B>On Behalf Of </B>Sjur
Kolberg<BR><B>Sent:</B> 10. desember 2009 00:48<BR><B>To:</B>
gdal-dev@lists.osgeo.org<BR><B>Cc:</B> Kolbjørn Engeland<BR><B>Subject:</B>
[gdal-dev] GDALOpen crash on large GRIB files<BR></FONT><BR></DIV>
<DIV></DIV>
<DIV><FONT face=Arial size=2><SPAN
class=046403922-09122009></SPAN></FONT> </DIV>
<DIV><FONT face=Arial size=2><SPAN class=046403922-09122009>Hello,
list:</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN
class=046403922-09122009></SPAN></FONT> </DIV>
<DIV><FONT face=Arial size=2><SPAN class=046403922-09122009>I have problems
reading some large GRIB files from ECMWF. I have several files, 29MB and 50MB
works fine, 141 MB and above crashes. The crash is within the first GDALOpen()
function, and I track it through GRIBDataset::Open() to
GDALDataset::Setband(). The error arises in the last statement given below, and
because papoBands==NULL. By the two breakpoints I can tell that papoBands is not
NULL when entering the SetBand() function, so it seems to be the VSIRealloc
function that returns NULL. The error doesn't occur for the same band each time
I try, but typically runs to band number somewhere around 50000. (bandno is a
combined index of forecast reference time, forecast lead time and
variable).</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN
class=046403922-09122009></SPAN></FONT> </DIV>
<DIV><FONT face=Arial size=2><SPAN class=046403922-09122009>I run GDAL 1.6.3 on
Windows XP, compiled with Visual Studio 2008. The error appears both in debug
and release version. The application uses about 144 MB when it crashes, but has
been up to a peak memory usage of 283 MB before the crash. I have 3.5 GB memory
on the PC, of which 2.5 appears to be available. Both GDAL and the application
is compiled with extended memory sizes and /LARGEADDRESSAWARE, but I am not sure
about how this works. The error message is:</SPAN></FONT></DIV>
<DIV><FONT size=2><SPAN class=046403922-09122009></SPAN></FONT> </DIV>
<DIV><FONT size=2><SPAN class=046403922-09122009>Unhandled exception at
0x30c36acd (gdal16d-vc9.dll) in RegModel.exe: 0xC0000005: Access violation
writing location 0x00036c74.</SPAN></FONT></DIV>
<DIV><FONT size=2><SPAN class=046403922-09122009></SPAN></FONT> </DIV>
<DIV><FONT face=Arial size=2><SPAN class=046403922-09122009><SPAN
class=046403922-09122009>Has anyone had similar
problems?</SPAN></SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN class=046403922-09122009><SPAN
class=046403922-09122009><SPAN class=046403922-09122009><FONT size=2>Best
regards,</FONT></SPAN></SPAN></SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN class=046403922-09122009><SPAN
class=046403922-09122009><SPAN class=046403922-09122009></SPAN><SPAN
class=046403922-09122009><FONT size=2>Sjur K :-)</FONT></SPAN></DIV>
<P><SPAN class=046403922-09122009><FONT
size=2></FONT></SPAN> </P></SPAN></SPAN></FONT>
<DIV><FONT color=#0000ff><FONT color=#0000ff>
<P><FONT size=2>void</FONT></FONT></FONT><FONT size=2> GDALDataset::SetBand(
<FONT color=#0000ff><FONT color=#0000ff>int</FONT></FONT> nNewBand,
GDALRasterBand * poBand )</FONT></P>
<P><FONT size=2>{</FONT></P><FONT color=#008000><FONT color=#008000>
<P><FONT size=2>/*
--------------------------------------------------------------------
*/</FONT></P>
<P><FONT size=2>/* Do we need to grow the bands list? */</FONT></P>
<P><FONT size=2>/*
--------------------------------------------------------------------
*/</FONT></P></FONT></FONT>
<P><FONT size=2><FONT color=#0000ff><FONT color=#0000ff>if</FONT></FONT>( nBands
< nNewBand || papoBands == NULL ) {</FONT></P>
<P><FONT size=2><FONT color=#0000ff><SPAN
class=046403922-09122009> </SPAN>int</FONT> i;</FONT></P>
<P><FONT size=2><FONT color=#0000ff><SPAN
class=046403922-09122009> </SPAN>if</FONT>( papoBands == NULL
)</FONT></P>
<P><FONT size=2><SPAN
class=046403922-09122009>
</SPAN>papoBands = (GDALRasterBand **)<SPAN
class=046403922-09122009> <FONT color=#ff0000>/* Breakpoint before
this statement */</FONT></SPAN></FONT></P>
<P><SPAN class=046403922-09122009></SPAN><FONT size=2><SPAN
class=046403922-09122009>
</SPAN>VSICalloc(<FONT color=#0000ff><FONT
color=#0000ff>sizeof</FONT></FONT>(GDALRasterBand*),
MAX(nNewBand,nBands));</FONT></P>
<P><FONT color=#0000ff><FONT size=2><SPAN
class=046403922-09122009> </SPAN>else</FONT></FONT></P>
<P><FONT size=2><SPAN class=046403922-09122009>
</SPAN>papoBands = (GDALRasterBand **)<SPAN
class=046403922-09122009>
</SPAN></FONT></P>
<P><SPAN class=046403922-09122009></SPAN><SPAN
class=046403922-09122009></SPAN><FONT size=2><SPAN
class=046403922-09122009>
</SPAN>VSIRealloc(papoBands, <FONT color=#0000ff><FONT
color=#0000ff>sizeof</FONT></FONT>(GDALRasterBand*) *</FONT></P>
<P><FONT size=2><SPAN
class=046403922-09122009>
</SPAN>MAX(nNewBand,nBands));</FONT></P>
<P><FONT size=2><FONT color=#0000ff><SPAN
class=046403922-09122009> </SPAN>for</FONT>( i = nBands; i
< nNewBand; i++ )<SPAN class=046403922-09122009> <FONT
color=#ff0000>/* Breakpoint before this statement; condition: papoBands=0
*/</FONT></SPAN></FONT></P>
<P><FONT size=2><SPAN
class=046403922-09122009>
</SPAN>papoBands[i] = NULL;<SPAN class=046403922-09122009>
</SPAN><FONT color=#ff0000>/* <SPAN class=046403922-09122009>Offending
statement, papoBands==NULL.</SPAN></FONT></FONT></P>
<P><FONT size=2><FONT color=#ff0000><SPAN
class=046403922-09122009></SPAN></FONT></FONT> </P>
<P><SPAN class=046403922-09122009><FONT size=2>State of poBand at
error:</FONT></SPAN></P>
<P><FONT size=2>- poBand 0x357f4228 {start=172766438 subgNum=0
longFstLevel=0x746a3e28 "0[-] SFC (Ground or water surface)"
...} GDALRasterBand
*<BR>+ [GRIBRasterBand] {start=172766438 subgNum=0
longFstLevel=0x746a3e28 "0[-] SFC (Ground or water surface)"
...} GRIBRasterBand<BR>+ GDALMajorObject {nFlags=49
sDescription={...} oMDMD={...}
} GDALMajorObject<BR>+ poDS 0x3424f4b0 {fp=0x3424fb08
pszProjection=0x35b3ce68 "GEOGCS["Coordinate System imported from GRIB
file",DATUM["unknown",SPHEROID["Sphere",6367470,0]],PRIMEM["Greenwich",0],UNIT["degree",0.0174532925199433]]"
pszDescription=0xcdcdcdcd <Bad Ptr> ...} GDALDataset
*<BR> nBand 56094 int<BR> nRasterXSize 0 int<BR> nRasterYSize 0 int<BR> eDataType GDT_Float64 GDALDataType<BR> eAccess GA_ReadOnly GDALAccess<BR> nBlockXSize 45 int<BR> nBlockYSize 1 int<BR> nBlocksPerRow 0 int<BR> nBlocksPerColumn 0 int<BR> bSubBlockingActive 0 int<BR> nSubBlocksPerRow 0 int<BR> nSubBlocksPerColumn 0 int<BR>+ papoBlocks 0x00000000 GDALRasterBlock
*
*<BR> nBlockReads 0 int<BR> bForceCachedIO 0 int<BR>+ poMask 0x00000000
{poDS=??? nBand=??? nRasterXSize=??? ...} GDALRasterBand
*<BR> bOwnMask false bool<BR> nMaskFlags 0 int<BR></FONT></P>
<P><SPAN class=046403922-09122009><FONT size=2>Status of this
GDALDataset:</FONT></SPAN></P>
<P><SPAN class=046403922-09122009><FONT face=Arial
size=2> i -858993460 int<BR> nBands 49093 int<BR> nNewBand 49094 int<BR>+ papoBands 0x00000000 GDALRasterBand
* *<BR>- this 0x3424f4b0 {fp=0x3424fab8 pszProjection=0x35b31048
"GEOGCS["Coordinate System imported from GRIB
file",DATUM["unknown",SPHEROID["Sphere",6367470,0]],PRIMEM["Greenwich",0],UNIT["degree",0.0174532925199433]]"
pszDescription=0xcdcdcdcd <Bad Ptr> ...} GDALDataset *
const<BR>+ [GRIBDataset] {fp=0x3424fab8 pszProjection=0x35b31048
"GEOGCS["Coordinate System imported from GRIB
file",DATUM["unknown",SPHEROID["Sphere",6367470,0]],PRIMEM["Greenwich",0],UNIT["degree",0.0174532925199433]]"
pszDescription=0xcdcdcdcd <Bad Ptr>
...} GRIBDataset<BR>+ GDALMajorObject {nFlags=33
sDescription={...} oMDMD={...}
} GDALMajorObject<BR>+ poDriver 0x00000000 {pfnOpen=???
pfnCreate=??? pfnDelete=??? ...} GDALDriver
*<BR> eAccess GA_ReadOnly GDALAccess<BR> nRasterXSize 45 int<BR> nRasterYSize 33 int<BR> nBands 49093 int<BR>+ papoBands 0x00000000 GDALRasterBand
*
*<BR> bForceCachedIO 0 int<BR> nRefCount 1 int<BR> bShared 0 int<BR>+ oOvManager {poDS=0x00000000
poODS=0x00000000 osOvrFilename={...}
...} GDALDefaultOverviews<BR></FONT></SPAN></P>
<P><SPAN class=046403922-09122009><FONT size=2></FONT></SPAN> </P>
<P><SPAN class=046403922-09122009><FONT size=2></FONT></SPAN> </P>
<P><SPAN class=046403922-09122009><FONT
size=2></FONT></SPAN> </P></DIV></BODY></HTML>