[gdal-dev] With 1.6.2, still can't read ArcInfo binary grids: GDAL Error 3:Attempt to read past EOF inC:\Temp\GeoEcoTemp_Jason\tmpopx_s3\projected0a/../info/arc.dir.

Sat Aug 8 10:16:16 EDT 2009

Evan and Roger,

Thanks for clarifying the situation and coming up with a solution. I'm very
happy to hear that this can probably be addressed with the existing 1.6.2
release candidate. Kudos to both of you.

The idea of bubbling this non-fatal error up as a warning from rgdal sounds
ok, but not great as a long term solution. If I bubble it up to my user (an
ecologist, typically, sitting in front of a GUI) I will need to document
that they can safely ignore this warning because it does not really mean
anything. Or I could try to catch this specific warning and just not report
it. Roger faces the same decision about bubbling for his code (he just has a
more programming-savvy user, usually sitting in front of the R console).
Ultimately it would be best if GDAL would not report this warning. If the
root problem is that ESRI is generating grids that violate their own format,
then we should tell them (I can contact them, if you don't want to). But if
their grids are correct, GDAL should be fixed whenever the resources are
available.

Thanks again!

Jason

-----Original Message-----
From: Roger Bivand [mailto:Roger.Bivand at nhh.no] 
Sent: Saturday, August 08, 2009 9:21 AM
To: Even Rouault
Cc: Jason Roberts; gdal-dev at lists.osgeo.org
Subject: Re: [gdal-dev] With 1.6.2,still can't read ArcInfo binary grids:
GDAL Error 3:Attempt to read past EOF
inC:\Temp\GeoEcoTemp_Jason\tmpopx_s3\projected0a/../info/arc.dir.

On Sat, 8 Aug 2009, Even Rouault wrote:

>
> ----- Original Message ----- From: Jason Roberts
> To: gdal-dev at lists.osgeo.org
> Cc: 'Roger Bivand'
> Sent: Saturday, August 08, 2009 12:05 AM
> Subject: [gdal-dev] With 1.6.2,still can't read ArcInfo binary grids: GDAL

> Error 3:Attempt to read past EOF 
> inC:\Temp\GeoEcoTemp_Jason\tmpopx_s3\projected0a/../info/arc.dir.
>
>
>> Hi all,
>
>> I am using rgdal to read ArcInfo binary grids produced by ArcGIS. The 
>> latest public version of rgdal, which binds with GDAL 1.6.1, fails with
the
>> error message in the subject line of this email. I reported this a couple

>> of weeks ago in 
>> http://lists.osgeo.org/pipermail/gdal-dev/2009-July/021369.html.
>
>> Today, the rgdal developer (Roger Bivand) provided me with an rgdal built

>> against the 1.6.2 release candidate issued 31 July. The problem still 
>> exists.
>
> OK, I've looked more closely at 
>
http://rgdal.cvs.sourceforge.net/viewvc/rgdal/rgdal/src/gdal-bindings.cpp?re
vision=1.32&view=markup 
> and I see RGDAL defines its own error handler for GDAL errors (around
lines 
> 114-134). So the fix pushed in 1.6.2 that solves the issue for Python and 
> other bindings languages will not help for the RGDAL case as the error()
call 
> probably causes a R exception to be thrown. As I've no knowledge of R, it
is 
> just wild guessing.

Even, Jason,

Thanks, this looks helpful. I'll try to see how it fits into the error 
handler put in place from the R side. An R error() unwinds all the way up 
the call trace in C/C++ through to the R code; there is a warning() that 
could be used for non-fatal GDAL errors (though it isn't obvious which 
ones shouldn't be fatal, and since we're in a running R session, we cannot 
really allow anything not safe to persist (like a handle to a dataset not 
opened properly).

Jason, please send me off-list an example offending file (or link if big) 
so that I'll know when I'm making progress.

Roger

>
> RGDAL_OpenDataset() could be modified so that its own error handler is not

> used during GDALOpen. Something like this might do it :
>
> ...
> CPLErrorReset();
> CPLPushErrorHandler(CPLDefaultErrorHandler);
> GDALDataset *pDataset = (GDALDataset *) GDALOpen(fn, RWFlag);
> CPLPopErrorHandler();
> // Similarly to SWIG bindings, the following lines will cause 
> RGDAL_OpenDataset() to fail on - unclearred - errors even if pDataset is
not 
> NULL.
> // They could also be just removed. While pDataset != NULL, there's some
hope 
> ;-)
> CPLErr eclass = CPLGetLastErrorType();
> if (pDataset != NULL && eclass == CE_Failure)
> {
>   GDALClose(pDataset);
>   pDataset = NULL;
>   __errorHandler(eclass, CPLGetLastErrorNo(), CPLGetLastErrorMsg());
> }
>
> if (pDataset == NULL)
>   error("Could not open file: %s\n", filename);
> ....
>
> Note this is uncompiled & untested, but that should avoid similar
situations 
> in other drivers (non fatal errors triggered during dataset opening and 
> cleared by CPLErrorReset() ) to occur .
>
>> The problem had apparently been seen before. My original email prompted a

>> review of the issue, including mention of it being tracked in tickets
>> #2447 and #3031. Evan made some checkins on 22 July that I thought would 
>> result in this being fixed in both the trunk and the next release of 1.6
>> (which would be 1.6.2). But I guess I did not fully understand it, and I 
>> missed the fact that neither of these issues appeared in Howard's 31 July
>> release candidate email.
>
> The tickets are not mentionned in Howard's email because the fix is only 
> partial and the tickets still open.
>
>> So: is there any chance the GDAL team can fix this in either 1.6.2 or, if

>> it is now too late, 1.6.3?
>
>> It may be that this fell off the radar because the severity of the
tickets 
>> (for example, #3031 is minor), or because there was a suggested
workaround
>> that users could simply rename the "info" directory to prevent GDAL from 
>> failing. Well the issue is indeed severe if you are an ArcGIS user. In my
>> case, the rasters were produced by ArcGIS 9.3.1. Presumably ESRI is 
>> generating the rasters correctly. The workaround of renaming the info
>> directory is not a good idea. That renders all of the rasters in a 
>> directory inaccessible to ArcGIS. It is a major corruption of the ArcInfo

>> binary grid
>> format to do so. I'm not sure why it was suggested in the first place,
but 
>> it is a very bad idea.
>
> Eh! workarounds are workarounds and generally not very elegant. It enables

> GDAL to avoids reading the VAT table, but if ArcGIS doesn't like it,
that's 
> another story... The advantage with renaming is that you can undo it when 
> necessary... Anyway, you're free of ignoring it if you don't like it.
>
>> I'm sorry for not staying on top of this issue and somehow missing that
it 
>> was not put into 1.6.2 and that the workaround is such as bad idea. In
any
>> case, could you please consider fixing this in 1.6 soon? I can provide 
>> example data if needed.
>
> #3031 has already a link to sample data that triggers the error message.
You 
> just need to find and convince someone that has time and knowledge of the
AVC 
> driver.
>
>> Thanks,
>
>> Jason
>
>
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev 
>
>

-- 
Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Helleveien 30, N-5045 Bergen,
Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43
e-mail: Roger.Bivand at nhh.no