[GRASS5] error encountered in r.terraflow
Shaun Walbridge
walbridge at nceas.ucsb.edu
Thu Apr 27 20:04:41 EDT 2006
Apologies, I forgot to re-attach the patch to this list. The only
changes I've found necessary are those couple spots in ami_stream.h.
We've successfully run Terraflow on 4GB+ datasets after the swap to
fseeko/ftello.
Cheers,
Shaun
Andrew Danner wrote:
>Yes, the problem is related to using fseek and ftell (which are only 32
>bit) instead of fseeko/ftello which support off_t offsets which can be
>64-bits long if CPPFLAGS includes -D_FILE_OFFSET_BITS=64
>-D_LARGEFILE_SOURCE.
>
>I'm not sure if this completely fixes the problem. I would have to see
>the patch, but I thought some of the terraflow types for storing offsets
>were of type "long" instead of "off_t", so some additional tweaking
>needs to be done.
>
>I have some familiarity with Terraflow, but I have been concentrating my
>efforts in developing a new Terraflow-like module that should be more
>robust and have some additional features. If you have a patch that
>works, could you submit it to CVS?
>
>If I get some free time, I can look into fixes if things still do not
>work after the patch.
>
>-Andy
>
>On Thu, 2006-04-27 at 13:38 -0700, Shaun Walbridge wrote:
>
>
>>My response to the same question, on GRASSLIST:
>>
>>This error occurs because currently Terraflow uses the standard C file
>>stream calls. The permanent solution to this would be to add some
>>Makefile magic to detect the current platform, and use the appropriate
>>stream types. The temporary solution is to use the patch I've attached.
>>You'll need the grass source tree, to apply the patch do something like:
>>
>>cd /grass6/raster/r.terraflow/IOStream/include
>>patch < ami_stream.h.patch
>>cd ../..
>>make clean
>>make
>>
>>Then install this modified terraflow.
>>
>>Do any GRASS devs have the time and expertise to make terraflow
>>largefile aware? In my experience this (very minor) patch does the
>>trick, should be just fseek / ftell which need proper detection
>>
>>Cheers,
>>Shaun Walbridge
>>
>>abhay menon wrote:
>>
>>
>>
>>>Hello list,
>>>
>>>after the prior process of getting 2gb data conversion using gdal with
>>>grass61cvs been a success.
>>>
>>>i am facing a problem with terraflow process on the 2gb dataset(ESRI
>>>Ascii Raster Grid) which i converted to grass raster grid file.
>>>
>>>r.terraflow December 2003
>>>region size is 19954 x 21199
>>>cell newdata header compatible with region header
>>>elevation stored as FLOAT (4B) WARNING: raster newdata is of type
>>>CELL_TYPE --you should use r.terraflow.short
>>>
>>>STREAM temporary files in /var/tmp (THESE INTERMEDIATE STREAMS WILL
>>>NOT BE DELETED IN CASE OF ABNORMAL TERMINATION OF THE PROGRAM. TO SAVE
>>>SPACE PLEASE DELETE THESE FILES MANUALLY!)
>>>SFD (D8) flow direction
>>>D8CUT=999999986991104.000000
>>>memory size: 1000.00M (1048576000) bytes
>>>Memory manager registering memory in MM_WARN_ON_MEMORY_EXCEEDED mode.
>>>reading data from newdata to stream /var/tmp/STREAM_NnrFqS:
>>>
>>>total elements=423004846, nodata elements=153409740
>>>largest temporary files:
>>> FILL: 22.06G (23688271376) [423004846 elements, 56B each]
>>> FLOW: 20.09G (21567608480) [269595106 elements, 80B each]
>>>Will need at least 44.12G (47376542752) space available in /var/tmp
>>>------------------------------
>>>COMPUTING FLOW DIRECTIONS
>>>classifying nodata (inner & boundary)
>>>finding nodata
>>>relabeling nodata
>>>MM warning: limit=1048576000B. allocating 524140056B. limit exceeded
>>>by 58762B.
>>>:
>>>:
>>>merging relabeled grid
>>>MM warning: limit=1048576000B. allocating 524140056B. limit exceeded
>>>by 58762B.
>>>:
>>>:
>>>----------
>>>assigning preliminary directions
>>>----------
>>>finding flat areas (plateaus and depressions)
>>>plateau.cc:starting memscan
>>>plateau.cc:memscan done
>>>removing duplicate plateaus
>>>MM warning: limit=1048576000B. allocating 524140060B. limit exceeded
>>>by 58770B.
>>>:
>>>relabeling plateaus
>>>MM warning: limit=1048576000B. allocating 524140060B. limit exceeded
>>>by 58766B.
>>>generating plateau statistics
>>>MM warning: limit=1048576000B. allocating 523975900B. limit exceeded
>>>by 58766B.
>>>----------
>>>assigning directions on plateaus
>>>MM warning: limit=1048576000B. allocating 524222112B. limit exceeded
>>>by 48098B.
>>>--------------
>>>generating watersheds and watershed graph
>>>creating windows
>>>warning: using slower scan
>>>r.terraflow: 3scan.h:163: void scan3line(FUN&, AMI_STREAM<T>*,
>>>AMI_STREAM<T>*, AMI_STREAM<T>*, BASETYPE, dimension_type) [with T =
>>>waterGridType, BASETYPE = waterWindowBaseType, FUN = waterWindower]:
>>>Assertion `ae == AMI_ERROR_END_OF_STREAM' failed.
>>>
>>>
>>>and also seem not to process data above 1gb(ESRI Ascii Raster Grid) too.
>>>is there anything else i missed while compilation of grass?
>>>
>>>PS. i had this was my configuration, compilation and installation
>>>statement from grass61cvs.
>>>
>>>./configure --enable-shared --with-fftw --with-includes=/usr/include
>>>--with-libs=/usr/lib --with-motif --with-freetype=yes
>>>--with-freetype-includes=/usr/include/freetype2 --with-nls
>>>--with-gdal=/usr/local/bin/gdal-config --with-proj
>>>--with-proj-includes=/usr/include --with-proj-libs=/usr/lib --with-glw
>>>--with-cxx --with-mysql --with-mysql-includes=/usr/include/mysql
>>>--with-mysql-libs=/usr/lib/mysql --with-postgres
>>>--with-postgres-includes=/usr/include/pgsql
>>>--with-postgres-libs=/usr/lib --with-odbc=yes
>>>--with-odbc-includes=/usr/include --with-odbc-libs=/usr/lib
>>>--with-glw=yes --enable-largefile;
>>>make;
>>>make install
>>>
>>>--
>>>Best Regrads
>>>
>>>Abhay Menon
>>>
>>>
>>_______________________________________________
>>grass5 mailing list
>>grass5 at grass.itc.it
>>http://grass.itc.it/mailman/listinfo/grass5
>>
>>
>
>
>
-------------- next part --------------
--- ami_stream.h 2004-11-09 05:29:58.000000000 -0800
+++ ami_stream.h.edit 2006-04-27 10:44:48.000000000 -0700
@@ -394,8 +394,8 @@
seek_offset = offset * sizeof(T);
}
- if (fseek(fp, seek_offset, SEEK_SET) == -1) {
- cerr << "AMI_STREAM::seek offset=" << seek_offset << "failed.\n";
+ if (fseeko(fp, seek_offset, SEEK_SET) == -1) {
+ cerr << "AMI_STREAM::seek offset=" << seek_offset << " failed.\n";
assert(0);
exit(1);
}
@@ -463,7 +463,7 @@
assert(fp);
//if we go past substream range
- if ((logical_eos >= 0) && ftell(fp) >= sizeof(T) * logical_eos) {
+ if ((logical_eos >= 0) && ftello(fp) >= sizeof(T) * logical_eos) {
return AMI_ERROR_END_OF_STREAM;
} else {
@@ -489,7 +489,7 @@
assert(fp);
//if we go past substream range
- if ((logical_eos >= 0) && ftell(fp) >= sizeof(T) * logical_eos) {
+ if ((logical_eos >= 0) && ftello(fp) >= sizeof(T) * logical_eos) {
return AMI_ERROR_END_OF_STREAM;
} else {
@@ -512,7 +512,7 @@
assert(fp);
//if we go past substream range
- if ((logical_eos >= 0) && ftell(fp) >= sizeof(T) * logical_eos) {
+ if ((logical_eos >= 0) && ftello(fp) >= sizeof(T) * logical_eos) {
return AMI_ERROR_END_OF_STREAM;
} else {
@@ -532,7 +532,7 @@
assert(fp);
//if we go past substream range
- if ((logical_eos >= 0) && ftell(fp) >= sizeof(T) * logical_eos) {
+ if ((logical_eos >= 0) && ftello(fp) >= sizeof(T) * logical_eos) {
return AMI_ERROR_END_OF_STREAM;
} else {
More information about the grass-dev
mailing list