[GRASS5] error encountered in r.terraflow

[Andrew Danner]

Yes, the problem is related to using fseek and ftell (which are only 32
bit) instead of fseeko/ftello which support off_t offsets which can be
64-bits long if CPPFLAGS includes -D_FILE_OFFSET_BITS=64
-D_LARGEFILE_SOURCE.

I'm not sure if this completely fixes the problem. I would have to see
the patch, but I thought some of the terraflow types for storing offsets
were of type "long" instead of "off_t", so some additional tweaking
needs to be done. 

I have some familiarity with Terraflow, but I have been concentrating my
efforts in developing a new Terraflow-like module that should be more
robust and have some additional features.  If you have a patch that
works, could you submit it to CVS?

If I get some free time, I can look into fixes if things still do not
work after the patch. 

-Andy 

On Thu, 2006-04-27 at 13:38 -0700, Shaun Walbridge wrote:
> My response to the same question, on GRASSLIST:
> 
> This error occurs because currently Terraflow uses the standard C file 
> stream calls. The permanent solution to this would be to add some 
> Makefile magic to detect the current platform, and use the appropriate 
> stream types.  The temporary solution is to use the patch I've attached. 
> You'll need the grass source tree, to apply the patch do something like:
> 
> cd /grass6/raster/r.terraflow/IOStream/include
> patch < ami_stream.h.patch
> cd ../..
> make clean
> make
> 
> Then install this modified terraflow.
> 
> Do any GRASS devs have the time and expertise to make terraflow 
> largefile aware? In my experience this (very minor) patch does the 
> trick, should be just fseek / ftell which need proper detection
> 
> Cheers,
> Shaun Walbridge
> 
> abhay menon wrote:
> 
> > Hello list,
> >  
> > after the prior process of getting 2gb data conversion using gdal with 
> > grass61cvs been a success.
> >  
> > i am facing a problem with terraflow process on the 2gb dataset(ESRI 
> > Ascii Raster Grid) which i converted to grass raster grid file.
> >  
> > r.terraflow December 2003
> > region size is 19954 x 21199
> > cell newdata header compatible with region header
> > elevation stored as FLOAT (4B) WARNING: raster newdata is of type 
> > CELL_TYPE --you should use r.terraflow.short
> >
> > STREAM temporary files in /var/tmp  (THESE INTERMEDIATE STREAMS WILL 
> > NOT BE DELETED IN CASE OF ABNORMAL TERMINATION OF THE PROGRAM. TO SAVE 
> > SPACE PLEASE DELETE THESE FILES MANUALLY!)
> > SFD (D8) flow direction
> > D8CUT=999999986991104.000000
> > memory size: 1000.00M (1048576000) bytes
> > Memory manager registering memory in MM_WARN_ON_MEMORY_EXCEEDED mode.
> > reading data from newdata to stream /var/tmp/STREAM_NnrFqS:
> >
> > total elements=423004846, nodata elements=153409740
> > largest temporary files:
> >    FILL: 22.06G (23688271376) [423004846 elements, 56B each]
> >    FLOW: 20.09G (21567608480) [269595106 elements, 80B each]
> > Will need at least 44.12G (47376542752) space available in /var/tmp
> > ------------------------------
> > COMPUTING FLOW DIRECTIONS
> > classifying nodata (inner & boundary)
> > finding nodata
> > relabeling nodata
> > MM warning: limit=1048576000B. allocating 524140056B.  limit exceeded 
> > by 58762B.
> > :
> > :
> > merging relabeled grid
> > MM warning: limit=1048576000B. allocating 524140056B.  limit exceeded 
> > by 58762B.
> > :
> > :
> > ----------
> > assigning preliminary directions
> > ----------
> > finding flat areas (plateaus and depressions)
> > plateau.cc:starting memscan
> > plateau.cc:memscan done
> > removing duplicate plateaus
> > MM warning: limit=1048576000B. allocating 524140060B.  limit exceeded 
> > by 58770B.
> > :
> > relabeling plateaus
> > MM warning: limit=1048576000B. allocating 524140060B.  limit exceeded 
> > by 58766B.
> > generating plateau statistics
> > MM warning: limit=1048576000B. allocating 523975900B.  limit exceeded 
> > by 58766B.
> > ----------
> > assigning directions on plateaus
> > MM warning: limit=1048576000B. allocating 524222112B.  limit exceeded 
> > by 48098B.
> > --------------
> > generating watersheds and watershed graph
> > creating windows
> > warning: using slower scan
> > r.terraflow: 3scan.h:163: void scan3line(FUN&, AMI_STREAM<T>*, 
> > AMI_STREAM<T>*, AMI_STREAM<T>*, BASETYPE, dimension_type) [with T = 
> > waterGridType, BASETYPE = waterWindowBaseType, FUN = waterWindower]: 
> > Assertion `ae == AMI_ERROR_END_OF_STREAM' failed.
> >
> >
> > and also seem not to process data above 1gb(ESRI Ascii Raster Grid) too.
> > is there anything else i missed while compilation of grass?
> >  
> > PS. i had this was my configuration, compilation and installation 
> > statement from grass61cvs.
> >  
> > ./configure --enable-shared --with-fftw --with-includes=/usr/include 
> > --with-libs=/usr/lib --with-motif --with-freetype=yes 
> > --with-freetype-includes=/usr/include/freetype2 --with-nls 
> > --with-gdal=/usr/local/bin/gdal-config --with-proj 
> > --with-proj-includes=/usr/include --with-proj-libs=/usr/lib --with-glw 
> > --with-cxx --with-mysql --with-mysql-includes=/usr/include/mysql 
> > --with-mysql-libs=/usr/lib/mysql --with-postgres 
> > --with-postgres-includes=/usr/include/pgsql 
> > --with-postgres-libs=/usr/lib --with-odbc=yes 
> > --with-odbc-includes=/usr/include --with-odbc-libs=/usr/lib 
> > --with-glw=yes --enable-largefile;
> > make;
> > make install
> >  
> > -- 
> > Best Regrads
> >
> > Abhay Menon
> 
> _______________________________________________
> grass5 mailing list
> grass5 at grass.itc.it
> http://grass.itc.it/mailman/listinfo/grass5