[GRASS5] r.terraflow failure

[Andrew Danner]

My initial hunch is that this may be an fseek error. The function
scan3line where the assertion fails is opening three file descriptors or
"substreams" on a single file and calling fseek to move to the beginning
of a particular row.  However, fseek takes a 32-bit long offset which
overflows at 2GB. There is an fseeko in most stdio.h implementations
that takes a 64 bit off_t offset and works for larger files. 

Unfortunately I do not have time in the near future to investigate this
further. My initial suggestion is to change fseek to fseeko in line 397
of 

raster/r.terraflow/IOStream/include/ami_stream.h

rebuild the terraflow module and see if that works. Certainly, calling
fseek instead of fseeko is a problem, but I'm not sure if that is the
only problem. Terraflow is using 64-bit types that handle offsets larger
than 2GB in a lot of other places, but it may be a bit buggy in spots. 

If you try the fix, let me know what happens and perhaps I can help out
some more if things are still not working.  


-Andy

On Mon, 2006-02-20 at 08:24 -0800, endeitz wrote:
> Just a followup to my previous message.  Instead of using the snapshot binary
> I compiled GRASS myself.  This apparently fixed the 2GB issue.  However,
> r.terraflow is still crashing on what appears to be a seek error of some
> kind on a file of approximately 7GB (again, is it a file size limit?).  I
> was wondering if anyone could give me a clue.  The error is near the end of
> the listing below.  I also show ulimit results below, if that is relevant.
> 
> -----------------------------
> 
> GRASS 6.1.cvs (lcra):~/grassdata > ulimit -a
> core file size        (blocks, -c) 1000000
> data seg size         (kbytes, -d) unlimited
> file size             (blocks, -f) unlimited
> max locked memory     (kbytes, -l) unlimited
> max memory size       (kbytes, -m) unlimited
> open files                    (-n) 1024
> pipe size          (512 bytes, -p) 8
> stack size            (kbytes, -s) 8192
> cpu time             (seconds, -t) unlimited
> max user processes            (-u) 3071
> virtual memory        (kbytes, -v) unlimited
> 
> ---------------------------------------------------------------
> 
> r.terraflow December 2003
> region size is 13458 x 13438
> cell elev15000 header compatible with region header
> elevation stored as FLOAT (4B)
> STREAM temporary files in /var/tmp  (THESE INTERMEDIATE STREAMS WILL NOT BE
> DELETED IN CASE OF ABNORMAL TERM
> file stats.out exists - renaming.
> reading data from elev15000 to stream /var/tmp/STREAM_djueQG:    
> 
> total elements=180848604, nodata elements=0
> largest temporary files:
>                  FILL: 9.43G (10127521824) [180848604 elements, 56B each]
>                  FLOW: 13.47G (14467888320) [180848604 elements, 80B each]
> Will need at least 26.95G (28935776640) space available in /var/tmp
> MM warning: limit=314572800B. allocating 157138456B.  limit exceeded by
> 17722B.
> MM warning: limit=314572800B. allocating 157138456B.  limit exceeded by
> 17753B.
> MM warning: limit=314572800B. allocating 157138456B.  limit exceeded by
> 17784B.
> MM warning: limit=314572800B. allocating 157138456B.  limit exceeded by
> 17815B.
> MM warning: limit=314572800B. allocating 157138456B.  limit exceeded by
> 17846B.
> MM warning: limit=314572800B. allocating 157138456B.  limit exceeded by
> 17718B.
> MM warning: limit=314572800B. allocating 157138456B.  limit exceeded by
> 17749B.
> MM warning: limit=314572800B. allocating 157138456B.  limit exceeded by
> 17780B.
> MM warning: limit=314572800B. allocating 157138456B.  limit exceeded by
> 17811B.
> MM warning: limit=314572800B. allocating 157015336B.  limit exceeded by
> 17718B.
> MM warning: limit=314572800B. allocating 157015336B.  limit exceeded by
> 17749B.
> MM warning: limit=314572800B. allocating 157015336B.  limit exceeded by
> 17749B.
> MM warning: limit=314572800B. allocating 157015336B.  limit exceeded by
> 17780B.
> MM warning: limit=314572800B. allocating 157015336B.  limit exceeded by
> 17811B.
> MM warning: limit=314572800B. allocating 157220512B.  limit exceeded by
> 14518B.
> MM warning: limit=314572800B. allocating 157220512B.  limit exceeded by
> 14549B.
> MM warning: limit=314572800B. allocating 157220512B.  limit exceeded by
> 14580B.
> MM warning: limit=314572800B. allocating 157220512B.  limit exceeded by
> 14611B.
> MM warning: limit=314572800B. allocating 157220512B.  limit exceeded by
> 14642B.
> MM warning: limit=314572800B. allocating 157220512B.  limit exceeded by
> 14673B.
> r.terraflow: 3scan.h:163: void scan3line(FUN&, AMI_STREAM<T>*,
> AMI_STREAM<T>*, AMI_STREAM<T>*, BASETYPE, short int) [with T =
> waterGridType, BASETYPE = waterWindowBaseType, FUN = waterWindower]:
> Assertion `ae == AMI_ERROR_END_OF_STREAM' failed.
> 
> --------------------------------------------------
> 
> GRASS 6.1.cvs (lcra):~/grassdata > ls -lh /var/tmp
> total 11G
> -rw-------    1 nedeeds  nedeeds      690M Feb 18 16:29 STREAM_a5xEvs
> -rw-------    1 nedeeds  nedeeds      7.0G Feb 18 17:14 STREAM_LjcOQz
> -rw-------    1 nedeeds  nedeeds      2.7G Feb 18 16:59 STREAM_TtzchB
> GRASS 6.1.cvs (lcra):~/grassdata > 
> 
> 
> --
> View this message in context: http://www.nabble.com/Re%3A-GRASSLIST%3A7448-Re%3A-r.terraflow-failured-t130887.html#a3033415
> Sent from the Grass5 - Dev forum at Nabble.com.
> 
> _______________________________________________
> grass5 mailing list
> grass5 at grass.itc.it
> http://grass.itc.it/mailman/listinfo/grass5