[GRASS-dev] r.hydrodem addon fails on very big rasters (150, 000 x 120, 000 pixels)

[Blumentrath, Stefan]

Hei Markus and Markus

Thanks for your reply and the information that r.hydrodem would not improve the results of r.watershed.

The manual of r.hydrodem btw., said different. There it is written e.g. that "If the conditioned elevation map is going to be used as input elevation for r.watershed, only small sinks should be removed (...)" or "the removal of one-cell extrema could already be sufficient to improve r.watershed results in difficult terrain, particularly nearly flat areas." I guess the manual applies / applied for an older version of r.watershed?

Anyway, I am impressed of the performance of the hydrological models in GRASS and I shall add a note about the Performance of GRASS to the wiki.

Cheers
Stefan



-----Original Message-----
From: Markus Metz [mailto:markus.metz.giswork at gmail.com] 
Sent: 10. februar 2014 09:03
To: Markus Neteler
Cc: Blumentrath, Stefan; GRASS developers list
Subject: Re: [GRASS-dev] r.hydrodem addon fails on very big rasters (150, 000 x 120, 000 pixels)

On Thu, Feb 6, 2014 at 5:57 PM, Markus Neteler <neteler at osgeo.org> wrote:
> On Thu, Feb 6, 2014 at 1:14 PM, Blumentrath, Stefan 
> <Stefan.Blumentrath at nina.no> wrote:
>> Dear all,
>>
>> I am testing, trying, learning GRASS 7`s hydrology moduls at the 
>> moment. And I am very, very satisfied with the available tools. 
>> r.watershed managed to process a really big elevation model (150,000 
>> x 120,000 pixels), which is amazing.
>
> Wow! Please consider to add a note to
> http://grasswiki.osgeo.org/wiki/GRASS_GIS_Performance#Large_raster_dat
> a_processing
>
> (or in a blog entry or the like)
>
>> However, when I use r.hydrodem addon on the same big grid with 
>> 150,000 x
>> 120,000 pixels (with the memory option) I get the following warning 
>> (and finally error) message:
>>
>> r.hydrodem input=dem_10m memory=10000 output=dem_10m _hydro
> ...
>> WARNING: segment_format(fd,1,-221837149,1,262144,16): illegal 
>> value(s)
>
> This is an integer overflow of ncols etc:
>
> [neteler at pgis_north r.hydrodem]$ grep nrows seg.c seg_open(SSEG *sseg, 
> int nrows, int ncols, int row_in_seg, int col_in_seg,
>
> In r.watershed it is defined differently:
>
> [neteler at pgis_north r.watershed]$ grep nrows seg/sseg_open.c 
> seg_open(SSEG * sseg, GW_LARGE_INT nrows, GW_LARGE_INT ncols, int 
> row_in_seg, int col_in_seg,
>     if (0 > (errflag = segment_format(fd, nrows, ncols,
>
> I suppose that this is an "easy" fix given the experience of r.watershed.
> To not interfere, I'll leave the change to the module author.

The fix would be relatively easy, I will do it as soon as time permits...

Note that r.hydrodem and r.watershed use the same mechanism, therefore r.hydrodem can not improve a DEM for r.watershed. The r.hydrodem module is meant to perform hydrological conditioning (sink removal) of a DEM when the DEM is used with other modules or other software.
However, to my knowledge no other software exists that is able to perform surface flow accumulation on such a large raster. Two examples for sophisticated hydrological modelling packages are TauDEM and WhiteBox (formerly TAS). The last time I checked, both were not able to process DEMs with more than 2 billion grid cells. Even r.terraflow, designed for massive grids, can not process a 150,000 x 120,000 DEM.

Markus M