[GRASS-dev] r.hydrodem addon fails on very big rasters (150, 000 x 120, 000 pixels)

[Markus Metz]

On Thu, Feb 6, 2014 at 5:57 PM, Markus Neteler <neteler at osgeo.org> wrote:
> On Thu, Feb 6, 2014 at 1:14 PM, Blumentrath, Stefan
> <Stefan.Blumentrath at nina.no> wrote:
>> Dear all,
>>
>> I am testing, trying, learning GRASS 7`s hydrology moduls at the moment. And
>> I am very, very satisfied with the available tools. r.watershed managed to
>> process a really big elevation model (150,000 x 120,000 pixels), which is
>> amazing.
>
> Wow! Please consider to add a note to
> http://grasswiki.osgeo.org/wiki/GRASS_GIS_Performance#Large_raster_data_processing
>
> (or in a blog entry or the like)
>
>> However, when I use r.hydrodem addon on the same big grid with 150,000 x
>> 120,000 pixels (with the memory option) I get the following warning (and
>> finally error) message:
>>
>> r.hydrodem input=dem_10m memory=10000 output=dem_10m _hydro
> ...
>> WARNING: segment_format(fd,1,-221837149,1,262144,16): illegal value(s)
>
> This is an integer overflow of ncols etc:
>
> [neteler at pgis_north r.hydrodem]$ grep nrows seg.c
> seg_open(SSEG *sseg, int nrows, int ncols, int row_in_seg, int col_in_seg,
>
> In r.watershed it is defined differently:
>
> [neteler at pgis_north r.watershed]$ grep nrows seg/sseg_open.c
> seg_open(SSEG * sseg, GW_LARGE_INT nrows, GW_LARGE_INT ncols, int
> row_in_seg, int col_in_seg,
>     if (0 > (errflag = segment_format(fd, nrows, ncols,
>
> I suppose that this is an "easy" fix given the experience of r.watershed.
> To not interfere, I'll leave the change to the module author.

The fix would be relatively easy, I will do it as soon as time permits...

Note that r.hydrodem and r.watershed use the same mechanism, therefore
r.hydrodem can not improve a DEM for r.watershed. The r.hydrodem
module is meant to perform hydrological conditioning (sink removal) of
a DEM when the DEM is used with other modules or other software.
However, to my knowledge no other software exists that is able to
perform surface flow accumulation on such a large raster. Two examples
for sophisticated hydrological modelling packages are TauDEM and
WhiteBox (formerly TAS). The last time I checked, both were not able
to process DEMs with more than 2 billion grid cells. Even r.terraflow,
designed for massive grids, can not process a 150,000 x 120,000 DEM.

Markus M