[GRASS-dev] [GRASS GIS] #2676: r.neighbors on large rasters
GRASS GIS
trac at osgeo.org
Fri May 15 06:14:17 PDT 2015
#2676: r.neighbors on large rasters
--------------------------+---------------------------------------
Reporter: dnewcomb | Owner: grass-dev@…
Type: enhancement | Status: reopened
Priority: normal | Milestone: 7.0.1
Component: Default | Version: svn-trunk
Resolution: | Keywords: r.neighbors large rasters
CPU: x86-64 | Platform: Linux
--------------------------+---------------------------------------
Comment (by dnewcomb):
Replying to [comment:4 glynn]:
> Replying to [comment:2 dnewcomb]:
>
> > > What are the actual dimensions (rows x columns) of the current
region? (The dimensions of the input map aren't directly relevant).
>
> Sorry, that's incorrect. If you don't use the -a flag, r.neighbors sets
the region to match the input map.
>
> In any case, the cause is the use of G_alloca() for allocating temporary
row buffers within the raster library. This is a macro which expands to
alloca(), which allocates memory on the stack.
>
> While this is vastly more efficient than malloc() etc (alloca() may
compile to just 2 CPU instructions), it doesn't report allocation failure;
the next instruction to push something onto the stack will result in a
segfault.
>
> 1875705 columns at 8 bytes per cell corresponds to 15 MB for a row. On
my system, the default maximum stack size (ulimit -s) is 8192 KiB (8 MiB).
Changing this to 50 MiB avoids the segfault (although I have neither the
patience nor the free disk space to see whether it runs to completion).
>
> While I doubt that this will be a genuine issue for many people, it can
result in the upper limit on map dimensions being smaller than it could
be.
>
> Consequently, we may want to think about whether the use of alloca()
should be optional. Currently, it's used on any system which is believed
to provide it (those which lack it use malloc/free). See the top of
include/defs/gis.h for the details.
>
> If most users can live with the existing behaviour (an 8 MiB default
stack allows for just short of a million columns with DCELL data) and most
of the rest can live with explicitly increasing the stack size, it may
suffice to just add e.g. "#ifndef DONT_USE_ALLOCA" to allow the remainder
to override the behaviour at compile time.
Just to give an idea of the use case, I have done a land cover change
ndvi analysis based off of 4 band 1m resolution digital aerial photography
for the state of NC that I'm averaging to a coarser resolution. I can
see down the road a couple of years, as image data gets denser, the
possibility of more folks running into the limits you describe.
Of course, one could always chop the data set into overlapping blocks,
process, and patch together later.
--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2676#comment:5>
GRASS GIS <http://grass.osgeo.org>
More information about the grass-dev
mailing list