[GRASS-user] g_malloc on r.watershed with 30GB free ram + r.fill.dir crash

Fri Sep 11 07:12:57 PDT 2015

Here is my region:

g.region -p                                                            projection: 1 (UTM)
zone:       22
datum:      wgs84
ellipsoid:  wgs84
north:      636094.00956457
south:      233878.203125
west:       98924.52245413
east:       430413.65625
nsres:      30.00043309
ewres:      29.99901663
rows:       13407
cols:       11050
cells:      148147350

>Given my test on a small laptop, it is very likely that your actual >computational region does not match the input map.

I define the computational area with 'g.region raster=dem zoom=dem' ('dem' being my raster) before launching the script so it should match the elevation map.

>Why do you use the r.fill.dir module?

r.fill.dir is to extract depression areas as sinks, sinks being one of the outputs of my script.

Mael

De : Markus Neteler [via OSGeo.org] [mailto:ml-node+s1560n5223631h72 at n6.nabble.com]
Envoyé : jeudi 10 septembre 2015 17:07
À : MORENI Mael
Objet : Re: g_malloc on r.watershed with 30GB free ram + r.fill.dir crash

On Thu, Sep 10, 2015 at 9:00 PM, mael <[hidden email]</user/SendEmail.jtp?type=node&node=5223631&i=0>> wrote:

> I am working on a 13407 lines * 11050 columns, 565 MB, raster file (SRTM DEM,
> 30m) with a script that calls a series of hydrological modules in GRASS
> 7.0.0.
> The system I am using is a Windows server 2012 R2, 64 bit Intel Xeon 2.5 GHz
> processor, 32GB of RAM.
>
> The first module called in the script is r.watershed. When lunched on that
> data set, it generates the following error : G_malloc: impossible to
> allocate 1185910608 bytes to raster/r.watershed/ram/init_vars.c:149
> It apears to be running out of memory even though there is a lot more
> available than what it's asking for.

Very likely your computational region is wrong, i.e. too large.
Check it with

g.region -p

I just used the EU DEM 25m to make a test on my tiny ASUS laptop (4GB RAM,
Intel i3), using Fedora 22, 64bit:

> 13407  * 11050  <<--- your DEM
[1] 148147350
>
> 12880 * 16370  <<-- my DEM
[1] 210845600

(I just had this DEM ready here to play with)

GRASS 7.1.svn (eu_laea):~ > g.region -p
projection: 99 (Lambert Azimuthal Equal Area)
zone:       0
datum:      etrs89
ellipsoid:  grs80
north:      2699750
south:      2377750
west:       4126750
east:       4536000
nsres:      25
ewres:      25
rows:       12880
cols:       16370
cells:      210845600

Results:

RAM: it used some swap memory since I have browser etc open at the same time.

[neteler at oboe ~]$ free
              total        used        free      shared  buff/cache   available
Mem:        3930508     3600648       32408       93164      297452      165584
Swap:       3932156     2735164     1196992

Timing:

GRASS 7.1.svn (eu_laea):~ > time -p r.watershed elevation=eu_dem_25_TN accumulation=eu_dem_25_TN.acc basin=eu_dem_25_TN.watersheds threshold=10000
SECTION 1a (of 5): Initiating Memory.
SECTION 1b (of 5): Determining Offmap Flow.
 100%
SECTION 2: A* Search.
 100%
SECTION 3a: Accumulating Surface Flow with MFD.
 100%
SECTION 3b: Adjusting drainage directions.
 100%
SECTION 4: Watershed determination.
 100%
SECTION 5: Closing Maps.

real 1270.30
user 1048.68
sys 54.00

... 21 minutes.

> I managed to pass that first hurdle using the -m flag on r.watershed. The
> scripts now requires 7 GB of disk space; even if it's slow and I don't
> understand why it can't use the ram in the first place, r.watershed finishes
> correctly.
Given my test on a small laptop, it is very likely that your actual computational region
does not match the input map.

> However, when the script gets to the next module, r.fill.dir, GRASS crashes
> at 'Reading input elevation raster map...' with the following message:
Why do you use the r.fill.dir module?

Markus

--
http://consulting.neteler.org
http://gis.cri.fmach.it/neteler/
http://courses.neteler.org/blog

_______________________________________________
grass-user mailing list
[hidden email]</user/SendEmail.jtp?type=node&node=5223631&i=1>
http://lists.osgeo.org/mailman/listinfo/grass-user
________________________________
If you reply to this email, your message will be added to the discussion below:
http://osgeo-org.1560.x6.nabble.com/g-malloc-on-r-watershed-with-30GB-free-ram-r-fill-dir-crash-tp5223620p5223631.html
To unsubscribe from g_malloc on r.watershed with 30GB free ram + r.fill.dir crash, click here<http://osgeo-org.1560.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5223620&code=bWFlbC5tb3JlbmlAb25mLmZyfDUyMjM2MjB8LTEyMzYwODQ4NTI=>.
NAML<http://osgeo-org.1560.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>

--
View this message in context: http://osgeo-org.1560.x6.nabble.com/g-malloc-on-r-watershed-with-30GB-free-ram-r-fill-dir-crash-tp5223620p5223818.html
Sent from the Grass - Users mailing list archive at Nabble.com.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-user/attachments/20150911/28ad6bb2/attachment.html>