[GRASS-dev] Improving the G_calloc: unable to allocate xx bytes of memory message?

[Markus Neteler]

On Mon, Oct 27, 2014 at 4:33 PM, Glynn Clements
<glynn at gclements.plus.com> wrote:
> Markus Neteler wrote:
>
>> Ok, I tried that patch but I don't manage to trigger it.
>>
>> GRASS 7.1.svn (nc_spm_08_grass7):~ > g.region rast=elevation -p res=0.01
>
>> rows:       1350000
>> cols:       1500000
>> cells:      2025000000000
>
> That's 1.8 * 2^40 cells; at 24 bytes/cell, it would be 44 TiB.

Yes, I tried to do some extreme computing. But GRASS/Linux resisted
easily as you point out below.

>> GRASS 7.1.svn (nc_spm_08_grass7):~ > r.cost roadsmajor output=bla
>> start_coordinates=637500.000000,221750.000000
>> WARNING: segment zero_fill(): Unable to write (No such file or directory)
>
> So it's not trying to allocate 44 TiB of RAM, it's trying to create a
> 44 TiB segment file, which fails for other reasons.
>
> You'll need to find something which actually allocates RAM rather than
> a temporary file. Try "r.proj memory=..."; this allocates the cache
> memory before creating any temporary file.
>
>> WARNING: Could not write segment file
>> ERROR: Can not create temporary file

OK, no success with r.proj (to crash it) but this one triggered the new message:


GRASS 7.1.svn (nc_spm_08_grass7):~ > g.region rast=roadsmajor res=0.1 -p
projection: 99 (Lambert Conformal Conic)
zone:       0
datum:      nad83
ellipsoid:  a=6378137 es=0.006694380022900787
north:      228500
south:      215000
west:       630000
east:       645000
nsres:      0.1
ewres:      0.1
rows:       135000
cols:       150000
cells:      20250000000

GRASS 7.1.svn (nc_spm_08_grass7):~ > v.to.rast roadsmajor out=test
use=attr attrcol=PROPYEAR memory=3000000 --o
WARNING: No areas selected from vector map <roadsmajor>
Current region rows: 135000, cols: 150000
ERROR: G_calloc: unable to allocate 18446744072484715136 * 4 bytes of
       memory at vector/v.to.rast/raster.c:84

... only, it does not return to command line (have to use CTRL-C).

>> Yet a bit unhelpful :-) It comes from lib/segment/format.c but I
>> didn't manage to improve that.
>
> Specifically:
>
>         if (write(fd, buf, n) != n) {
>             G_warning("segment zero_fill(): Unable to write (%s)", strerror(errno));
>
> ENOENT ("No such file or directory") isn't a valid error code for
> write(). But errno is only set if write() returns -1, so it's likely
> that write() returned a short count rather than failing and the errno
> value is left over from a previous system call (sucessful calls don't
> reset errno to zero). In all probability, the next call to write()
> would have failed due to ENOSPC, EDQUOT, or EFBIG (or raised SIGXFSZ
> if RLIMIT_FSIZE is being exceeded).

I see and now darkly remember something about discussion in this list
about clearing the previous error state.

Markus