[GRASS-dev] i.colors.enhance: G_calloc() error in r.quantile with large maps

[Markus Neteler]

....
.On Sat, Apr 29, 2017 at 11:16 PM, Markus Neteler <neteler at osgeo.org> wrote:
> On Fri, Apr 28, 2017 at 5:05 PM, Markus Metz
...
> rows:       88265
> cols:       169411
> cells:      14953061915
...
> I have locally backported it, works too (I'd suggest to backport).

Mhh, there is still another issue:

GRASS 7.2.1svn (grass):~ > i.colors.enhance red=B04_255 green=B11_255
blue=B8A_255
Processing...
ERROR: G_calloc: unable to allocate 2181527539 * 8 bytes of memory at
       raster/r.quantile/main.c:356
Process Process-1:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in
_bootstrap
    self.run()
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/mundialis/software/grass72_svn/dist.x86_64-pc-linux-gnu/scripts/i.colors.enhance",
line 105, in get_percentile_mp
    result = get_percentile(map, percentiles)
  File "/home/mundialis/software/grass72_svn/dist.x86_64-pc-linux-gnu/scripts/i.colors.enhance",
line 89, in get_percentile
    percentiles=values, quiet=True)
  File "/home/mundialis/software/grass72_svn/dist.x86_64-pc-linux-gnu/etc/python/grass/script/core.py",
line 461, in read_command
    return handle_errors(returncode, stdout, args, kwargs)
  File "/home/mundialis/software/grass72_svn/dist.x86_64-pc-linux-gnu/etc/python/grass/script/core.py",
line 329, in handle_errors
    returncode=returncode)
CalledModuleError: Module run None ['r.quantile', '--q',
'input=B04_255', 'percentiles=2,98'] ended with error
Process ended with non-zero return code 1. See errors in the (error) output.
Traceback (most recent call last):
  File "/home/mundialis/software/grass72_svn/dist.x86_64-pc-linux-gnu/scripts/i.colors.enhance",
line 223, in <module>
    main()
  File "/home/mundialis/software/grass72_svn/dist.x86_64-pc-linux-gnu/scripts/i.colors.enhance",
line 163, in main
    (v0, v1) = input_pipe.recv()
Process Process-3:
...

Using a new extra swapfile I have now 32GB RAM and 18GB swap available.


Next test: tracking memory consumption (using a loop):

start:
2017-04-30 11:32:10,63907664
2017-04-30 11:32:11,63906728
2017-04-30 11:32:12,63895512
...
GRASS 7.2.1svn (grass):~ > i.colors.enhance red=B04_255 green=B11_255
blue=B8A_255
Computing histogram
...
2017-04-30 11:34:44,59963812
...
2017-04-30 11:36:16,56004076
...
Binning data
...
2017-04-30 11:39:54,332164
...
2017-04-30 11:40:39,264508
...
2017-04-30 11:41:57,250920
...
2017-04-30 11:42:09,480848
...
Process Process-3:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in
_bootstrap
    self.run()
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/mundialis/software/grass72_svn/dist.x86_64-pc-linux-gnu/scripts/i.colors.enhance",
line 105, in get_percentile_mp
[.... as above...]

in the other terminal in which I had the memory check running, I got:

[mundialis at fedora-calc ~]$ for i in `seq 1 10000` ; do
memory_RAM_usage_timestamp.sh ; sleep 1 ; done >
i_colors_enhance_RAM.csv
-bash: fork: Cannot allocate memory
Connection to xxxxx.mundialis.de closed.

ops. It seems that the memory footprint becomes big at some point in
r.quantile. Could there be a leak somewhere? Or is it due to the 14,95
gigapixels?

I'll add another swap file and try again.

markusN