[GRASS-dev] i.colors.enhance: G_calloc() error in r.quantile with large maps

[Markus Neteler]

Hi,

I am trying to improve the color balance of a large Sentinel-2 scene.
The job fail like this:

i.colors.enhance ....
creating color enhanced composite B8A B11 B04....
Processing...
ERROR: G_calloc: unable to allocate 18446744071588213116 * 8 bytes of
memory at raster/r.quantile/main.c:355
Process Process-2:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in
_bootstrap
    self.run()
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/mundialis/software/grass72_svn/dist.x86_64-pc-linux-gnu/scripts/i.colors.enhance",
line 105, in get_percentile_mp
    result = get_percentile(map, percentiles)
  File "/home/mundialis/software/grass72_svn/dist.x86_64-pc-linux-gnu/scripts/i.colors.enhance",
line 89, in get_percentile
    percentiles=values, quiet=True)
  File "/home/mundialis/software/grass72_svn/dist.x86_64-pc-linux-gnu/etc/python/grass/script/core.py",
line 461, in read_command
    return handle_errors(returncode, stdout, args, kwargs)
  File "/home/mundialis/software/grass72_svn/dist.x86_64-pc-linux-gnu/etc/python/grass/script/core.py",
line 329, in handle_errors
    returncode=returncode)
CalledModuleError: Module run None ['r.quantile', '--q',
'input=B11_255', 'percentiles=2,98'] ended with error
Process ended with non-zero return code 1. See errors in the (error) output.

If I'm no wrong then this is 147.57 exabytes needed :-)

The Sentinel-2 scene is 169410 * 88264 rows/cols large which does not
explain this problem.

I recall that i.colors.enhance uses the Python "multiprocessing" backend.
Assuming 4 parallel threads (not sure how many the script will take?
not documented... an nproc parameter would be better) I estimate this
RAM usage:

> 169410*88264 *4
[1] 478489735680 bytes
> 169410*88264 *4 /1014/1014/1014
[1] 57.36788 GB

which exceeds the existing 32GB RAM + 8 GB swap.

Questions:

a) is my assumption right? Still the 18446744071588213116 are weird.
The scripts runs fine for 10th of other Sentinel-2 scenes.

b) how to control the number of parallel threads in i.colors.enhance
expect for the very limiting -s flag?

thanks,
Markus