[GRASS-dev] i.colors.enhance: G_calloc() error in r.quantile with large maps

[Anna Petrášová]

On Thu, Apr 27, 2017 at 7:31 AM, Markus Neteler <neteler at osgeo.org> wrote:
> Hi,
>
> I am trying to improve the color balance of a large Sentinel-2 scene.
> The job fail like this:
>
> i.colors.enhance ....
> creating color enhanced composite B8A B11 B04....
> Processing...
> ERROR: G_calloc: unable to allocate 18446744071588213116 * 8 bytes of
> memory at raster/r.quantile/main.c:355
> Process Process-2:
> Traceback (most recent call last):
>   File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in
> _bootstrap
>     self.run()
>   File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
>     self._target(*self._args, **self._kwargs)
>   File "/home/mundialis/software/grass72_svn/dist.x86_64-pc-linux-gnu/scripts/i.colors.enhance",
> line 105, in get_percentile_mp
>     result = get_percentile(map, percentiles)
>   File "/home/mundialis/software/grass72_svn/dist.x86_64-pc-linux-gnu/scripts/i.colors.enhance",
> line 89, in get_percentile
>     percentiles=values, quiet=True)
>   File "/home/mundialis/software/grass72_svn/dist.x86_64-pc-linux-gnu/etc/python/grass/script/core.py",
> line 461, in read_command
>     return handle_errors(returncode, stdout, args, kwargs)
>   File "/home/mundialis/software/grass72_svn/dist.x86_64-pc-linux-gnu/etc/python/grass/script/core.py",
> line 329, in handle_errors
>     returncode=returncode)
> CalledModuleError: Module run None ['r.quantile', '--q',
> 'input=B11_255', 'percentiles=2,98'] ended with error
> Process ended with non-zero return code 1. See errors in the (error) output.
>
> If I'm no wrong then this is 147.57 exabytes needed :-)
>
> The Sentinel-2 scene is 169410 * 88264 rows/cols large which does not
> explain this problem.
>
> I recall that i.colors.enhance uses the Python "multiprocessing" backend.
> Assuming 4 parallel threads (not sure how many the script will take?
> not documented... an nproc parameter would be better) I estimate this
> RAM usage:

looking briefly at the source code, it's using 3 (red, green, blue)
>
>> 169410*88264 *4
> [1] 478489735680 bytes
>> 169410*88264 *4 /1014/1014/1014
> [1] 57.36788 GB
>
> which exceeds the existing 32GB RAM + 8 GB swap.
>
> Questions:
>
> a) is my assumption right? Still the 18446744071588213116 are weird.
> The scripts runs fine for 10th of other Sentinel-2 scenes.
>
> b) how to control the number of parallel threads in i.colors.enhance
> expect for the very limiting -s flag?

someone would need to rewrite it,  using Pool.map_async it should be
relatively easy

Anna

>
> thanks,
> Markus
> _______________________________________________
> grass-dev mailing list
> grass-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/grass-dev