[GRASS-dev] i.colors.enhance: G_calloc() error in r.quantile with large maps

[Markus Metz]

On Thu, Apr 27, 2017 at 1:31 PM, Markus Neteler <neteler at osgeo.org> wrote:
>
> Hi,
>
> I am trying to improve the color balance of a large Sentinel-2 scene.
> The job fail like this:
>
> i.colors.enhance ....
> creating color enhanced composite B8A B11 B04....
> Processing...
> ERROR: G_calloc: unable to allocate 18446744071588213116 * 8 bytes of
> memory at raster/r.quantile/main.c:355

The number 18446744071588213116 is unrealistically large because it should
not be larger than rows * cols

[...]
>
> The Sentinel-2 scene is 169410 * 88264 rows/cols large which does not
> explain this problem.

I would rather expect a call to G_calloc requesting a maximum of
14952804240 * 8 bytes, most likely less.

The reason for the wrong number is integer overflow because num_values is
of type int. Please try trunk r70982.

Markus M

>
> I recall that i.colors.enhance uses the Python "multiprocessing" backend.
> Assuming 4 parallel threads (not sure how many the script will take?
> not documented... an nproc parameter would be better) I estimate this
> RAM usage:
>
> > 169410*88264 *4
> [1] 478489735680 bytes
> > 169410*88264 *4 /1014/1014/1014
> [1] 57.36788 GB
>
> which exceeds the existing 32GB RAM + 8 GB swap.
>
> Questions:
>
> a) is my assumption right? Still the 18446744071588213116 are weird.
> The scripts runs fine for 10th of other Sentinel-2 scenes.
>
> b) how to control the number of parallel threads in i.colors.enhance
> expect for the very limiting -s flag?
>
> thanks,
> Markus
> _______________________________________________
> grass-dev mailing list
> grass-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/grass-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-dev/attachments/20170428/5ba758c5/attachment.html>