[GRASS-user] i.segment.stats memory usage error

[James Duffy]

On 7 November 2016 at 14:13, Moritz Lennert <mlennert at club.worldonline.be>
wrote:

> Hi James,
>
> On 04/11/16 22:08, James Duffy wrote:
>
>> Sorry to post again on this long thread...
>>
>
> No worries :-)
>
>
>> I have just managed to get my hands on a 64bit osgeolive install with
>> 12GB RAM available and the afformentioned function worked. Just to
>> reiterate this was i.segment.stats on gp_seg_optimum_clump (which was
>> the output of i.segment, run through r.clump). I haven't created the
>> vectormap, but it seems that the csv was created correctly.
>>
>> I don't know if it was the 64bit or the bit of extra RAM that made the
>> difference here.
>>
>
> I would guess the extra RAM, but am no expert on these questions.
>
> I'm working on a revised version of i.segment.stats to use
> r.stats.quantile. It would be great if you could test that once its done.
>

Yes, happy to test it out for you. Please let me know when to reinstall the
extension.


>
> Moritz
>
>
>> James
>>
>> On 4 November 2016 at 15:34, Markus Metz <markus.metz.giswork at gmail.com
>> <mailto:markus.metz.giswork at gmail.com>> wrote:
>>
>>     On Fri, Nov 4, 2016 at 3:51 PM, Moritz Lennert
>>     <mlennert at club.worldonline.be <mailto:mlennert at club.worldonline.be>>
>>     wrote:
>>     > On 04/11/16 14:29, Markus Metz wrote:
>>     >>
>>     >> On Wed, Nov 2, 2016 at 5:15 PM, Markus Metz
>>     >> <markus.metz.giswork at gmail.com
>>     <mailto:markus.metz.giswork at gmail.com>> wrote:
>>     >>>
>>     >>> On Wed, Nov 2, 2016 at 3:47 PM, Moritz Lennert
>>     >>> <mlennert at club.worldonline.be
>>     <mailto:mlennert at club.worldonline.be>> wrote:
>>     >>>>
>>     >>>> On 02/11/16 11:05, James Duffy wrote:
>>     >>>>>
>>     >>>>>
>>     >>>>> I can't register with osgeo to submit a bug as no-one has
>>     replied with
>>     >>>>> a
>>     >>>>> 'mantra' for me to do so... what a convoluted bug reporting
>>     system!
>>     >>>>>
>>     >>>>> And in your opinion Moritz, does it look like my workflow will
>>     not be
>>     >>>>> possible unless I find a 64bit machine with a decent amount of
>>     RAM?
>>     >>>>
>>     >>>>
>>     >>>>
>>     >>>> I'm not sure about the necessity of 64bit, but more RAM probably.
>>     >>>>
>>     >>>> IIUC, all the stats of all the zones are put into memory during
>>     the run
>>     >>>> of
>>     >>>> the module. Maybe a file-based store of the information might be
>>     >>>> possible
>>     >>>> which would avoid this memory usage, but I'm not sure.
>>     >>>>
>>     >>>> @MarkusM: could you have a look at this ? See
>>     >>>>
>>     https://lists.osgeo.org/pipermail/grass-user/2016-October/075493.html
>> <https://lists.osgeo.org/pipermail/grass-user/2016-October/075493.html>
>>
>>     >>>> for
>>     >>>> the beginning of the thread. In short: James seems to be
>>     hitting memory
>>     >>>> limits of r.univar and I'm wondering out loud whether there is
>>     something
>>     >>>> that could be done about this...
>>     >>>
>>     >>>
>>     >>> As Anna wrote, the -e flag increases memory consumption of
>> r.univar
>>     >>> considerably. A file based approach for extended stats of
>>     r.univar is
>>     >>> difficult because the number of values per segment is initially
>>     >>> unknown to r.univar, thus the amount of disk space needed for
>>     extended
>>     >>> stats for each segment is unknown.
>>     >>>
>>     >>> I would rather use r.univar for standard stats and
>> r.stats.quantile
>>     >>> for extended stats.
>>     >>
>>     >>
>>     >> There were some bugs in r.stats.quantile, fixed in r69770-2 in all
>> G7
>>     >> branches.
>>     >
>>     >
>>     > Great, thanks !
>>     >
>>     >>
>>     >> About the limit to MAX_CATS categories in the base map (ticket
>>     #3198),
>>     >> there is not really a reason for it. I guess it is meant to protect
>>     >> against out-of-memory errors.
>>     >
>>     >
>>     > Does this mean we can just take out the lines referencing it ?
>>
>>     Maybe replace the error with a warning. I tested with a base map with
>>     1.4 million categories (output of i.segment) and it works, but I
>>     needed to reduce the number of bins from 1000 to 100 to avoid OOM.
>>     With a low number of bins, the memory consumption of r.stats.quantile
>>     is much less than what r.univar requires, but a low number of bins can
>>     in some cases (histogram of the cell values with a peak) lead to
>>     higher memory consumption.
>>
>>     r.univar does memory reallocation on demand which means that for a
>>     short time, double the amount of memory is needed. r.univar2 makes a
>>     guess about how much memory is required and then allocates all at
>>     once. Still, the binning approach of r.(stats.)quantile usually needs
>>     less memory.
>>
>>     Markus M
>>
>>
>
>


-- 
*James Duffy*
PhD Researcher
Environment and Sustainability Institute
Penryn Campus
University of Exeter
Penryn
Cornwall
TR10 9FE
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-user/attachments/20161107/8296af37/attachment-0001.html>