<div dir="ltr">On 7 November 2016 at 14:13, Moritz Lennert <span dir="ltr"><<a href="mailto:mlennert@club.worldonline.be" target="_blank">mlennert@club.worldonline.be</a>></span> wrote:<br><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi James,<span class=""><br>
<br>
On 04/11/16 22:08, James Duffy wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Sorry to post again on this long thread...<br>
</blockquote>
<br></span>
No worries :-)<span class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
I have just managed to get my hands on a 64bit osgeolive install with<br>
12GB RAM available and the afformentioned function worked. Just to<br>
reiterate this was i.segment.stats on gp_seg_optimum_clump (which was<br>
the output of i.segment, run through r.clump). I haven't created the<br>
vectormap, but it seems that the csv was created correctly.<br>
<br>
I don't know if it was the 64bit or the bit of extra RAM that made the<br>
difference here.<br>
</blockquote>
<br></span>
I would guess the extra RAM, but am no expert on these questions.<br>
<br>
I'm working on a revised version of i.segment.stats to use r.stats.quantile. It would be great if you could test that once its done.<br></blockquote><div><br></div><div>Yes, happy to test it out for you. Please let me know when to reinstall the extension.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Moritz<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
<br>
James<br>
<br>
On 4 November 2016 at 15:34, Markus Metz <<a href="mailto:markus.metz.giswork@gmail.com" target="_blank">markus.metz.giswork@gmail.com</a><br></span><span class="">
<mailto:<a href="mailto:markus.metz.giswork@gmail.com" target="_blank">markus.metz.giswork@gm<wbr>ail.com</a>>> wrote:<br>
<br>
On Fri, Nov 4, 2016 at 3:51 PM, Moritz Lennert<br></span>
<<a href="mailto:mlennert@club.worldonline.be" target="_blank">mlennert@club.worldonline.be</a> <mailto:<a href="mailto:mlennert@club.worldonline.be" target="_blank">mlennert@club.worldonl<wbr>ine.be</a>>><span class=""><br>
wrote:<br>
> On 04/11/16 14:29, Markus Metz wrote:<br>
>><br>
>> On Wed, Nov 2, 2016 at 5:15 PM, Markus Metz<br>
>> <<a href="mailto:markus.metz.giswork@gmail.com" target="_blank">markus.metz.giswork@gmail.com</a><br></span><span class="">
<mailto:<a href="mailto:markus.metz.giswork@gmail.com" target="_blank">markus.metz.giswork@gm<wbr>ail.com</a>>> wrote:<br>
>>><br>
>>> On Wed, Nov 2, 2016 at 3:47 PM, Moritz Lennert<br>
>>> <<a href="mailto:mlennert@club.worldonline.be" target="_blank">mlennert@club.worldonline.be</a><br></span><span class="">
<mailto:<a href="mailto:mlennert@club.worldonline.be" target="_blank">mlennert@club.worldonl<wbr>ine.be</a>>> wrote:<br>
>>>><br>
>>>> On 02/11/16 11:05, James Duffy wrote:<br>
>>>>><br>
>>>>><br>
>>>>> I can't register with osgeo to submit a bug as no-one has<br>
replied with<br>
>>>>> a<br>
>>>>> 'mantra' for me to do so... what a convoluted bug reporting<br>
system!<br>
>>>>><br>
>>>>> And in your opinion Moritz, does it look like my workflow will<br>
not be<br>
>>>>> possible unless I find a 64bit machine with a decent amount of<br>
RAM?<br>
>>>><br>
>>>><br>
>>>><br>
>>>> I'm not sure about the necessity of 64bit, but more RAM probably.<br>
>>>><br>
>>>> IIUC, all the stats of all the zones are put into memory during<br>
the run<br>
>>>> of<br>
>>>> the module. Maybe a file-based store of the information might be<br>
>>>> possible<br>
>>>> which would avoid this memory usage, but I'm not sure.<br>
>>>><br>
>>>> @MarkusM: could you have a look at this ? See<br>
>>>><br></span>
<a href="https://lists.osgeo.org/pipermail/grass-user/2016-October/075493.html" rel="noreferrer" target="_blank">https://lists.osgeo.org/piperm<wbr>ail/grass-user/2016-October/<wbr>075493.html</a> <<a href="https://lists.osgeo.org/pipermail/grass-user/2016-October/075493.html" rel="noreferrer" target="_blank">https://lists.osgeo.org/piper<wbr>mail/grass-user/2016-October/<wbr>075493.html</a>><div><div class="h5"><br>
>>>> for<br>
>>>> the beginning of the thread. In short: James seems to be<br>
hitting memory<br>
>>>> limits of r.univar and I'm wondering out loud whether there is<br>
something<br>
>>>> that could be done about this...<br>
>>><br>
>>><br>
>>> As Anna wrote, the -e flag increases memory consumption of r.univar<br>
>>> considerably. A file based approach for extended stats of<br>
r.univar is<br>
>>> difficult because the number of values per segment is initially<br>
>>> unknown to r.univar, thus the amount of disk space needed for<br>
extended<br>
>>> stats for each segment is unknown.<br>
>>><br>
>>> I would rather use r.univar for standard stats and r.stats.quantile<br>
>>> for extended stats.<br>
>><br>
>><br>
>> There were some bugs in r.stats.quantile, fixed in r69770-2 in all G7<br>
>> branches.<br>
><br>
><br>
> Great, thanks !<br>
><br>
>><br>
>> About the limit to MAX_CATS categories in the base map (ticket<br>
#3198),<br>
>> there is not really a reason for it. I guess it is meant to protect<br>
>> against out-of-memory errors.<br>
><br>
><br>
> Does this mean we can just take out the lines referencing it ?<br>
<br>
Maybe replace the error with a warning. I tested with a base map with<br>
1.4 million categories (output of i.segment) and it works, but I<br>
needed to reduce the number of bins from 1000 to 100 to avoid OOM.<br>
With a low number of bins, the memory consumption of r.stats.quantile<br>
is much less than what r.univar requires, but a low number of bins can<br>
in some cases (histogram of the cell values with a peak) lead to<br>
higher memory consumption.<br>
<br>
r.univar does memory reallocation on demand which means that for a<br>
short time, double the amount of memory is needed. r.univar2 makes a<br>
guess about how much memory is required and then allocates all at<br>
once. Still, the binning approach of r.(stats.)quantile usually needs<br>
less memory.<br>
<br>
Markus M<br>
<br>
</div></div></blockquote>
<br>
<br>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><span style="color:rgb(39,78,19)"><span style="font-family:tahoma,sans-serif"><font size="4"><b>James Duffy</b></font><br>PhD Researcher<br>Environment and Sustainability Institute<br>Penryn Campus<br>University of Exeter<br>Penryn<br>Cornwall<br>TR10 9FE</span></span><br></div></div></div></div></div></div>
</div></div>