<div dir="auto"><div>This solution is amazing and fits into what I am doing perfectly!<div dir="auto">Thank you Stefan and wish I had talked about the nature of my work in the first place.</div><div dir="auto">If you could explain more by an example about </div><div dir="auto">"<span style="font-family:sans-serif">carefully consider the resolution for your analysis based on the required precision..." </span></div><div dir="auto"><span style="font-family:sans-serif">just to make sure I wouldn't missing anything.</span></div><div dir="auto"><br></div><div dir="auto">I appreciate everyone's feedbacks and solutions offered.</div><div dir="auto"><br></div><div dir="auto">Kind regards,</div><div dir="auto"><br></div><div dir="auto">Mehrdad </div><br><br><div class="gmail_quote"><div dir="ltr">On Sat, Mar 24, 2018, 05:37 Stefan Blumentrath <<a href="mailto:Stefan.Blumentrath@nina.no">Stefan.Blumentrath@nina.no</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="NO-BOK" link="blue" vlink="purple">
<div class="m_-4496778070544474910WordSection1">
<p class="m_-4496778070544474910MsoPlainText"><span lang="EN-GB">Hei again Mehrdad,<u></u><u></u></span></p>
<p class="m_-4496778070544474910MsoPlainText"><span lang="EN-GB"><u></u> <u></u></span></p>
<p class="m_-4496778070544474910MsoPlainText"><span lang="EN-GB">A different appoach to tackle your problem could be to:<u></u><u></u></span></p>
<p class="m_-4496778070544474910MsoPlainText"><span lang="EN-GB">1) run r.neighbors with a circular window that has a size according to your buffer distance while computing the statistic of interest<u></u><u></u></span></p>
<p class="m_-4496778070544474910MsoPlainText"><span lang="EN-GB">2) use v.what.rast or r.what to get neighbourhood values at your point locations...<u></u><u></u></span></p>
<p class="m_-4496778070544474910MsoPlainText"><span lang="EN-GB"><u></u> <u></u></span></p>
<p class="m_-4496778070544474910MsoPlainText"><span lang="EN-GB">That way you can circumvent overlap issues and for the points you do not even have to build topology (or can use a linked OGR map (v.external), if I am not mistaken)...<u></u><u></u></span></p>
<p class="m_-4496778070544474910MsoPlainText"><span lang="EN-GB"><u></u> <u></u></span></p>
<p class="m_-4496778070544474910MsoPlainText"><span lang="EN-GB">In any case you would have to carefully consider the resolution for your analysis based on the required precision...<u></u><u></u></span></p>
<p class="m_-4496778070544474910MsoPlainText"><span lang="EN-GB"><u></u> <u></u></span></p>
<p class="m_-4496778070544474910MsoPlainText">Cheers<u></u><u></u></p>
<p class="m_-4496778070544474910MsoPlainText">Stefan<u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Markus Metz <<a href="mailto:markus.metz.giswork@gmail.com" target="_blank" rel="noreferrer">markus.metz.giswork@gmail.com</a>>
<br>
<b>Sent:</b> fredag 23. mars 2018 17.57<br>
<b>To:</b> Mehrdad Varedi <<a href="mailto:varedi@waterlix.com" target="_blank" rel="noreferrer">varedi@waterlix.com</a>><br>
<b>Cc:</b> Stefan Blumentrath <<a href="mailto:Stefan.Blumentrath@nina.no" target="_blank" rel="noreferrer">Stefan.Blumentrath@nina.no</a>>; <a href="mailto:grass-user@lists.osgeo.org" target="_blank" rel="noreferrer">grass-user@lists.osgeo.org</a><br>
<b>Subject:</b> Re: [GRASS-user] Need Help with speeding up v.import and reducing/removing null values from v.rast.stat results<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
<br>
On Fri, Mar 23, 2018 at 4:30 PM, Mehrdad Varedi <<a href="mailto:varedi@waterlix.com" target="_blank" rel="noreferrer">varedi@waterlix.com</a>> wrote:<br>
><br>
> Thank you very much Stefan and Moritz for your attention and replies.<br>
> I found that you both exactly understand where is the problem. Let me be more specific about my primary issue.<br>
><br>
> - v.buffer and v.import (v.in.ogr) - They freez.<br>
> I found that my problem was reported in a post: #2185 new defect Painfully Slow 'v.in.ogr' Vector Import
<br>
> Here is the link: <a href="https://trac.osgeo.org/grass/ticket/2185#no1" target="_blank" rel="noreferrer">https://trac.osgeo.org/grass/ticket/2185#no1</a><br>
><br>
> Now two questions: <br>
><br>
> 1 - Two patches has been offered in this post that apparently has solved the problem. I really don't know how to use patches and when they should be executed or compiled. Would you please refer me to some resources to educate myself to how to use the patches,
where to get the compiler, etc.<br>
><br>
> 2 - Are these patches applied already in GRASS 7.4.0?<u></u><u></u></p>
</div>
<p class="MsoNormal">The issue in #2185 has been fixed already, but differently. The suggested patches were fixing symptoms but not the cause.<u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">><br>
> - Some background about the data I have:<br>
><br>
> As I said there are 700,000 points , the distance between the points are 3 meters. I create buffers with the size of 3, 7, 20, and 50 meters around each point. Then will use v.rast.stats to calculate statistics on rasters with the resolution of 1 square meter.<br>
> <br>
> I could import the buffer layers created in QGIS using fixed distance buffer function and import them for the 3 and 7 meter buffer layers. then v.rast.stats could complete the calculations after 2 hours for the 3 meter buffer and after 4.5 hours for the 7
meter layer.<br>
><br>
> v.import for the 20 meter buffer was frozen on "Breaking Boundaries ... " step. I have provided the log of messages that I get. all of the steps is done in few minutes and it stuck on the "Breaking Boundaries ..." for more than 12 hours for the 20 meter buffer.
( I am sure it would be the same for the 50 meter buffer)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">If the buffer radius is a bit larger than half the distance between points, you will get some overlapping buffers, but that should not be too bad. If the buffer radius is much larger than the distance between
points, e.g. distance between points = 3 and buffer distance = 50, the result will be a topological challenge with multiple overlapping buffers.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Alternatively you can approximate a common buffer for all points by 1) creating a concave hull with v.concave.hull [0], 2) buffering that hull.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">HTH,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Markus M<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><br>
[0] <a href="https://grass.osgeo.org/grass74/manuals/addons/v.concave.hull.html" target="_blank" rel="noreferrer">
https://grass.osgeo.org/grass74/manuals/addons/v.concave.hull.html</a><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">><br>
><br>
> (Thu Mar 22 22:41:46 2018) <br>
> v.import input=E:\Projects-Analysis\London\Data\Output\P3ntsOnMains\Buffers\LondonBuffers layer=Buffer20_P3intsOnMains output=Buffer20_P3intsOnMains --overwrite<br>
> Check if OGR layer <Buffer20_P3intsOnMains> contains polygons...<br>
> WARNING: Vector map <Buffer20_P3intsOnMains> already exists and will be overwritten<br>
> WARNING: Table <Buffer20_P3intsOnMains> linked to vector map <Buffer20_P3intsOnMains> does not exist<br>
> Creating attribute table for layer <Buffer20_P3intsOnMains>...<br>
> Column name <cat> renamed to <cat_><br>
> Importing 706628 features (OGR layer <Buffer20_P3intsOnMains>)...<br>
> -----------------------------------------------------<br>
> Registering primitives...<br>
> 706628 primitives registered<br>
> 14839188 vertices registered<br>
> Number of nodes: 681288<br>
> Number of primitives: 706628<br>
> Number of points: 0<br>
> Number of lines: 0<br>
> Number of boundaries: 706628<br>
> Number of centroids: 0<br>
> Number of areas: -<br>
> Number of isles: -<br>
> -----------------------------------------------------<br>
> Cleaning polygons<br>
> -----------------------------------------------------<br>
> Breaking polygons...<br>
> Breaking polygons (pass 1: select break points)...<br>
> Breaking polygons (pass 2: break at selected points)...<br>
> -----------------------------------------------------<br>
> Removing duplicates...<br>
> -----------------------------------------------------<br>
> Breaking boundaries...<br>
><br>
><br>
><br>
> In another try when the distance between my points were 12 meters, everything was working relatively fast and with no problem. increasing the density of points (3 meter distance) has resulted in a very long delay, so much that I think it is in an infinite
loop. <br>
><br>
> I appreciate for your help and looking forward hearing from you.<br>
><br>
> Kind regards,<br>
><br>
> Mehrdad<br>
><br>
><br>
><br>
> Mehrdad Varedi, <a href="http://M.A.Sc" target="_blank" rel="noreferrer">M.A.Sc</a>.<br>
> Cell: +1 (519)722-7057<br>
><br>
> On Fri, Mar 23, 2018 at 9:15 AM, Stefan Blumentrath <<a href="mailto:Stefan.Blumentrath@nina.no" target="_blank" rel="noreferrer">Stefan.Blumentrath@nina.no</a>> wrote:<br>
>><br>
>> Hei Mehrdad,<br>
>><br>
>> If your buffers are overlapping, topology will become an issue, as Moritz pointed out.<br>
>> Johannes Radinger once wrote a workaround for a specific use case that might be of help. See [1]. No idea if it would be possible to come up with something more generic for v.rast.stats…<br>
>><br>
>> In v.rast.stas NAs occur most likely due to vector areas that are not rasterized, which usually happens when polygons are relatively small / narrow compared to resolution of the raster map.<br>
>><br>
>> However, during community sprint these days in Bonn, Martin Landa added centroids to the rasterization algorithm (v.to.rast) that v.rast.stats uses, than can help to reduce the number of NAs.<br>
>><br>
>> Also during community sprint, I created a patch for v.rast.stats [2] that aims at improving the performance of the module by allowing multiple raster input (it needs testing).<br>
>><br>
>> There are possible further improvements, but these require some more discussion with the other devs regarding implementation.<br>
>><br>
>> Cheers<br>
>> Stefan<br>
>><br>
>> 1: <a href="https://lists.osgeo.org/pipermail/grass-user/2017-June/076670.html" target="_blank" rel="noreferrer">
https://lists.osgeo.org/pipermail/grass-user/2017-June/076670.html</a><br>
>> 2: <a href="https://trac.osgeo.org/grass/ticket/3523" target="_blank" rel="noreferrer">https://trac.osgeo.org/grass/ticket/3523</a><br>
>><br>
>> -----Original Message-----<br>
>> From: grass-user <<a href="mailto:grass-user-bounces@lists.osgeo.org" target="_blank" rel="noreferrer">grass-user-bounces@lists.osgeo.org</a>> On Behalf Of Moritz Lennert<br>
>> Sent: fredag 23. mars 2018 10.52<br>
>> To: Mehrdad Varedi <<a href="mailto:varedi@waterlix.com" target="_blank" rel="noreferrer">varedi@waterlix.com</a>>;
<a href="mailto:grass-user@lists.osgeo.org" target="_blank" rel="noreferrer">grass-user@lists.osgeo.org</a><br>
>> Subject: Re: [GRASS-user] Need Help with speeding up v.import and reducing/removing null values from v.rast.stat results<br>
>><br>
>> On 22/03/18 22:40, Mehrdad Varedi wrote:<br>
>> > Hi Everyone,<br>
>> ><br>
>> > I am trying to import few layers with nearly 700,000 buffer areas<br>
>> > around the same number of points. Fr each layer t takes more than 4-5<br>
>> > hours and most of the time, it just stops in the middle of the process.<br>
>> > Any idea that which kind of layers can be imported with a faster speed?<br>
>> > I need o load 4-5 of such layers and it is now 5 days that I am busy<br>
>> > with building a buffer or importing the results.<br>
>> ><br>
>> > Actually I tried v.buffer and it took more than a day until I stopped<br>
>> > the procedure, although I could create the layer with Fixed distance<br>
>> > buffer function in QGIS in 10 minutes.<br>
>> > Any idea to accelerate the process of v.buffer ot v.import in GRASS?<br>
>><br>
>> I would guess that the difference comes from the difference in output<br>
>> format: QGIS will just create non-topological, overlapping buffers (unless you ask it to create unique buffers) while GRASS has to take care of all the topological issues in order to create a topologically clean vector map.<br>
>><br>
>> The same is true both for buffer creation and for import.<br>
>><br>
>> ><br>
>> > Also I want to calculate statistics in each of the buffers and have<br>
>> > the same issue with v.rast.stats function<br>
>><br>
>> You might get faster results rasterizing your buffers and using r.univar with the zones= parameter. You can then import the resulting text file into your database and link it to the vector map, as long as you make sure that the pixel values in the zone map
correspond to the cat values of the vector map.<br>
>><br>
>> ><br>
>> > Another issue with this function is a lot of NAs in the result. No<br>
>> > hole exists in the raster files and it should be something with<br>
>> > snapping I guess which results in removal of many points.<br>
>><br>
>> You would have to be a bit more precise about what exactly you are doing and what exactly the results are. Are you talking about points or polygons ?<br>
>><br>
>> Moritz<br>
>><br>
>><br>
>> _______________________________________________<br>
>> grass-user mailing list<br>
>> <a href="mailto:grass-user@lists.osgeo.org" target="_blank" rel="noreferrer">grass-user@lists.osgeo.org</a><br>
>> <a href="https://lists.osgeo.org/mailman/listinfo/grass-user" target="_blank" rel="noreferrer">https://lists.osgeo.org/mailman/listinfo/grass-user</a><br>
><br>
><br>
><br>
> _______________________________________________<br>
> grass-user mailing list<br>
> <a href="mailto:grass-user@lists.osgeo.org" target="_blank" rel="noreferrer">grass-user@lists.osgeo.org</a><br>
> <a href="https://lists.osgeo.org/mailman/listinfo/grass-user" target="_blank" rel="noreferrer">https://lists.osgeo.org/mailman/listinfo/grass-user</a><u></u><u></u></p>
</div>
</div>
</div>
</div>
</blockquote></div></div></div>