<div dir="ltr"><div>Dear Aaron, <br></div><div><br></div><div>Thanks for your report and all your work to make GRASS raster modules run faster, we like that :) Huge thanks as well to Vaclav, Huidae and Maris for their commitment to the project and mentoring! You all did a great team work!<br></div><div><br></div><div><div>One question: are these changes planned to be merged before the
creation of grass 8 branch or will they remain for 8+? Maybe add a
milestone to clarify? </div><div></div></div><div>One minor observation: all links currently point to the same PR</div><br><div>We hope to keep seeing you around, Aaron!</div><div><br></div><div>All the best,</div><div>Vero<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">El mar, 24 ago 2021 a las 3:18, Aaron Saw Min Sern (<<a href="mailto:aaronsms@u.nus.edu">aaronsms@u.nus.edu</a>>) escribió:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hello everyone,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Here is my final report for GSoC 2021 project, <i>Parallelization of Raster Modules for GRASS GIS</i>.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<b>Abstract</b></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
The goal of this project is to introduce parallelization to existing raster modules in GRASS GIS using OpenMP. This will allow users to take advantage of more cores in their hardware to speed up the computation time especially for large raster files with large
computation cost. The key challenge of this project is to separate the parallelizable components from the sequential part of the modules without introducing too much overhead in terms of memory, disk or computation resources<b>.<br>
</b></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<b><br>
</b></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<b>Milestones</b></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<b><br>
</b></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
In total, I have introduced OpenMP support to 8 raster modules in GRASS GIS. The pull requests to each module are as follows:<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<ul>
<li>r.univar - <a href="https://github.com/OSGeo/grass/pull/1634" target="_blank">https://github.com/OSGeo/grass/pull/1634</a></li><li>r.neighbors - <a href="https://github.com/OSGeo/grass/pull/1634" id="gmail-m_6379382274284092570LPlnk571158" target="_blank">
https://github.com/OSGeo/grass/pull/</a><a href="https://github.com/OSGeo/grass/pull/1724" target="_blank">1724</a></li><li>r.mfilter - <a href="https://github.com/OSGeo/grass/pull/1634" id="gmail-m_6379382274284092570LPlnk809911" target="_blank">
https://github.com/OSGeo/grass/pull/</a><a href="https://github.com/OSGeo/grass/pull/1708" target="_blank">1708</a></li><li>r.resamp.filter - <a href="https://github.com/OSGeo/grass/pull/1634" id="gmail-m_6379382274284092570LPlnk" target="_blank">
https://github.com/OSGeo/grass/pull/</a><a href="https://github.com/OSGeo/grass/pull/1759" target="_blank">1759</a></li><li>r.resamp.interp - <a href="https://github.com/OSGeo/grass/pull/1634" id="gmail-m_6379382274284092570LPlnk" target="_blank">
https://github.com/OSGeo/grass/pull/</a><a href="https://github.com/OSGeo/grass/pull/1771" target="_blank">1771</a></li><li>r.slope.aspect - <a href="https://github.com/OSGeo/grass/pull/1634" id="gmail-m_6379382274284092570LPlnk" target="_blank">
https://github.com/OSGeo/grass/pull/</a><a href="https://github.com/OSGeo/grass/pull/1767" target="_blank">1767</a></li><li>r.series - <a href="https://github.com/OSGeo/grass/pull/1634" id="gmail-m_6379382274284092570LPlnk620193" target="_blank">
https://github.com/OSGeo/grass/pull/</a><a href="https://github.com/OSGeo/grass/pull/1776" target="_blank">1776</a></li><li>r.patch - <a href="https://github.com/OSGeo/grass/pull/1634" id="gmail-m_6379382274284092570LPlnk" target="_blank">https://github.com/OSGeo/grass/pull/</a><a href="https://github.com/OSGeo/grass/pull/1782" target="_blank">1782</a></li></ul>
<div>
<p>Firstly, I have greatly underestimated the complexity of the work. Up to 20 modules were initially proposed at first but after the second week. However, it became clear that we had to cut down on the number of target modules and focus more on designing the
algorithms. The modules we targeted behave differently as compared to some modules that had received OpenMP support in the past such as
<em>r.sun</em>. In particular, the modules need to keep the same of behavior of having low memory footprint even after the parallelization, unlike
<em>r.sun</em> which loads the entire raster map in-memory.</p>
<p><br>
</p>
<div style="margin-top:0px;margin-bottom:0px">During the first half of the GSoC, with the mentors’ discussion, we have come out with three different approaches to introducing parallel support to
<em>r.neighbors</em>. After benchmarking their performance and taking account of their memory/disk usage, we decided to settle with the last approach which requires us to add an extra parameter
<em>memory</em> to allow users to adjust their memory consumption. With this approach, we have to allow the modules to process the raster map by chunks. Once we settled about the design, we started applying the same approach to other similar modules with low
memory footprints. <br>
</div>
<div style="margin-top:0px;margin-bottom:0px"><br>
</div>
<div style="margin-top:0px;margin-bottom:0px">For more information regarding the implementation, see
<a href="https://grasswiki.osgeo.org/wiki/Raster_Parallelization_with_OpenMP" id="gmail-m_6379382274284092570LPlnk" target="_blank">
https://grasswiki.osgeo.org/wiki/Raster_Parallelization_with_OpenMP</a>.</div>
<div></div>
<br>
<p>Furthermore, test scripts were included in the modules to ensure the consistency of the results. Benchmark scripts were added to allow users to easily benchmark the performance of the parallelization to monitor the speedup in their own local machine. User
documentations were also modified to include sections detailing how to make use of the newly added features.</p>
</div>
<b></b></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<b><br>
</b></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<b>Future Work</b></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<p>In the future, more raster modules can be parallelized using similar approach. Then, we can consider tackling more complex modules such as
<em>r.watershed</em> and <em>r.mapcalc</em>. Also, we could consider exploring 3D raster modules as well.</p>
<p><br>
</p>
<p>Furthermore, when we implement parallelization for <em>r.univar</em>, we notice that modules that produce statistics involving arithmetic can often have floating point discrepancies when dealing with large summation. Because of this, computation using different
number of threads will now produce different results due to having different order of arithmetic. One idea would be to introduce
<i>Kahan Summation algorithm</i> to reduce the floating-point discrepancies. However, this still would not guarantee the consistency of results.</p>
<b></b></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<b><br>
</b></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<b>Permanent Links</b></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<b><br>
</b></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<div style="margin-top:0px;margin-bottom:0px">For the project overview, please visit the
<a href="https://summerofcode.withgoogle.com/dashboard/project/6280792767987712/overview/" id="gmail-m_6379382274284092570LPlnk421846" target="_blank">
https://summerofcode.withgoogle.com/dashboard/project/6280792767987712/overview</a>/.</div>
<div style="margin-top:0px;margin-bottom:0px">For the project timeline and logs, please visit the
<a href="https://trac.osgeo.org/grass/wiki/GSoC/2021/RasterParallelization" id="gmail-m_6379382274284092570LPlnk" target="_blank">
https://trac.osgeo.org/grass/wiki/GSoC/2021/RasterParallelization</a>.</div>
<div></div>
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I would like to huge thanks to Huidae Cho, Vaclav Petras and Māris Nartišs for their valuable guidance. And I would like to thank the GRASS community for the valuable feedbacks and support. Lastly, I would like to thank the GSoC team for this opportunity.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Thanks all! <br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Warmest regards,<br>
Aaron Saw Min Sern<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
</div>
_______________________________________________<br>
grass-dev mailing list<br>
<a href="mailto:grass-dev@lists.osgeo.org" target="_blank">grass-dev@lists.osgeo.org</a><br>
<a href="https://lists.osgeo.org/mailman/listinfo/grass-dev" rel="noreferrer" target="_blank">https://lists.osgeo.org/mailman/listinfo/grass-dev</a><br>
</blockquote></div>