<p dir="ltr">Thanks, </p>
<p dir="ltr">This is a very interesting development. I am trying to do a PhD using only open source software (Windows will soon go out the window!). I will take a good look at this post.</p>
<p dir="ltr">Nicolas Cadieux M.Sc.<br>
Les Entreprises Archéotec inc. <br>
8548, rue Saint-Denis Montréal H2P 2H2<br>
Téléphone: 514.381.5112 Fax: 514.381.4995<br>
www.archeotec.ca</p>
<div class="gmail_quote">On Jun 5, 2015 7:24 PM, Carl Godkin <cgodkin@gmail.com> wrote:<br type='attribution'><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br /><div>Hi Nicolas,</div><div><br /></div><div>You might also be interested in a new development discussed over the past couple of</div><div>days on the PDAL list.</div><div><br /></div><div>See <a href="http://lists.osgeo.org/pipermail/pdal/2015-June/000590.html">http://lists.osgeo.org/pipermail/pdal/2015-June/000590.html</a></div><div><br /></div><div>Hope it helps,</div><div><br /></div><div>carl</div><div><br /></div><div><br /><div class="elided-text">On Fri, Jun 5, 2015 at 4:13 PM, Even Rouault <span dir="ltr"><<a href="mailto:even.rouault@spatialys.com">even.rouault@spatialys.com</a>></span> wrote:<br /><blockquote style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb( 204 , 204 , 204 );border-left-style:solid;padding-left:1ex"><div><div>Le samedi 06 juin 2015 00:18:45, Nicolas Cadieux a écrit :<br />
> Hi,<br />
> I have been using gdal_grid to interpolate LiDAR data for some time now.<br />
> This would be a typical command:<br />
><br />
> gdal_grid -l P0344_sol2004_XYZI_MTM8_CGVD28_150529 -txe 348733.0 401135.0<br />
> -tye 5568587.0 5773796.0 -a<br />
> nearest:radius1=3.0:radius2=3.0:angle=0.0:nodata=0.0 -outsize 52402.0<br />
> 205209.0 -of GTiff<br />
> H:/Nicolas/pytemp/OutPut\P0344_sol2004_XYZI_MTM8_CGVD28_150529.vrt<br />
> H:/Nicolas/pytemp/OutPut\P0344_sol2004_XYZI_MTM8_CGVD28_150529.tif --confi<br />
> g GDAL_NUM_THREADS ALL_CPUS<br />
><br />
> As you can see, this is a huge raster and the csv. file (Pointed with a<br />
> .vrt) contains over a billion points. I am trying to do this in one shot<br />
> to avoid boundary effects. (I know I can split this .csv file into<br />
> smaller bits and make this more efficient). The computer I am using has<br />
> 12 cores, 64BG of memory (smaller than the csv file) and a 1TB pcie solid<br />
> state drive. It should manage the task but it has been running for 5 days<br />
> with less than 10% done). I have access to a super computer with gdal if<br />
> all else fails.<br />
><br />
> This is my question. The "--config GDAL_NUM_THREADS ALL_CPUS" or "--config<br />
> GDAL_NUM_THREADS 12" does not seem to work with the nearest neighbor<br />
> algorithm. (It seem to work with the -a count command) If I use this on a<br />
> smaller data set, I see no difference in speed or cpu usage between using<br />
> the "ALL_CPUS" or not using this switch. Is this a bug or simply a<br />
> limitation of the nearest neighbor algorithm?<br />
<br />
</div></div>Nicolas,<br />
<br />
The threading mechanism is generic for all algorithms . Add "--debug on" and<br />
you should see traces like "GDAL_GRID: Using 12 threads". Now, a reason for it<br />
not being efficient is that the work buffer used by gdalgrid (16 MB) is too small<br />
w.r.t to nearest speed. Hum, actually looking more closely at the code, I see<br />
that the quad tree to index the points will be rebuilt for each work buffer.<br />
This is likely the main cause of the inefficiency and why you don't see<br />
measurable differences between non threaded or threaded computations (since the<br />
building of the quadtree is not multithreaded). If you compiled from source,<br />
you could try changing the "const GUInt32 nDesiredBufferSize = 16*1024*1024;"<br />
line in apps/gdal_grid.cpp to a much higher value. A more proper solution<br />
would be to avoid rebuilding the quad tree for each work buffer. This is mainly<br />
code restructuring.<br />
<br />
An alternative to gdal_grid would be to use gdal_rasterize + gdal_fillnodata.<br />
This is usually much much faster, so you could try to check if it gives the<br />
results you expect.<br />
<br />
You could also try scipy :<br />
<a href="http://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.interpolate.griddata.html#scipy.interpolate.griddata">http://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.interpolate.griddata.html#scipy.interpolate.griddata</a><br />
Not sure how it behaves with that number of points.<br />
<br />
Even<br />
<br />
> I don't want to move this<br />
> to a super computer if it will not multi-thread properly.<br />
><br />
> Thanks,<br />
> Nicolas Cadieux<br />
><br />
> (Gdal1.11.2 downloaded with OSGEO4W 64bit install with QGIS 2.8 on windows<br />
> 7_64) I use GDAL from the command line not from QGIS.<br />
><br />
><br />
> Nicolas Cadieux M.Sc.<br />
> Les Entreprises Archéotec inc.<br />
> 8548, rue Saint-Denis Montréal H2P 2H2<br />
> Téléphone: 514.381.5112 Fax: 514.381.4995<br />
> <a href="http://www.archeotec.ca">www.archeotec.ca</a><br />
> _______________________________________________<br />
> gdal-dev mailing list<br />
> <a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a><br />
> <a href="http://lists.osgeo.org/mailman/listinfo/gdal-dev">http://lists.osgeo.org/mailman/listinfo/gdal-dev</a><br />
<br />
<font color="#888888">--<br />
Spatialys - Geospatial professional services<br />
<a href="http://www.spatialys.com">http://www.spatialys.com</a><br />
</font><div><div>_______________________________________________<br />
gdal-dev mailing list<br />
<a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a><br />
<a href="http://lists.osgeo.org/mailman/listinfo/gdal-dev">http://lists.osgeo.org/mailman/listinfo/gdal-dev</a></div></div></blockquote></div><br /></div></div>
</blockquote></div>