<div dir="ltr">Hi Nicolas,<div><br></div><div>I see someone already posted something about lidar2dems. We also wanted to use only open-source, so put together this project to easily be able to make DEMs using what's out there already. </div><div>The documentation gives an overview of what it can do and a tutorial for processing a large area.</div><div><br></div><div><a href="http://applied-geosolutions.github.io/lidar2dems/">http://applied-geosolutions.github.io/lidar2dems/</a><br></div><div><br></div><div>I'd like to add multiprocessing support, as it processes in pieces as it is, so wouldn't take much more work to split it up over processors.</div><div><br></div><div>-matt</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jun 15, 2015 at 3:00 PM, Nicolas Cadieux <span dir="ltr"><<a href="mailto:nicolas.cadieux@archeotec.ca" target="_blank">nicolas.cadieux@archeotec.ca</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Thanks Even,<br>
One last question about the buffer. Is 2GB is the max buffer size for gdal_grid? If I select that, am I likely to run into problem with some values? Apart from trial and error, is there a balance that should be stuck between available computer memory (currently at 64GB), input file size and the buffer or can I simply select a 2GB buffer?<br>
<br>
Merci beaucoup pour votre aide.<br>
<span class="im HOEnZb"><br>
Nicolas Cadieux M.Sc.<br>
Les Entreprises Archéotec inc. <br>
8548, rue Saint-Denis Montréal H2P 2H2<br>
Téléphone: <a href="tel:514.381.5112" value="+15143815112">514.381.5112</a> Fax: <a href="tel:514.381.4995" value="+15143814995">514.381.4995</a><br>
<a href="http://www.archeotec.ca" rel="noreferrer" target="_blank">www.archeotec.ca</a><br>
<br>
</span><div class="HOEnZb"><div class="h5">On Jun 12, 2015 17:29, Even Rouault <<a href="mailto:even.rouault@spatialys.com">even.rouault@spatialys.com</a>> wrote:<br>
><br>
> Le vendredi 12 juin 2015 23:18:26, Nicolas Cadieux a écrit :<br>
> > Hi Even,<br>
> > Thank you very much. I increased the buffer to 1.5 GB<br>
> > (512+512+514)*1024*1024 and I can now work much much faster with<br>
> > gdal_grid.<br>
> ><br>
> > Does gdal_fillnodata have the same type of variable that could speed things<br>
> > up? It is also slow and does not seem speeded up by the "All_CPUS".<br>
><br>
> No, no tunable for fillnodata, and it doesn't support multi-threaded<br>
> parallelization.<br>
><br>
> ><br>
> > I will use warp and the raster calculator in the future so you have more<br>
> > magic up your sleeve, I would appreciate it.<br>
> ><br>
> > Cheers and thanks again.<br>
> ><br>
> > Nicolas Cadieux M.Sc.<br>
> > Les Entreprises Archéotec inc.<br>
> > 8548, rue Saint-Denis Montréal H2P 2H2<br>
> > Téléphone: 514.381.5112 Fax: 514.381.4995<br>
> > <a href="http://www.archeotec.ca" rel="noreferrer" target="_blank">www.archeotec.ca</a><br>
> ><br>
> > On Jun 5, 2015 19:13, Even Rouault <<a href="mailto:even.rouault@spatialys.com">even.rouault@spatialys.com</a>> wrote:<br>
> > > Le samedi 06 juin 2015 00:18:45, Nicolas Cadieux a écrit :<br>
> > > > Hi,<br>
> > > > I have been using gdal_grid to interpolate LiDAR data for some time<br>
> > > > now. This would be a typical command:<br>
> > > ><br>
> > > > gdal_grid -l P0344_sol2004_XYZI_MTM8_CGVD28_150529 -txe 348733.0<br>
> > > > 401135.0 -tye 5568587.0 5773796.0 -a<br>
> > > > nearest:radius1=3.0:radius2=3.0:angle=0.0:nodata=0.0 -outsize 52402.0<br>
> > > > 205209.0 -of GTiff<br>
> > > > H:/Nicolas/pytemp/OutPut\P0344_sol2004_XYZI_MTM8_CGVD28_150529.vrt<br>
> > > > H:/Nicolas/pytemp/OutPut\P0344_sol2004_XYZI_MTM8_CGVD28_150529.tif<br>
> > > > --confi g GDAL_NUM_THREADS ALL_CPUS<br>
> > > ><br>
> > > > As you can see, this is a huge raster and the csv. file (Pointed with a<br>
> > > > .vrt) contains over a billion points. I am trying to do this in one<br>
> > > > shot to avoid boundary effects. (I know I can split this .csv file<br>
> > > > into smaller bits and make this more efficient). The computer I am<br>
> > > > using has 12 cores, 64BG of memory (smaller than the csv file) and a<br>
> > > > 1TB pcie solid state drive. It should manage the task but it has been<br>
> > > > running for 5 days with less than 10% done). I have access to a super<br>
> > > > computer with gdal if all else fails.<br>
> > > ><br>
> > > > This is my question. The "--config GDAL_NUM_THREADS ALL_CPUS" or<br>
> > > > "--config GDAL_NUM_THREADS 12" does not seem to work with the nearest<br>
> > > > neighbor algorithm. (It seem to work with the -a count command) If I<br>
> > > > use this on a smaller data set, I see no difference in speed or cpu<br>
> > > > usage between using the "ALL_CPUS" or not using this switch. Is this<br>
> > > > a bug or simply a limitation of the nearest neighbor algorithm?<br>
> > ><br>
> > > Nicolas,<br>
> > ><br>
> > > The threading mechanism is generic for all algorithms . Add "--debug on"<br>
> > > and you should see traces like "GDAL_GRID: Using 12 threads". Now, a<br>
> > > reason for it not being efficient is that the work buffer used by<br>
> > > gdalgrid (16 MB) is too small w.r.t to nearest speed. Hum, actually<br>
> > > looking more closely at the code, I see that the quad tree to index the<br>
> > > points will be rebuilt for each work buffer. This is likely the main<br>
> > > cause of the inefficiency and why you don't see measurable differences<br>
> > > between non threaded or threaded computations (since the building of the<br>
> > > quadtree is not multithreaded). If you compiled from source, you could<br>
> > > try changing the "const GUInt32 nDesiredBufferSize = 16*1024*1024;" line<br>
> > > in apps/gdal_grid.cpp to a much higher value. A more proper solution<br>
> > > would be to avoid rebuilding the quad tree for each work buffer. This is<br>
> > > mainly code restructuring.<br>
> > ><br>
> > > An alternative to gdal_grid would be to use gdal_rasterize +<br>
> > > gdal_fillnodata. This is usually much much faster, so you could try to<br>
> > > check if it gives the results you expect.<br>
> > ><br>
> > > You could also try scipy :<br>
> > > <a href="http://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.interpol" rel="noreferrer" target="_blank">http://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.interpol</a><br>
> > > ate.griddata.html#scipy.interpolate.griddata Not sure how it behaves with<br>
> > > that number of points.<br>
> > ><br>
> > > Even<br>
> > ><br>
> > > > I don't want to move this<br>
> > > > to a super computer if it will not multi-thread properly.<br>
> > > ><br>
> > > > Thanks,<br>
> > > > Nicolas Cadieux<br>
> > > ><br>
> > > > (Gdal1.11.2 downloaded with OSGEO4W 64bit install with QGIS 2.8 on<br>
> > > > windows 7_64) I use GDAL from the command line not from QGIS.<br>
> > > ><br>
> > > ><br>
> > > > Nicolas Cadieux M.Sc.<br>
> > > > Les Entreprises Archéotec inc.<br>
> > > > 8548, rue Saint-Denis Montréal H2P 2H2<br>
> > > > Téléphone: 514.381.5112 Fax: 514.381.4995<br>
> > > > <a href="http://www.archeotec.ca" rel="noreferrer" target="_blank">www.archeotec.ca</a><br>
> > > > _______________________________________________<br>
> > > > gdal-dev mailing list<br>
> > > > <a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a><br>
> > > > <a href="http://lists.osgeo.org/mailman/listinfo/gdal-dev" rel="noreferrer" target="_blank">http://lists.osgeo.org/mailman/listinfo/gdal-dev</a><br>
><br>
> --<br>
> Spatialys - Geospatial professional services<br>
> <a href="http://www.spatialys.com" rel="noreferrer" target="_blank">http://www.spatialys.com</a><br>
_______________________________________________<br>
gdal-dev mailing list<br>
<a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a><br>
<a href="http://lists.osgeo.org/mailman/listinfo/gdal-dev" rel="noreferrer" target="_blank">http://lists.osgeo.org/mailman/listinfo/gdal-dev</a></div></div></blockquote></div><br></div>