[pdal] Entwine performance (recommended hardware)
Luigi Pirelli
luipir at gmail.com
Wed May 27 09:28:55 PDT 2020
Hi
I did mine 30GB laz repo in less than 1 day with an old i7 4core and 16GB
ram and SSD.
To avoid any memory problems you should (thanks to the help of Adam steer)
1) morton ordering all files (this facilitate a lot processing)
eg with:
#!/bin/bash
files=$( ls ./*.laz | sort )
for file in ${files}
do
filename=$( basename ${file} )
echo "Morton ordering: $file"
if [[ -f ./morton_ordered/${filename} ]]; then
echo "... already processed"
continue
fi
pdal translate -i ${file} -o "./morton_ordered/${filename}" --filter
mortonorder
if [ $? -ne 0 ]; then
echo "failed with: $?"
exit 1
fi
done
2) then build in parts e.g. with
total_parts=256
for (( part=1; part<=$total_parts; part++ ))
do
echo "************************* ${part} of ${total_parts}"
entwine build -i ./morton_ordered/ -o ./entwine/Avila/ --tmp
./temp_process/ --threads 1 -s $part $total_parts
if [ $? -ne 0 ]; then
echo "$part/$total_parts exit with error: $?"
exit 1
fi
done
I leaved thread to 1 to reduce memory occupation during runtime... I do not
know what would be the limit. For sure using all 8 thread will create crash
for me and my data
3) do entwine merging with:
entiwine merge <path for generated in step 2>
regards
Luigi Pirelli
**************************************************************************************************
* LinkedIn: https://www.linkedin.com/in/luigipirelli
* Stackexchange: http://gis.stackexchange.com/users/19667/luigi-pirelli
* GitHub: https://github.com/luipir
* Book: Mastering QGIS3 - 3rd Edition
<https://www.packtpub.com/eu/application-development/mastering-geospatial-development-qgis-3x-third-edition>
* Hire a team: http://www.qcooperative.net
**************************************************************************************************
On Wed, 27 May 2020 at 17:59, Albion SHABANI <albi.dony at hotmail.com> wrote:
> Hello ,
>
>
> I have very large files like : >500Go and bigger (e57 and las).
>
> What is the average time to convert these amount of data with Entwine
> (using the docker : https://hub.docker.com/r/connormanning/entwine) or is
> even possible ?
>
> What hardware configuration would you recommend ( ? CPU and ? RAM) ?
>
> Best Regards,
> Albion
>
> _______________________________________________
> pdal mailing list
> pdal at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/pdal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pdal/attachments/20200527/e63eca3e/attachment-0001.html>
More information about the pdal
mailing list