[pdal] Entwine performance (recommended hardware)

[Luigi Pirelli]

Hi

I did mine 30GB laz repo in less than 1 day with an old i7 4core and 16GB
ram and SSD.

To avoid any memory problems you should (thanks to the help of Adam steer)

1) morton ordering all files (this facilitate a lot processing)
eg with:
#!/bin/bash
files=$( ls ./*.laz | sort )
for file in ${files}
do
    filename=$( basename ${file} )
    echo "Morton ordering: $file"
    if [[ -f ./morton_ordered/${filename} ]]; then
echo "... already processed"
        continue
    fi
    pdal translate -i ${file} -o "./morton_ordered/${filename}" --filter
mortonorder
    if [ $? -ne 0 ]; then
        echo "failed with: $?"
        exit 1
    fi
done

2) then build in parts e.g. with
total_parts=256
for (( part=1; part<=$total_parts; part++ ))
do
    echo "************************* ${part} of ${total_parts}"
    entwine build -i ./morton_ordered/ -o ./entwine/Avila/ --tmp
./temp_process/ --threads 1 -s $part $total_parts
    if [ $? -ne 0 ]; then
        echo "$part/$total_parts exit with error: $?"
        exit 1
    fi
done

I leaved thread to 1 to reduce memory occupation during runtime... I do not
know what would be the limit. For sure using all 8 thread will create crash
for me and my data

3) do entwine merging with:
entiwine merge <path for generated in step 2>

regards

Luigi Pirelli

**************************************************************************************************
* LinkedIn: https://www.linkedin.com/in/luigipirelli
* Stackexchange: http://gis.stackexchange.com/users/19667/luigi-pirelli
* GitHub: https://github.com/luipir
* Book: Mastering QGIS3 - 3rd Edition
<https://www.packtpub.com/eu/application-development/mastering-geospatial-development-qgis-3x-third-edition>
* Hire a team: http://www.qcooperative.net
**************************************************************************************************


On Wed, 27 May 2020 at 17:59, Albion SHABANI <albi.dony at hotmail.com> wrote:

> Hello ,
>
>
> I have very large files like : >500Go and bigger (e57 and las).
>
> What is the average time to convert these amount of data with Entwine
> (using the docker : https://hub.docker.com/r/connormanning/entwine) or is
> even possible ?
>
> What hardware configuration would you recommend ( ? CPU and ? RAM) ?
>
> Best Regards,
> Albion
>
> _______________________________________________
> pdal mailing list
> pdal at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/pdal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pdal/attachments/20200527/e63eca3e/attachment-0001.html>