[pdal] Entwine performance (recommended hardware)

[Luigi Pirelli]

On Fri, 29 May 2020 at 10:57, Albion SHABANI <albi.dony at hotmail.com> wrote:

> Thank you for your advice.
>
> I have to convert lots of files (like more than 1To and counting) for
> different projects.
>
> I have one with 500g almost (451 las files)
>
> I tried to build all of the input files at once but even If I have an EC2
> with 16CPU and 128 Gio RAM it did not work (memory at 100%), convertion
> killed by the system. I don't know if the memory problem is the use of
> docker image.
>

no... it's not a docker problem, but (AFAIK) allocating meomory for all
that points to maintain a scale independent visualization. BTW pdal crew
can be more precise on thsi aspect.


> So, I am trying your solution (Luigi) to avoid memory problems.
>
> For example, If I have 500go to convert so let's say 700 las files.
>
> If I want to convert them, I have to specify maximum number of files per
> sequence ? or just give total_part=700
>

> In your case u have 256, this number is the number of las files in that
> folder ? or it is the maximum number of las files in a sequence to be put
> by entwine.
>

that's a documentation fault and mine toio... I've to contribute back to
documentation adding more datails. Parts is the number of suwares in which
the covered areas have to be divided => HAVE to be a 2^ number => low
number => big areas and points to index => BIIIIIG memory use! There you
have to find a compromise with processing time and memory occupation.

>
> I have the feeling that if I leave the total_parts at 700, entwine is
> putting everything in the first sequence and when I will merge there will
> be only one id (nothing to merge).  So it's the same problem as if i launch
> all of them at once !
>

"I leave the total_parts at 700" could be a bug... should give error
because in not a 2^ number. BTW it's amost important for the processing
time to spatially order all points in morton ordering, this facilitate a
lot the building of the indexes and memory occupation.

>
> Thank you very much for your help, I'm struggling to understand how does
> this work and be able to convert lots of files with entwine.
>
> and by the way from this docker
> https://hub.docker.com/r/connormanning/entwine *entwine does not work
> with e57 files !!*
>

I see many traffic in dev list/ciommits about this... btw during morton
ordering with pdal you can also translate in other formats becasue pdal is
able to manage e57

>
> Best regards,
> Albion
>
> _______________________________________________
> pdal mailing list
> pdal at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/pdal


Luigi Pirelli

**************************************************************************************************
* LinkedIn: https://www.linkedin.com/in/luigipirelli
* Stackexchange: http://gis.stackexchange.com/users/19667/luigi-pirelli
* GitHub: https://github.com/luipir
* Book: Mastering QGIS3 - 3rd Edition
<https://www.packtpub.com/eu/application-development/mastering-geospatial-development-qgis-3x-third-edition>
* Hire a team: http://www.qcooperative.net
**************************************************************************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pdal/attachments/20200529/af978936/attachment-0001.html>