[pdal] Entwine performance (recommended hardware)

[Albion SHABANI]

Thank you for your advice.

I have to convert lots of files (like more than 1To and counting) for different projects.

I have one with 500g almost (451 las files)

I tried to build all of the input files at once but even If I have an EC2 with 16CPU and 128 Gio RAM it did not work (memory at 100%), convertion killed by the system. I don't know if the memory problem is the use of docker image.

So, I am trying your solution (Luigi) to avoid memory problems.

For example, If I have 500go to convert so let's say 700 las files.

If I want to convert them, I have to specify maximum number of files per sequence ? or just give total_part=700

In your case u have 256, this number is the number of las files in that folder ? or it is the maximum number of las files in a sequence to be put by entwine.

I have the feeling that if I leave the total_parts at 700, entwine is putting everything in the first sequence and when I will merge there will be only one id (nothing to merge).  So it's the same problem as if i launch all of them at once !

Thank you very much for your help, I'm struggling to understand how does this work and be able to convert lots of files with entwine.

and by the way from this docker https://hub.docker.com/r/connormanning/entwine entwine does not work with e57 files !!

Best regards,
Albion

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pdal/attachments/20200529/beacc921/attachment.html>