[pdal] Memory management with PDAL/Docker

[Matt Beckley]

Hi Andrew.

Thanks for the response.  Are there plans to implement the streaming option
to the EPT reader?  If so, do you have a feel for the timeline?  6 months?
a year?  I know it is hard to estimate, but I was just curious.

thanks.
---------------------------
Matthew Beckley
Data Engineer
UNAVCO/OpenTopography
beckley at unavco.org
303-381-7487


On Mon, Apr 1, 2019 at 7:33 AM Andrew Bell <andrew.bell.ia at gmail.com> wrote:

> This isn't surprising.  150 million points at even 30 bytes per point (and
> it's more than 30 bytes per point) is 4.5 gig.  Until the EPT reader
> supports stream mode, you're not going to be able to get around this
> limitation with PDAL.  There are no EPT options you can set that will
> change this.
>
> On Mon, Apr 1, 2019 at 9:26 AM Matt Beckley <beckley at unavco.org> wrote:
>
>> Is there a way to optimize docker or PDAL to be able to run larger
>> queries on ept files?  Or Is there some info about memory management, or
>> some settings I should be using to facilitate larger queries?  I have been
>> trying to run a test on the JSON pipeline below, and I am only able to get
>> it to run successfully if I set my docker memory to over 6GB....anything
>> less than 6GB will fail.  When successfully run, the resulting LAZ file is
>> only about 150 million points - so, not huge by lidar standards.    I've
>> tried increasing the "threads" keyword to the ept reader, but that didn't
>> seem to help.
>>
>> {
>>     "pipeline": [{
>>         "type": "readers.ept",
>>         "filename": "
>> https://s3-us-west-2.amazonaws.com/usgs-lidar-public/USGS_LPC_HI_Oahu_2012_LAS_2015
>> ",
>>         "bounds": "([-17606581.532235783,
>> -17598784.955350697],[2441398.6834285296,2448889.512200477])"
>>     },
>>     {
>>         "type" : "writers.las",
>>         "filename": "HI_LargeOutput_Local.laz"
>>     }]}
>>
>> Any ideas would be appreciated.
>>
>> Thanks!
>>
>> matt.
>> ---------------------------
>> Matthew Beckley
>> Data Engineer
>> UNAVCO/OpenTopography
>> beckley at unavco.org
>> 303-381-7487
>> _______________________________________________
>> pdal mailing list
>> pdal at lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/pdal
>
>
>
> --
> Andrew Bell
> andrew.bell.ia at gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pdal/attachments/20190401/5bd377d8/attachment.html>