[pdal] Memory management with PDAL/Docker
Matt Beckley
beckley at unavco.org
Mon Apr 1 11:15:06 PDT 2019
Hi Andrew.
Thanks for the response. Are there plans to implement the streaming option
to the EPT reader? If so, do you have a feel for the timeline? 6 months?
a year? I know it is hard to estimate, but I was just curious.
thanks.
---------------------------
Matthew Beckley
Data Engineer
UNAVCO/OpenTopography
beckley at unavco.org
303-381-7487
On Mon, Apr 1, 2019 at 7:33 AM Andrew Bell <andrew.bell.ia at gmail.com> wrote:
> This isn't surprising. 150 million points at even 30 bytes per point (and
> it's more than 30 bytes per point) is 4.5 gig. Until the EPT reader
> supports stream mode, you're not going to be able to get around this
> limitation with PDAL. There are no EPT options you can set that will
> change this.
>
> On Mon, Apr 1, 2019 at 9:26 AM Matt Beckley <beckley at unavco.org> wrote:
>
>> Is there a way to optimize docker or PDAL to be able to run larger
>> queries on ept files? Or Is there some info about memory management, or
>> some settings I should be using to facilitate larger queries? I have been
>> trying to run a test on the JSON pipeline below, and I am only able to get
>> it to run successfully if I set my docker memory to over 6GB....anything
>> less than 6GB will fail. When successfully run, the resulting LAZ file is
>> only about 150 million points - so, not huge by lidar standards. I've
>> tried increasing the "threads" keyword to the ept reader, but that didn't
>> seem to help.
>>
>> {
>> "pipeline": [{
>> "type": "readers.ept",
>> "filename": "
>> https://s3-us-west-2.amazonaws.com/usgs-lidar-public/USGS_LPC_HI_Oahu_2012_LAS_2015
>> ",
>> "bounds": "([-17606581.532235783,
>> -17598784.955350697],[2441398.6834285296,2448889.512200477])"
>> },
>> {
>> "type" : "writers.las",
>> "filename": "HI_LargeOutput_Local.laz"
>> }]}
>>
>> Any ideas would be appreciated.
>>
>> Thanks!
>>
>> matt.
>> ---------------------------
>> Matthew Beckley
>> Data Engineer
>> UNAVCO/OpenTopography
>> beckley at unavco.org
>> 303-381-7487
>> _______________________________________________
>> pdal mailing list
>> pdal at lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/pdal
>
>
>
> --
> Andrew Bell
> andrew.bell.ia at gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pdal/attachments/20190401/5bd377d8/attachment.html>
More information about the pdal
mailing list