[pdal] Memory management with PDAL/Docker

[adam steer]

Hi Matt

I see you have a pretty big area (are those US feet or metres?) specified
down to nanometre (nanofoot?) precision - can you divide-and-conquer? this
would be my approach

Cheers

Adam

On Tue, 2 Apr 2019 at 05:15, Matt Beckley <beckley at unavco.org> wrote:

> Hi Andrew.
>
> Thanks for the response.  Are there plans to implement the streaming
> option to the EPT reader?  If so, do you have a feel for the timeline?  6
> months?  a year?  I know it is hard to estimate, but I was just curious.
>
> thanks.
> ---------------------------
> Matthew Beckley
> Data Engineer
> UNAVCO/OpenTopography
> beckley at unavco.org
> 303-381-7487
>
>
> On Mon, Apr 1, 2019 at 7:33 AM Andrew Bell <andrew.bell.ia at gmail.com>
> wrote:
>
>> This isn't surprising.  150 million points at even 30 bytes per point
>> (and it's more than 30 bytes per point) is 4.5 gig.  Until the EPT reader
>> supports stream mode, you're not going to be able to get around this
>> limitation with PDAL.  There are no EPT options you can set that will
>> change this.
>>
>> On Mon, Apr 1, 2019 at 9:26 AM Matt Beckley <beckley at unavco.org> wrote:
>>
>>> Is there a way to optimize docker or PDAL to be able to run larger
>>> queries on ept files?  Or Is there some info about memory management, or
>>> some settings I should be using to facilitate larger queries?  I have been
>>> trying to run a test on the JSON pipeline below, and I am only able to get
>>> it to run successfully if I set my docker memory to over 6GB....anything
>>> less than 6GB will fail.  When successfully run, the resulting LAZ file is
>>> only about 150 million points - so, not huge by lidar standards.    I've
>>> tried increasing the "threads" keyword to the ept reader, but that didn't
>>> seem to help.
>>>
>>> {
>>>     "pipeline": [{
>>>         "type": "readers.ept",
>>>         "filename": "
>>> https://s3-us-west-2.amazonaws.com/usgs-lidar-public/USGS_LPC_HI_Oahu_2012_LAS_2015
>>> ",
>>>         "bounds": "([-17606581.532235783,
>>> -17598784.955350697],[2441398.6834285296,2448889.512200477])"
>>>     },
>>>     {
>>>         "type" : "writers.las",
>>>         "filename": "HI_LargeOutput_Local.laz"
>>>     }]}
>>>
>>> Any ideas would be appreciated.
>>>
>>> Thanks!
>>>
>>> matt.
>>> ---------------------------
>>> Matthew Beckley
>>> Data Engineer
>>> UNAVCO/OpenTopography
>>> beckley at unavco.org
>>> 303-381-7487
>>> _______________________________________________
>>> pdal mailing list
>>> pdal at lists.osgeo.org
>>> https://lists.osgeo.org/mailman/listinfo/pdal
>>
>>
>>
>> --
>> Andrew Bell
>> andrew.bell.ia at gmail.com
>>
> _______________________________________________
> pdal mailing list
> pdal at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/pdal



-- 
Dr. Adam Steer
http://spatialised.net
https://www.researchgate.net/profile/Adam_Steer
http://au.linkedin.com/in/adamsteer
http://orcid.org/0000-0003-0046-7236
+61 427 091 712 ::  @adamdsteer

I won’t turn up in a suit: http://www.spatialised.net/business-penguins/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pdal/attachments/20190402/94f0392c/attachment.html>