[pdal] How to handle las files that don't fit in memory

[Tom van Tilburg]

Hi Andrew,

Increasing SWAP is possible but wouldn't this seriously degrade the
performance? I have about 30 gigs of RAM and it already takes ages to load
everything in there.

Streaming would definitely be a way to go, as it works fine with lastools
for me. I'll look into it. Chopping the data into smaller spatial blocks
would be my first approach here.
Any estimate on when streaming will get into the api?

Best, Tom


On Wed, 16 Dec 2015 at 23:54 Andrew Bell <andrew.bell.ia at gmail.com> wrote:

> On Wed, Dec 16, 2015 at 4:38 PM, Tom van Tilburg <
> tom.van.tilburg at gmail.com> wrote:
>
>> I have a las-zip file of 3 gigs and try to run a pdal splitter filter on
>> it. PDAL will try to load the entire file into memory (30 GB) which doesn't
>> fit and segfaults upon that.
>> Would there be an option to split the large las file first into smaller
>> blocks based on extent by simply running over all points and writing them
>> directly to output instead of first caching it?
>>
>
> When you say it doesn't fit, I'm not sure what you're saying.  Are you
> saying that you don't have sufficient virtual memory to deal with the
> dataset?  If that's the case, the easiest solution is to increase the
> virtual memory on your system by adding more swap space.  You can Google
> specific information on your system on how to do this.
>
> I tried the crop filter as well but it also seems to load all points into
>> memory first.
>> It is possible with lastools but I would prefer to keep my process within
>> a pdal-based script.
>>
>
> We have just added "streaming" to the repository which works with some
> data sources and filters (LAS and the crop filter are included), however,
> you'd have to write C++ code at the current time to take advantage of it.
> This would let you, say, divide your dataset into two files that could then
> be run through the splitter individually (the splitter doesn't support
> streaming).  If you're interested in tackling this, take a look at
> StreamingTest.cpp, or alternatively many of the tests for filters now
> contain examples to test the streaming capabilities.  You can also write
> back and I'd be happy to help.
>
> Still, the easiest thing is to add some swap and be done unless you're
> running into some other limitation that I don't understand.
>
> --
> Andrew Bell
> andrew.bell.ia at gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pdal/attachments/20151221/86665adb/attachment.html>