[pdal] How to handle las files that don't fit in memory

[Andrew Bell]

On Wed, Dec 16, 2015 at 4:38 PM, Tom van Tilburg <tom.van.tilburg at gmail.com>
wrote:

> I have a las-zip file of 3 gigs and try to run a pdal splitter filter on
> it. PDAL will try to load the entire file into memory (30 GB) which doesn't
> fit and segfaults upon that.
> Would there be an option to split the large las file first into smaller
> blocks based on extent by simply running over all points and writing them
> directly to output instead of first caching it?
>

When you say it doesn't fit, I'm not sure what you're saying.  Are you
saying that you don't have sufficient virtual memory to deal with the
dataset?  If that's the case, the easiest solution is to increase the
virtual memory on your system by adding more swap space.  You can Google
specific information on your system on how to do this.

I tried the crop filter as well but it also seems to load all points into
> memory first.
> It is possible with lastools but I would prefer to keep my process within
> a pdal-based script.
>

We have just added "streaming" to the repository which works with some data
sources and filters (LAS and the crop filter are included), however, you'd
have to write C++ code at the current time to take advantage of it.  This
would let you, say, divide your dataset into two files that could then be
run through the splitter individually (the splitter doesn't support
streaming).  If you're interested in tackling this, take a look at
StreamingTest.cpp, or alternatively many of the tests for filters now
contain examples to test the streaming capabilities.  You can also write
back and I'd be happy to help.

Still, the easiest thing is to add some swap and be done unless you're
running into some other limitation that I don't understand.

-- 
Andrew Bell
andrew.bell.ia at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pdal/attachments/20151216/56738759/attachment.html>