[pdal] How to handle las files that don't fit in memory

[Tom van Tilburg]

Hi Andrew,

Reading my initial email it occured to me I should have been more verbose
on my problem. I wrongly took the splitter filter as an example, should
have been chipper. Sorry for that.

What I am trying to reach is to load a bunch of ~3Gb laz file into
pgpointcloud and therefore they need to be chipped in chunks of ~400
points. I know the chipper wouldn't do streaming so cutting the laz files
into smaller blocks (like cropping) before starting the chipping would make
sense I thought.

I will look into increasing the swap but I don't have a good feeling about
it since we already have 30 Gb of swap and a previous loading attempt with
a similar dataset but with smaller laz files already took us weeks (running
parallel) to chip and load.

Thanks for your effort.

On Mon, 21 Dec 2015 at 23:33 Andrew Bell <andrew.bell.ia at gmail.com> wrote:

> On Mon, Dec 21, 2015 at 2:10 PM, Tom van Tilburg <
> tom.van.tilburg at gmail.com> wrote:
>
>> Hi Andrew,
>>
>> Increasing SWAP is possible but wouldn't this seriously degrade the
>> performance?
>>
>
> It depends what you're comparing to.  It's really hard to say without
> giving it a go.  Unless you're short disk space, it seems like a relatively
> trivial exercise to add the swap and see.
>
>
>> I have about 30 gigs of RAM and it already takes ages to load everything
>> in there.
>>
>
> Make sure you're using a release version of PDAL.  It's significantly
> faster than the debug version.  Without know a lot about your system, I
> don't have much to offer.  However, if your time is worth anything, memory
> is cheap.
>
> Streaming would definitely be a way to go, as it works fine with lastools
>> for me. I'll look into it. Chopping the data into smaller spatial blocks
>> would be my first approach here.
>>
>
>
> Any estimate on when streaming will get into the api?
>>
>
> No, sorry.  Streaming won't really solve your problem exactly anyway,
> because you can't stream with the splitter.  If you've already split your
> data into chunks spatially, you've already done the work of the splitter,
> so I'm not sure what to advise.
>
> Is it necessary to break up your data?  Perhaps there's another solution
> that doesn't involve tiling?
>
> Best,
>
> --
> Andrew Bell
> andrew.bell.ia at gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pdal/attachments/20151222/b7c9eec5/attachment.html>