[pdal] How to handle las files that don't fit in memory

[Howard Butler]

It would seem that this operation needs to be done in two stages

1) Naively break the 3gb LAZ file into something smaller based on linear point count (or some other streamable operation -- if you have LAStools in reach, use it).
2) Chip/load the smaller pieces into pgpointcloud.

The good thing about loading the data in pgpointcloud is in the end it won't matter. Your database will provide the spatial selectivity you need after the data are loaded.

Howard

PS Who's workflow makes 3gb LAZ files and does that commonly work?

> On Dec 22, 2015, at 1:25 AM, Tom van Tilburg <tom.van.tilburg at gmail.com> wrote:
> 
> Hi Andrew,
> 
> Reading my initial email it occured to me I should have been more verbose on my problem. I wrongly took the splitter filter as an example, should have been chipper. Sorry for that.
> 
> What I am trying to reach is to load a bunch of ~3Gb laz file into pgpointcloud and therefore they need to be chipped in chunks of ~400 points. I know the chipper wouldn't do streaming so cutting the laz files into smaller blocks (like cropping) before starting the chipping would make sense I thought.
> 
> I will look into increasing the swap but I don't have a good feeling about it since we already have 30 Gb of swap and a previous loading attempt with a similar dataset but with smaller laz files already took us weeks (running parallel) to chip and load.
> 
> Thanks for your effort.
> 
> On Mon, 21 Dec 2015 at 23:33 Andrew Bell <andrew.bell.ia at gmail.com> wrote:
> On Mon, Dec 21, 2015 at 2:10 PM, Tom van Tilburg <tom.van.tilburg at gmail.com> wrote:
> Hi Andrew,
> 
> Increasing SWAP is possible but wouldn't this seriously degrade the performance?
> 
> It depends what you're comparing to.  It's really hard to say without giving it a go.  Unless you're short disk space, it seems like a relatively trivial exercise to add the swap and see.
> 
> I have about 30 gigs of RAM and it already takes ages to load everything in there.
> 
> Make sure you're using a release version of PDAL.  It's significantly faster than the debug version.  Without know a lot about your system, I don't have much to offer.  However, if your time is worth anything, memory is cheap.
> 
> Streaming would definitely be a way to go, as it works fine with lastools for me. I'll look into it. Chopping the data into smaller spatial blocks would be my first approach here.
> 
> Any estimate on when streaming will get into the api?
> 
> No, sorry.  Streaming won't really solve your problem exactly anyway, because you can't stream with the splitter.  If you've already split your data into chunks spatially, you've already done the work of the splitter, so I'm not sure what to advise.
> 
> Is it necessary to break up your data?  Perhaps there's another solution that doesn't involve tiling?
> 
> Best,
> 
> --
> Andrew Bell
> andrew.bell.ia at gmail.com
> _______________________________________________
> pdal mailing list
> pdal at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/pdal

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.osgeo.org/pipermail/pdal/attachments/20151222/e13f070d/attachment.sig>