[pdal] How to handle las files that don't fit in memory

[Tom van Tilburg]

Hi Howard, this is what I was doing up to now (split in 9 pieces on the
base of extent with lastools and then pipeline every piece via the chipper
into pgpointcloud). I was hoping it could be done with only pdal and in one
pipeline. I will stick to the lastools approach then.

Blame the government for the 3gb laz files, for some reason they want to
stick to the common national gridding system of 25 sqr km(!) each.  And no,
it doesn't commonly work. I'm also curious on how others deal with this.
If you, or anyone, want to try themselves, this is where you can get them:
https://www.pdok.nl/nl/ahn3-downloads
(select the desired gridcel on the map)

On Tue, 22 Dec 2015 at 15:50 Howard Butler <howard at hobu.co> wrote:

> It would seem that this operation needs to be done in two stages
>
> 1) Naively break the 3gb LAZ file into something smaller based on linear
> point count (or some other streamable operation -- if you have LAStools in
> reach, use it).
> 2) Chip/load the smaller pieces into pgpointcloud.
>
> The good thing about loading the data in pgpointcloud is in the end it
> won't matter. Your database will provide the spatial selectivity you need
> after the data are loaded.
>
> Howard
>
> PS Who's workflow makes 3gb LAZ files and does that commonly work?
>
> > On Dec 22, 2015, at 1:25 AM, Tom van Tilburg <tom.van.tilburg at gmail.com>
> wrote:
> >
> > Hi Andrew,
> >
> > Reading my initial email it occured to me I should have been more
> verbose on my problem. I wrongly took the splitter filter as an example,
> should have been chipper. Sorry for that.
> >
> > What I am trying to reach is to load a bunch of ~3Gb laz file into
> pgpointcloud and therefore they need to be chipped in chunks of ~400
> points. I know the chipper wouldn't do streaming so cutting the laz files
> into smaller blocks (like cropping) before starting the chipping would make
> sense I thought.
> >
> > I will look into increasing the swap but I don't have a good feeling
> about it since we already have 30 Gb of swap and a previous loading attempt
> with a similar dataset but with smaller laz files already took us weeks
> (running parallel) to chip and load.
> >
> > Thanks for your effort.
> >
> > On Mon, 21 Dec 2015 at 23:33 Andrew Bell <andrew.bell.ia at gmail.com>
> wrote:
> > On Mon, Dec 21, 2015 at 2:10 PM, Tom van Tilburg <
> tom.van.tilburg at gmail.com> wrote:
> > Hi Andrew,
> >
> > Increasing SWAP is possible but wouldn't this seriously degrade the
> performance?
> >
> > It depends what you're comparing to.  It's really hard to say without
> giving it a go.  Unless you're short disk space, it seems like a relatively
> trivial exercise to add the swap and see.
> >
> > I have about 30 gigs of RAM and it already takes ages to load everything
> in there.
> >
> > Make sure you're using a release version of PDAL.  It's significantly
> faster than the debug version.  Without know a lot about your system, I
> don't have much to offer.  However, if your time is worth anything, memory
> is cheap.
> >
> > Streaming would definitely be a way to go, as it works fine with
> lastools for me. I'll look into it. Chopping the data into smaller spatial
> blocks would be my first approach here.
> >
> > Any estimate on when streaming will get into the api?
> >
> > No, sorry.  Streaming won't really solve your problem exactly anyway,
> because you can't stream with the splitter.  If you've already split your
> data into chunks spatially, you've already done the work of the splitter,
> so I'm not sure what to advise.
> >
> > Is it necessary to break up your data?  Perhaps there's another solution
> that doesn't involve tiling?
> >
> > Best,
> >
> > --
> > Andrew Bell
> > andrew.bell.ia at gmail.com
> > _______________________________________________
> > pdal mailing list
> > pdal at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/pdal
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pdal/attachments/20151222/638e2f06/attachment-0001.html>