[pdal] How to handle las files that don't fit in memory

[Albert Godfrind]

Tom

Just checking. Can you clarify the hardware config you are using ? Does it have 30GB of physical RAM ? Or does it have less than that, and you have it configured for 30GB virtual memory ? Increasing virtual memory (swap space) if you do not have the memory capacity to support your process will not work (actually it is likely to have the opposite effect). 

How many cores do you have ?

Albert

When 

--
Albert Godfrind
+33 6 09 97 27 23
Sent from my iPhone

> On 22 déc. 2015, at 08:25, Tom van Tilburg <tom.van.tilburg at gmail.com> wrote:
> 
> Hi Andrew,
> 
> Reading my initial email it occured to me I should have been more verbose on my problem. I wrongly took the splitter filter as an example, should have been chipper. Sorry for that.
> 
> What I am trying to reach is to load a bunch of ~3Gb laz file into pgpointcloud and therefore they need to be chipped in chunks of ~400 points. I know the chipper wouldn't do streaming so cutting the laz files into smaller blocks (like cropping) before starting the chipping would make sense I thought. 
> 
> I will look into increasing the swap but I don't have a good feeling about it since we already have 30 Gb of swap and a previous loading attempt with a similar dataset but with smaller laz files already took us weeks (running parallel) to chip and load.
> 
> Thanks for your effort.
> 
>> On Mon, 21 Dec 2015 at 23:33 Andrew Bell <andrew.bell.ia at gmail.com> wrote:
>>> On Mon, Dec 21, 2015 at 2:10 PM, Tom van Tilburg <tom.van.tilburg at gmail.com> wrote:
>>> Hi Andrew,
>>> 
>>> Increasing SWAP is possible but wouldn't this seriously degrade the performance?
>> 
>> It depends what you're comparing to.  It's really hard to say without giving it a go.  Unless you're short disk space, it seems like a relatively trivial exercise to add the swap and see.
>>  
>>> I have about 30 gigs of RAM and it already takes ages to load everything in there.
>> 
>> Make sure you're using a release version of PDAL.  It's significantly faster than the debug version.  Without know a lot about your system, I don't have much to offer.  However, if your time is worth anything, memory is cheap.
>> 
>>> Streaming would definitely be a way to go, as it works fine with lastools for me. I'll look into it. Chopping the data into smaller spatial blocks would be my first approach here.
>>>  
>>> Any estimate on when streaming will get into the api?
>> 
>> No, sorry.  Streaming won't really solve your problem exactly anyway, because you can't stream with the splitter.  If you've already split your data into chunks spatially, you've already done the work of the splitter, so I'm not sure what to advise.
>> 
>> Is it necessary to break up your data?  Perhaps there's another solution that doesn't involve tiling?
>> 
>> Best,
>> 
>> -- 
>> Andrew Bell
>> andrew.bell.ia at gmail.com
> _______________________________________________
> pdal mailing list
> pdal at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/pdal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pdal/attachments/20151222/28fb1800/attachment.html>