[pdal] How to handle las files that don't fit in memory

[Tom van Tilburg]

Albert, indeed increasing swap feels like making things worse...
Here is the specs:

>free -mh
             total       used       free     shared    buffers     cached
Mem:           31G       9.2G        22G       1.4G       385M       7.6G
-/+ buffers/cache:       1.2G        30G
Swap:          31G       224M        31G

>lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                8
On-line CPU(s) list:   0-7
Thread(s) per core:    2
Core(s) per socket:    4
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 60
Stepping:              3
CPU MHz:               800.000



On Tue, 22 Dec 2015 at 08:57 Albert Godfrind <albert.godfrind at oracle.com>
wrote:

> Tom
>
> Just checking. Can you clarify the hardware config you are using ? Does it
> have 30GB of physical RAM ? Or does it have less than that, and you have it
> configured for 30GB virtual memory ? Increasing virtual memory (swap space)
> if you do not have the memory capacity to support your process will not
> work (actually it is likely to have the opposite effect).
>
> How many cores do you have ?
>
> Albert
>
> When
>
> --
> Albert Godfrind
> +33 6 09 97 27 23
> Sent from my iPhone
>
> On 22 déc. 2015, at 08:25, Tom van Tilburg <tom.van.tilburg at gmail.com>
> wrote:
>
> Hi Andrew,
>
> Reading my initial email it occured to me I should have been more verbose
> on my problem. I wrongly took the splitter filter as an example, should
> have been chipper. Sorry for that.
>
> What I am trying to reach is to load a bunch of ~3Gb laz file into
> pgpointcloud and therefore they need to be chipped in chunks of ~400
> points. I know the chipper wouldn't do streaming so cutting the laz files
> into smaller blocks (like cropping) before starting the chipping would make
> sense I thought.
>
> I will look into increasing the swap but I don't have a good feeling about
> it since we already have 30 Gb of swap and a previous loading attempt
> with a similar dataset but with smaller laz files already took us weeks
> (running parallel) to chip and load.
>
> Thanks for your effort.
>
> On Mon, 21 Dec 2015 at 23:33 Andrew Bell <andrew.bell.ia at gmail.com> wrote:
>
>> On Mon, Dec 21, 2015 at 2:10 PM, Tom van Tilburg <
>> tom.van.tilburg at gmail.com> wrote:
>>
>>> Hi Andrew,
>>>
>>> Increasing SWAP is possible but wouldn't this seriously degrade the
>>> performance?
>>>
>>
>> It depends what you're comparing to.  It's really hard to say without
>> giving it a go.  Unless you're short disk space, it seems like a relatively
>> trivial exercise to add the swap and see.
>>
>>
>>> I have about 30 gigs of RAM and it already takes ages to load everything
>>> in there.
>>>
>>
>> Make sure you're using a release version of PDAL.  It's significantly
>> faster than the debug version.  Without know a lot about your system, I
>> don't have much to offer.  However, if your time is worth anything, memory
>> is cheap.
>>
>> Streaming would definitely be a way to go, as it works fine with lastools
>>> for me. I'll look into it. Chopping the data into smaller spatial blocks
>>> would be my first approach here.
>>>
>>
>>
>> Any estimate on when streaming will get into the api?
>>>
>>
>> No, sorry.  Streaming won't really solve your problem exactly anyway,
>> because you can't stream with the splitter.  If you've already split your
>> data into chunks spatially, you've already done the work of the splitter,
>> so I'm not sure what to advise.
>>
>> Is it necessary to break up your data?  Perhaps there's another solution
>> that doesn't involve tiling?
>>
>> Best,
>>
>> --
>> Andrew Bell
>> andrew.bell.ia at gmail.com
>>
> _______________________________________________
> pdal mailing list
> pdal at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/pdal
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pdal/attachments/20151222/def42b1d/attachment-0001.html>