<div dir="ltr"><div>Albert, indeed increasing swap feels like making things worse...</div><div><span style="line-height:1.5">Here is the specs:</span><br></div><div><br></div><div>>free -mh</div><div> total used free shared buffers cached</div><div>Mem: 31G 9.2G 22G 1.4G 385M 7.6G</div><div>-/+ buffers/cache: 1.2G 30G</div><div>Swap: 31G 224M 31G</div><div><br></div><div><div>>lscpu</div><div>Architecture: x86_64</div><div>CPU op-mode(s): 32-bit, 64-bit</div><div>Byte Order: Little Endian</div><div>CPU(s): 8</div><div>On-line CPU(s) list: 0-7</div><div>Thread(s) per core: 2</div><div>Core(s) per socket: 4</div><div>Socket(s): 1</div><div>NUMA node(s): 1</div><div>Vendor ID: GenuineIntel</div><div>CPU family: 6</div><div>Model: 60</div><div>Stepping: 3</div><div>CPU MHz: 800.000</div><div><br></div></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, 22 Dec 2015 at 08:57 Albert Godfrind <<a href="mailto:albert.godfrind@oracle.com">albert.godfrind@oracle.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto"><div>Tom</div><div><br></div><div>Just checking. Can you clarify the hardware config you are using ? Does it have 30GB of physical RAM ? Or does it have less than that, and you have it configured for 30GB virtual memory ? Increasing virtual memory (swap space) if you do not have the memory capacity to support your process will not work (actually it is likely to have the opposite effect). </div><div><br></div><div>How many cores do you have ?</div><div><br></div><div>Albert</div><div><br></div><div>When <br><br>--<div>Albert Godfrind<div>+33 6 09 97 27 23<br><div>Sent from my iPhone</div></div></div></div></div><div dir="auto"><div><br>On 22 déc. 2015, at 08:25, Tom van Tilburg <<a href="mailto:tom.van.tilburg@gmail.com" target="_blank">tom.van.tilburg@gmail.com</a>> wrote:<br><br></div><blockquote type="cite"><div><div dir="ltr">Hi Andrew,<div><br></div><div>Reading my initial email it occured to me I should have been more verbose on my problem. I wrongly took the splitter filter as an example, should have been chipper. Sorry for that.</div><div><br></div><div>What I am trying to reach is to load a bunch of ~3Gb laz file into pgpointcloud and therefore they need to be chipped in chunks of ~400 points. I know the chipper wouldn't do streaming so cutting the laz files into smaller blocks (like cropping) before starting the chipping would make sense I thought. </div><div><br></div><div>I will look into increasing the swap but I don't have a good feeling about it since we already have 30 Gb of swap and a<span style="line-height:1.5"> previous loading attempt with a similar dataset but with smaller laz files already took us weeks (running parallel) to chip and load.</span></div><div><span style="line-height:1.5"><br></span></div><div><span style="line-height:1.5">Thanks for your effort.</span></div></div><br><div class="gmail_quote"><div dir="ltr">On Mon, 21 Dec 2015 at 23:33 Andrew Bell <<a href="mailto:andrew.bell.ia@gmail.com" target="_blank">andrew.bell.ia@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Mon, Dec 21, 2015 at 2:10 PM, Tom van Tilburg <span dir="ltr"><<a href="mailto:tom.van.tilburg@gmail.com" target="_blank">tom.van.tilburg@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Andrew,<div><br></div><div><span style="line-height:1.5">Increasing SWAP is possible but wouldn't this seriously degrade the performance?</span></div></div></blockquote><div><br></div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>It depends what you're comparing to. It's really hard to say without giving it a go. Unless you're short disk space, it seems like a relatively trivial exercise to add the swap and see.</div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><span style="line-height:1.5"> I have about 30 gigs of RAM and it already takes ages to load everything in there.</span></div></div></blockquote><div><br></div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>Make sure you're using a release version of PDAL. It's significantly faster than the debug version. Without know a lot about your system, I don't have much to offer. However, if your time is worth anything, memory is cheap.</div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Streaming would definitely be a way to go, as it works fine with lastools for me. I'll look into it. Chopping the data into smaller spatial blocks would be my first approach here.</div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"> </blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Any estimate on when streaming will get into the api?</div></div></blockquote><div><br></div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>No, sorry. Streaming won't really solve your problem exactly anyway, because you can't stream with the splitter. If you've already split your data into chunks spatially, you've already done the work of the splitter, so I'm not sure what to advise.</div><div><br></div><div>Is it necessary to break up your data? Perhaps there's another solution that doesn't involve tiling?</div><div><br></div><div>Best,</div><div><br></div><div>-- <br></div></div></div></div><div dir="ltr"><div class="gmail_extra"><div>Andrew Bell<br><a href="mailto:andrew.bell.ia@gmail.com" target="_blank">andrew.bell.ia@gmail.com</a></div>
</div></div></blockquote></div>
</div></blockquote></div><div dir="auto"><blockquote type="cite"><div><span>_______________________________________________</span><br><span>pdal mailing list</span><br><span><a href="mailto:pdal@lists.osgeo.org" target="_blank">pdal@lists.osgeo.org</a></span><br><span><a href="http://lists.osgeo.org/mailman/listinfo/pdal" target="_blank">http://lists.osgeo.org/mailman/listinfo/pdal</a></span></div></blockquote></div></blockquote></div>