<HTML>
<HEAD>
<TITLE>Re: [Liblas-devel] problems with lasblock and big datasets</TITLE>
</HEAD>
<BODY>
<FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'>Howard,<BR>
<BR>
Would it be easy to process in chunks when filesize exceeds memory? Essentially do internally what you have shown externally?<BR>
<BR>
Mike<BR>
<BR>
<BR>
On 9/10/10 3:06 PM, "Howard Butler" <<a href="hobu.inc@gmail.com">hobu.inc@gmail.com</a>> wrote:<BR>
<BR>
</SPAN></FONT><BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><BR>
<BR>
On Sep 10, 2010, at 2:47 AM, Hugo Ledoux wrote:<BR>
<BR>
> Greetings,<BR>
><BR>
> While testing lasblock yesterday (code checked out a few days ago and compiled under Linux 4.4) I ran into a problem with a big dataset:<BR>
><BR>
> hugo@TUDLEDOUX:~/data/las/ahn2_original/gefilterd$ time lasblock -c 10000 -p 3 bigone.las<BR>
> terminate called after throwing an instance of 'std::length_error'<BR>
> what(): vector::reserve<BR>
> Aborted<BR>
><BR>
> real 0m5.066s<BR>
> user 0m4.620s<BR>
> sys 0m0.432s<BR>
><BR>
><BR>
> Reading how lasblock works, I did expect it to struggle with very large dataset, but here it crashed after 4s. The dataset "bigone.las" has ~280 million points, and is 5.3GB. With smaller datasets (~20M points) it works flawlessly.<BR>
><BR>
> Is it simply that allocating 2 arrays of 280M is too much and then it aborts?<BR>
<BR>
<BR>
Yep. It's trying to reserve 3*280m though, and depending on the underlying stl implementation of std::vector::reserve, actually trying to allocate it. <BR>
<BR>
You're the first to hit the limit, although we've been running it on 64bit hardware with substantial RAM without too much trouble. I can't offer much in the way of work arounds other than the following at this point:<BR>
<BR>
las2las2 --split 1024 bigone.las<BR>
lasblock bigone_1.las<BR>
lasblock bigone_2.las<BR>
...<BR>
<BR>
In the interim, I'll see if there's any simple workarounds for needing to allocate so much memory for the sort(s).<BR>
<BR>
Howard<BR>
<BR>
<BR>
_______________________________________________<BR>
Liblas-devel mailing list<BR>
<a href="Liblas-devel@lists.osgeo.org">Liblas-devel@lists.osgeo.org</a><BR>
<a href="http://lists.osgeo.org/mailman/listinfo/liblas-devel">http://lists.osgeo.org/mailman/listinfo/liblas-devel</a><BR>
<BR>
</SPAN></FONT></BLOCKQUOTE>
</BODY>
</HTML>