[Liblas-devel] problems with lasblock and big datasets

[Michael Smith]

Howard,

Would it be easy to process in chunks when filesize exceeds memory?
Essentially do internally what you have shown externally?

Mike


On 9/10/10 3:06 PM, "Howard Butler" <hobu.inc at gmail.com> wrote:

> 
> 
> On Sep 10, 2010, at 2:47 AM, Hugo Ledoux wrote:
> 
>> > Greetings,
>> >
>> > While testing lasblock yesterday (code checked out a few days ago and
>> compiled under Linux 4.4) I ran into a problem with a big dataset:
>> >
>> > hugo at TUDLEDOUX:~/data/las/ahn2_original/gefilterd$ time lasblock -c 10000
>> -p 3 bigone.las
>> > terminate called after throwing an instance of 'std::length_error'
>> >  what():  vector::reserve
>> > Aborted
>> >
>> > real    0m5.066s
>> > user    0m4.620s
>> > sys     0m0.432s
>> >
>> >
>> > Reading how lasblock works, I did expect it to struggle with very large
>> dataset, but here it crashed after 4s. The dataset "bigone.las" has ~280
>> million points, and is 5.3GB. With smaller datasets (~20M points) it works
>> flawlessly.
>> >
>> > Is it simply that allocating 2 arrays of 280M is too much and then it
>> aborts?
> 
> 
> Yep.  It's trying to reserve 3*280m though, and depending on the underlying
> stl implementation of std::vector::reserve, actually trying to allocate it.
> 
> You're the first to hit the limit, although we've been running it on 64bit
> hardware with substantial RAM without too much trouble. I can't offer much in
> the way of work arounds other than the following at this point:
> 
> las2las2 --split 1024 bigone.las
> lasblock bigone_1.las
> lasblock bigone_2.las
> ...
> 
> In the interim, I'll see if there's any simple workarounds for needing to
> allocate so much memory for the sort(s).
> 
> Howard
> 
> 
> _______________________________________________
> Liblas-devel mailing list
> Liblas-devel at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/liblas-devel
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/liblas-devel/attachments/20100910/7b8877fc/attachment.html