[pdal] Streaming versus non-streaming LAS read perfromance

[Andrew Bell]

Hi,

I don't know what your problem might be.  In the past I have done timing of
stream/standard mode and they've been comparable.  I don't think we've
changed anything recently that would impact LAS reading performance unless
there's more to your code than you've shown.  I don't know what memory
would be allocated in 3MB chunks.  Generally, PDAL doesn't allocate heap
memory during the read process.

A couple of thoughts:

1) Rather than call prepare() on the stage, you can call preview().  This
returns a structure that contains the point count.  I don't think this
should have any impact on performance.
2) In order to debug, I would eliminate your processing code, whatever that
might be, just to simplify.  There are suggestions in the non-streaming
code that you are threading things in a way that we simply can't see or
analyze.

If you can pin things down better, please let me know and I can take a look.


On Thu, May 16, 2019 at 3:58 PM KB-Vis Technologies Pvt. Ltd. <3d at kbvis.com>
wrote:

> Greetings,
>
> I have a huge performance issue with reading LAS points on large files.
>
> I initially implemented a non-streaming read using PointViewSet as in the
> examples, and this worked fine. However on larger LAS files e.g. a 1GB file
> with 150 million points, the memory usage would go up to 15GB, and the load
> would take around 3-5 minutes.
>
> Then, I switched to a streaming read using FixedPointTable and
> StreamCallbackFilter. This uses much less memory but is VERY slow. The 150
> million points take 15+ minutes to load..The issue seems to be that memory
> is being allocated in increments of 3MB, and this takes forever. tried
> various values for the FixedPointTable size, but this seems to have no
> effect.
>
> I would prefer to use the streaming approach to do fast low-res partial
> reads etc. and to keep the memory usage down, if only I could improve the
> loading time.
>
> I am using PDAL 1.7.2. I was wondering if I am doing something very wrong,
> or whether I should just update to 1.9
>
> Any advice would be hugely appreciated!
>
> many thanks,
>
> Karthik Bala
>
>
>
> My code:
>
> *Original Non-Streaming reader (FAST but memory-intensive)*
> pdal::Option las_opt("filename", sFilenames[f]);
> pdal::Options las_opts;
> las_opts.add(las_opt);
> LasReader *sLasReader = new LasReader();
> sLasReader->setOptions(las_opts);
> PointTable *sTable = new PointTable();
> sLasReader->prepare(*sTable);
> PointViewSet sPointViewSet = sLasReader->execute(*sTable);
> PointViewPtr sPointView = *sPointViewSet.begin();
> Dimension::IdList sDims = sPointView->dims();
>
> int pointCount = sLasReader->getNumPoints();
> int chunkSize = pointCount / threadCount;
>
> parallel_for(0, threadCount, [&](int i) {
>
> pdal::PointId start = i*chunkSize;
> pdal::PointId end = (i == 0 || i < threadCount - 1) ? (i + 1)*chunkSize :
> pointCount;
>
> for (pdal::PointId idx = start; idx < end; ++idx) {
> double x = sPointView->getFieldAs<double>(Id::X, idx) - spOffsetpunkt.x;
> double y = sPointView->getFieldAs<double>(Id::Y, idx) - spOffsetpunkt.y;
> double z = sPointView->getFieldAs<double>(Id::Z, idx) - spOffsetpunkt.z;
> int red = sPointView->getFieldAs<int>(Id::Red, idx);
> int green = sPointView->getFieldAs<int>(Id::Green, idx);
> int blue = sPointView->getFieldAs<int>(Id::Blue, idx);
>
> //... process point
>
> }
> });
>
>
>
> *Streaming Callback reader (Low memory footprint, but SLOW):*
> pdal::Option las_opt("filename", string(filename));
> pdal::Options las_opts;
> las_opts.add(las_opt);
> LasReader sLasReader;
> sLasReader.setOptions(las_opts);
> PointTable sTable;
> sLasReader.prepare(sTable);
>
> int pointCount = sLasReader->getNumPoints();
>
> auto cb = [&](PointRef& point) {
>
> double x = point.getFieldAs<double>(Id::X) - spOffsetpunkt.x;
> double y = point.getFieldAs<double>(Id::Y) - spOffsetpunkt.y;
> double z = point.getFieldAs<double>(Id::Z) - spOffsetpunkt.z;
>
> int red = point.getFieldAs<int>(Id::Red);
> int green = point.getFieldAs<int>(Id::Green);
> int blue = point.getFieldAs<int>(Id::Blue);
>
> //... process point
>
> return true;
> };
>
> pdal::Option las_opt("filename", sFilename);
> pdal::Options las_opts;
> las_opts.add(las_opt);
> LasReader sLasReader;
> sLasReader.setOptions(las_opts);
>
> FixedPointTable t(1000);
> StreamCallbackFilter scf;
> scf.setCallback(cb);
> scf.setInput(sLasReader);
> scf.prepare(t);
> scf.execute(t);
>
>
> _______________________________________________
> pdal mailing list
> pdal at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/pdal



-- 
Andrew Bell
andrew.bell.ia at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pdal/attachments/20190516/8559255e/attachment.html>