[pdal] Streaming versus non-streaming LAS read perfromance

[KB-Vis Technologies Pvt. Ltd.]

Greetings,
I have a huge performance issue with reading LAS
points on large files.
I initially implemented a non-streaming read
using PointViewSet as in the examples, and this worked fine. However
on larger LAS files e.g. a 1GB file with 150 million points, the memory
usage would go up to 15GB, and the load would take around 3-5
minutes.
Then, I switched to a streaming read
using FixedPointTable and StreamCallbackFilter. This uses much less
memory but is VERY slow. The 150 million points take 15+ minutes to
load..The issue seems to be that memory is being allocated in increments
of 3MB, and this takes forever. tried various values for the
FixedPointTable size, but this seems to have no effect.
I would
prefer to use the streaming approach to do fast low-res partial reads etc.
and to keep the memory usage down, if only I could improve the loading
time.
I am using PDAL 1.7.2. I was wondering if I am doing something
very wrong, or whether I should just update to 1.9
Any advice would
be hugely appreciated!
many thanks,
Karthik
Bala
 
My code:
Original Non-Streaming
reader (FAST but memory-intensive)

	pdal::Option las_opt("filename", sFilenames[f]);

	pdal::Options las_opts;

	las_opts.add(las_opt);

	LasReader *sLasReader = new LasReader();

	sLasReader->setOptions(las_opts);

	PointTable *sTable = new PointTable();

	sLasReader->prepare(*sTable);

	PointViewSet sPointViewSet = sLasReader->execute(*sTable);

	PointViewPtr sPointView = *sPointViewSet.begin();

	Dimension::IdList sDims = sPointView->dims();

	 

	int pointCount = sLasReader->getNumPoints();

	int chunkSize = pointCount / threadCount;

	 

	parallel_for(0, threadCount, [&](int i) {

	 

	pdal::PointId start = i*chunkSize;

	pdal::PointId end = (i == 0 || i < threadCount - 1) ? (i +
1)*chunkSize : pointCount;

	 

	for (pdal::PointId idx = start; idx < end; ++idx) {

	double x = sPointView->getFieldAs<double>(Id::X, idx) -
spOffsetpunkt.x;

	double y = sPointView->getFieldAs<double>(Id::Y, idx) -
spOffsetpunkt.y;

	double z = sPointView->getFieldAs<double>(Id::Z, idx) -
spOffsetpunkt.z;

	int red = sPointView->getFieldAs<int>(Id::Red, idx);

	int green = sPointView->getFieldAs<int>(Id::Green, idx);

	int blue = sPointView->getFieldAs<int>(Id::Blue, idx);

	
		 
	
		//... process point


	 

	}

	});

 

	
Streaming Callback reader (Low memory footprint, but
SLOW):
	
		pdal::Option las_opt("filename", string(filename));
	
		pdal::Options las_opts;
	
		las_opts.add(las_opt);
	
		LasReader sLasReader;
	
		sLasReader.setOptions(las_opts);
	
		PointTable sTable;
	
		sLasReader.prepare(sTable);
	
		 
	
		int pointCount = sLasReader->getNumPoints();
	
		 
	
		auto cb = [&](PointRef& point) {
	
		 
	
		double x = point.getFieldAs<double>(Id::X) - spOffsetpunkt.x;
	
		double y = point.getFieldAs<double>(Id::Y) - spOffsetpunkt.y;
	
		double z = point.getFieldAs<double>(Id::Z) - spOffsetpunkt.z;
	
		 
	
		int red = point.getFieldAs<int>(Id::Red);
	
		int green = point.getFieldAs<int>(Id::Green);
	
		int blue = point.getFieldAs<int>(Id::Blue);
	
		 
	
		//... process point
	
		 
	
		return true;
	
		};
	
		 
	
		pdal::Option las_opt("filename", sFilename);
	
		pdal::Options las_opts;
	
		las_opts.add(las_opt);
	
		LasReader sLasReader;
	
		sLasReader.setOptions(las_opts);
	
		 
	
		FixedPointTable t(1000);
	
		StreamCallbackFilter scf;
	
		scf.setCallback(cb);
	
		scf.setInput(sLasReader);
	
		scf.prepare(t);
	
		scf.execute(t);
	
		 
	
		 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pdal/attachments/20190517/439474cd/attachment.html>