[pdal] PDAL Python3 issues

[Howard Butler]

> On Jan 23, 2017, at 11:31 AM, Jean-Francois Prieur <jfprieur at gmail.com> wrote:
> 
> When the student started (almost 2 years ago), we used OSGeo4W open source tools for development. The initial workflow was awesome. Read each file with PDAL, use pgwriter to send it to postgres, calculate all the metrics in the database. Worked like a charm until pgwriter dissapeared from the osgeo4w version of PDAL (we completely understand how this can happen, this is not a complaint!) so this production chain was broken. We both did not have the time (at the time) to figure out how to install everything in linux so she decided to press forward using Python. The end product is still in Postgres, it is the initial 'reading the LAS file' part that pgwriter performed flawlessly that is causing issues now.

Well that's a bummer. Your use case is actually a good one for pgpointcloud, and you had a good workflow going. Sorry to break things for you :(

I was recently contacted by NRCan about them paying to get a 1.8.1 OSGeo4W64 libLAS build together, but I have not heard back anything after I gave a quote.  I think 1.8.0 definitely had a memory management issue where it leaked file handles. IIRC, it was cleaned up in 1.8.1, but IMO pgointcloud, which you already had working is the better solution here.

An alternative that might give you traction is to use Docker http://www.pdal.io/quickstart.html The PDAL docker build is feature-complete with pgpointcloud support (and most other filters), and you could use it to get data in/out of your database by calling docker commands on windows. See the Quickstart http://www.pdal.io/quickstart.html for a teaser and the Workshop materials http://www.pdal.io/workshop/exercises/index.html for in-depth docker-on-windows examples. Docker might require Windows 10 for smooth usage, however.

A better solution of course is pgpointcloud support and current OSGeo4W64 binaries for windows. If you are willing to live dangerously, PDAL's continuous integration build, based off of OSGeo464, builds pgpointcloud-enabled binaries. It's just that you can only get a .zip file of the binaries, and you will need to do some %PATH% plumbing and other junk to get them to work with a current OSGeo4W64 environment. After every successful AppVeyor build, the zip file is placed at https://s3.amazonaws.com/pdal/osgeo4w/pdal.zip This means a constantly-changing but constantly up-to-date build is available. No promises.

A note for others watching PDAL's Windows situation: the problem is not getting builds done -- they're available via AppVeyor. The problem is smooth integration with OSGeo4W64, and a convenient packaging script to push releases at OSGeo4W64. I used to manually maintain this for libLAS, and it was awful. The first few OSGeo4W64 builds were the same. The task is an integration one, not so much a development one. 

> A python 3 script using libLAS opens the LAS tile, runs through each crown to find the points associated to it and stores the result as a LAS file. The issue is that an individual LAS file is created for each tree crown, when we have more than 40,000 crowns per tile the system starts swapping (windows and linux) and the process just gets very slow. Then another script reads the las points, calculates metrics which are then stored in the database. This 'clipping' operation for the tree crowns only happens once at the beginning, it is not a problem. But it would take a month right now using libLAS which is not acceptable.
> 
> So all I am looking for ;) is a linux python library that can write up tp 100,000 'mini-LAS' tree crowns from a las tile without running out of memory like libLAS does. Believe PDAL could do that quite simply via Python hence my attempts. I know that laspy exists but it is only for Python 2.

So you do indeed want to "touch the points"... but I think it would be best and cleanest to get back to pgpointcloud. You can get back there with Docker for i/o or try to bleed on the bleeding edge with the AppVeyor build and feather it into your OSGeo4W64 build.

> Thanks for any insights the list may have, keeping in mind we are relative programming noob scientists that don`t mind to work and read!

> Sorry for the book!

On the contrary, this kind of feedback lets us know how well or not well PDAL is doing the job for people. As I've said before, we have a particular set of use cases we use PDAL for, and it is encouraging that people are finding other ways to make it useful. We want to remove obvious blockers that prevent it from being so. Windows builds and integration are a tough one due to the fact that none of the PDAL developers work natively on that platform. 

Thanks for the feedback!

Howard