[pdal] PDAL Python3 issues

[Jean-Francois Prieur]

Thank you for the insights on how the sausage is made ;), I am not tied to
Windows and am actively trying to get away from it!

Will try the docker tools if we must follow the windows route, thanks
again. Will keep you posted on our progress. Keep focused on linux ;) as I
stated I am removing windows from my workflow as much as possible. You did
not break anything rest assured!

JF

On Mon, Jan 23, 2017 at 2:18 PM Howard Butler <howard at hobu.co> wrote:

>
> > On Jan 23, 2017, at 11:31 AM, Jean-Francois Prieur <jfprieur at gmail.com>
> wrote:
> >
> > When the student started (almost 2 years ago), we used OSGeo4W open
> source tools for development. The initial workflow was awesome. Read each
> file with PDAL, use pgwriter to send it to postgres, calculate all the
> metrics in the database. Worked like a charm until pgwriter dissapeared
> from the osgeo4w version of PDAL (we completely understand how this can
> happen, this is not a complaint!) so this production chain was broken. We
> both did not have the time (at the time) to figure out how to install
> everything in linux so she decided to press forward using Python. The end
> product is still in Postgres, it is the initial 'reading the LAS file' part
> that pgwriter performed flawlessly that is causing issues now.
>
> Well that's a bummer. Your use case is actually a good one for
> pgpointcloud, and you had a good workflow going. Sorry to break things for
> you :(
>
> I was recently contacted by NRCan about them paying to get a 1.8.1
> OSGeo4W64 libLAS build together, but I have not heard back anything after I
> gave a quote.  I think 1.8.0 definitely had a memory management issue where
> it leaked file handles. IIRC, it was cleaned up in 1.8.1, but IMO
> pgointcloud, which you already had working is the better solution here.
>
> An alternative that might give you traction is to use Docker
> http://www.pdal.io/quickstart.html The PDAL docker build is
> feature-complete with pgpointcloud support (and most other filters), and
> you could use it to get data in/out of your database by calling docker
> commands on windows. See the Quickstart http://www.pdal.io/quickstart.html
> for a teaser and the Workshop materials
> http://www.pdal.io/workshop/exercises/index.html for in-depth
> docker-on-windows examples. Docker might require Windows 10 for smooth
> usage, however.
>
> A better solution of course is pgpointcloud support and current OSGeo4W64
> binaries for windows. If you are willing to live dangerously, PDAL's
> continuous integration build, based off of OSGeo464, builds
> pgpointcloud-enabled binaries. It's just that you can only get a .zip file
> of the binaries, and you will need to do some %PATH% plumbing and other
> junk to get them to work with a current OSGeo4W64 environment. After every
> successful AppVeyor build, the zip file is placed at
> https://s3.amazonaws.com/pdal/osgeo4w/pdal.zip This means a
> constantly-changing but constantly up-to-date build is available. No
> promises.
>
> A note for others watching PDAL's Windows situation: the problem is not
> getting builds done -- they're available via AppVeyor. The problem is
> smooth integration with OSGeo4W64, and a convenient packaging script to
> push releases at OSGeo4W64. I used to manually maintain this for libLAS,
> and it was awful. The first few OSGeo4W64 builds were the same. The task is
> an integration one, not so much a development one.
>
> > A python 3 script using libLAS opens the LAS tile, runs through each
> crown to find the points associated to it and stores the result as a LAS
> file. The issue is that an individual LAS file is created for each tree
> crown, when we have more than 40,000 crowns per tile the system starts
> swapping (windows and linux) and the process just gets very slow. Then
> another script reads the las points, calculates metrics which are then
> stored in the database. This 'clipping' operation for the tree crowns only
> happens once at the beginning, it is not a problem. But it would take a
> month right now using libLAS which is not acceptable.
> >
> > So all I am looking for ;) is a linux python library that can write up
> tp 100,000 'mini-LAS' tree crowns from a las tile without running out of
> memory like libLAS does. Believe PDAL could do that quite simply via Python
> hence my attempts. I know that laspy exists but it is only for Python 2.
>
> So you do indeed want to "touch the points"... but I think it would be
> best and cleanest to get back to pgpointcloud. You can get back there with
> Docker for i/o or try to bleed on the bleeding edge with the AppVeyor build
> and feather it into your OSGeo4W64 build.
>
> > Thanks for any insights the list may have, keeping in mind we are
> relative programming noob scientists that don`t mind to work and read!
>
> > Sorry for the book!
>
> On the contrary, this kind of feedback lets us know how well or not well
> PDAL is doing the job for people. As I've said before, we have a particular
> set of use cases we use PDAL for, and it is encouraging that people are
> finding other ways to make it useful. We want to remove obvious blockers
> that prevent it from being so. Windows builds and integration are a tough
> one due to the fact that none of the PDAL developers work natively on that
> platform.
>
> Thanks for the feedback!
>
> Howard
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pdal/attachments/20170123/792b75ed/attachment.html>