[pdal] Python pipeline not reading ept.json

[Gelder, Brian K [A&BE]]

Thank you for the advice Howard. I was guessing that was the problem - I had tried pdal before and given up back when I used ArcMap 10.3 as that was even less friendly towards adding packages. I'll set up a clean non-ArcGIS Python environment for my pdal work.

I'll add a little more detail on what I'm planning in case anyone else comes across this thread (or others can give me advice on better ways to implement this). What I do is run a parent script in a recent version of Python and then use the Python subprocess module to launch child scripts from the command line where I can specify which Python interpreter/environment I want to use to launch each specific Python file. I pass general process input and output names via command line arguments and thus the next Python child program picks up where the previous Python child program ended. This does create some complications when debugging but I can manage that.

I do this currently using the Python executable from ArcMap 10.3 with Background Geoprocessing for some raster work as arcpy was still fast for raster processing since it still used ESRI GRIDs natively. Other, newer processes have been moved to Pro and use the cloned environment/interpreter executable I've set up there to do that processing. I'm just going to add another non-ArcGIS Python environment/interpreter to call so I can use pdal to process my pipeline JSON.

Thanks for supporting this great product!
Brian

Brian K. Gelder, PhD
“Mud” Research Manager (Soil and Water Research), Ag & Biosystems Engr. Dept.
Daily Erosion Project & Agricultural Conservation Planning Framework
Adjunct Assistant Professor, ABE, CRP & Agronomy Dept.
Instructor, CRP 454/554/LA 554 and CRP 456/556/ABE 556
3321 Elings Hall, Iowa State University
515-294-3144
My schedule

> -----Original Message-----
> From: Howard Butler <howard at hobu.co>
> Sent: Friday, August 19, 2022 07:46
> To: Gelder, Brian K [A&BE] <bkgelder at iastate.edu>
> Cc: pdal at lists.osgeo.org
> Subject: Re: [pdal] Python pipeline not reading ept.json
> 
> Howard
> 
> Ah, ArcGIS conda environments. That's an entirely different bucket of eels.
> 
> I would suggest trying to operate in a clean non-ArcGIS environment that's
> known to work. We are available for a support contract if being inside ArcGIS
> is a hard requirement for you, but it's not something that's usually simple to
> address.
> 
> I wish ArcGIS used the principle of "small environments", but unfortunately,
> their default environment installs the universe and then users are stuck
> waiting for it to never solve or end up with linking conflicts like these.
> 
> Howard
> 
> 
> 
> > On Aug 19, 2022, at 6:11 AM, Gelder, Brian K [A&BE]
> <bkgelder at iastate.edu> wrote:
> >
> > Howard,
> > Thanks for your tip; should have thought of that myself. It does appear to
> be an issue with Python and not pdal directly. I'm guessing it has to do with
> the fact that I am using ArcGIS Pro 3.0 for my Python 3.9 install. I installed
> Pro, cloned the default Python install, and then installed via "conda install -c
> conda-forge pdal" Does anyone have suggestions on how make these two
> play nicely together?
> >
> > I can run pipelines directly from the command line but not through
> Python. If I run the following pdal command inside my ArcGIS Pro configured
> Python command prompt "pdal pipeline iowa.json --debug"
> >
> > The pipeline then runs successfully. However if I try to run "pdal --drivers"
> at the same command line I get the following error messages. After
> acknowledging the errors it then proceeds to print the drivers and I can still
> run pipelines.
> >
> > "pdal.exe - Entry Point No Found"
> >
> > "The procedure entry point
> ?mergeMode at Filter@pdal@@EEBA?AW4WhereMergeMode at State@2 at XZ
> could not be located in the dynamic link library
> C:\Users\bkgelder\AppData\Local\ESRI\conda\envs\pda-testing-
> env\lib\site_packages\bin\libpdal_plugin_filter_numpy.dll"
> >
> > "The procedure entry point
> ?mergeMode at Reader@pdal@@EEBA?AW4WhereMergeMode at State@2@
> XZ could not be located in the dynamic link library
> C:\Users\bkgelder\AppData\Local\ESRI\conda\envs\pda-testing-
> env\lib\site_packages\bin\libpdal_plugin_reader_numpy.dll"
> >
> > Additionally - if I try to import pdal within a .py script file I receive the
> following warning about a circular import. I do not receive this when I
> import pdal in the IDLE shell.
> >
> > "Traceback (most recent call last):
> >  File "C:\Users\bkgelder\Box\Scripts\pdal.py", line 4, in <module>
> >    import pdal
> >  File "C:\Users\bkgelder\Box\Scripts\pdal.py", line 16, in <module>
> >    pipeline = pdal.Pipeline(json)
> > AttributeError: partially initialized module 'pdal' has no attribute 'Pipeline'
> (most likely due to a circular import)"
> >
> > I've also been trying to update pdal to 2.4.3 but not having luck due to
> listed conflicts. I've tried "conda update pdal" and "conda install -c conda-
> forge pdal=2.4.3"
> >
> > Thank you,
> > Brian
> >
> > -----Original Message-----
> > From: Howard Butler <howard at hobu.co>
> > Sent: Monday, August 15, 2022 2:26 PM
> > To: Gelder, Brian K [A&BE] <bkgelder at iastate.edu>
> > Cc: pdal at lists.osgeo.org
> > Subject: Re: [pdal] Python pipeline not reading ept.json
> >
> >
> >
> >> On Aug 15, 2022, at 12:53 PM, Gelder, Brian K [A&BE]
> <bkgelder at iastate.edu> wrote:
> >>
> >> Hello,
> >> I’m a pdal newbie and I’m trying to figure out how to successfully read
> ept.json files. I have pdal 2.4.2 installed via Conda on Python 3.9 on Win 10
> and am able to use the pipeline to process local datasets like interesting.las
> in the tutorials. However, when I try to read EPT web resources, like in
> https://pdal.io/en/latest/tutorial/iowa-entwine.html, I receive the following
> error:
> >>
> >> Traceback (most recent call last):
> >>  File "<pyshell#15>", line 1, in <module>
> >>    pipeline2.validate()
> >>  File "C:\Users\bkgelder\AppData\Local\ESRI\conda\envs\pdal-testing-
> env\lib\site-packages\pdal\pipeline.py", line 43, in validate
> >>    return self.p.validate()
> >>  File "libpdalpython.pyx", line 185, in
> pdal.libpdalpython.PyPipeline.validate
> >> RuntimeError: readers.ept: Could not read from s3-us-west-
> 2.amazonaws.com/usgs-lidar-public/IA_FullState/ept.json
> >>
> >> I get similar failure when I tried to query the smaller Hurricane Florence
> dataset Vishu wrote about last week. I can view the ept.json files in Firefox.
> Is there a firewall setting I’m missing?
> >
> > Brian,
> >
> > The first thing to do is to try a pipeline with 'pdal pipeline' and take Python
> out of it. Another thing to check is pull a new environment with PDAL 2.4.3
> and see if that works. There was a big memory leak of the LAZ
> decompression in 2.4.0-2.4.2 that you won't want if you're doing things in
> Python.
> >
> > Hope this helps,
> >
> > Howard
> >