<div dir="ltr">Hi Connor,<div><br></div><div>Thanks for the quick and informative reply. I will implement the filename as you suggested. A quick follow-up: Is streaming enabled on readers.ept?</div><div><br></div><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">---------------------------<div>Matthew Beckley</div><div>Data Engineer</div><div>UNAVCO/OpenTopography</div><div><a href="mailto:beckley@unavco.org" target="_blank">beckley@unavco.org</a></div>cell: 301-982-9819</div></div></div></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Dec 15, 2020 at 11:21 AM Connor Manning <<a href="mailto:connor@hobu.co">connor@hobu.co</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>In one of the last few releases (not sure which) we tried to move away from the "ept://" pseudo-protocol and instead use the presence of "ept.json" at the end to signify the EPT reader. So please try using a filename of <a href="https://s3-us-west-2.amazonaws.com/usgs-lidar-public/USGS_LPC_CA_Central_Valley_2017_LAS_2019/ept.json" target="_blank">https://s3-us-west-2.amazonaws.com/usgs-lidar-public/USGS_LPC_CA_Central_Valley_2017_LAS_2019/ept.json</a> instead - this is the recommended format from now on. I think we kept support for both formats for at least one release.<br></div><div><br></div><div>This change was for a few reasons: when accessing over a network, the double protocol (ept://http://...) is strange, and also that using the root directory rather than the ept.json filename means that your "filename" option is not a real file, e.g. <a href="https://s3-us-west-2.amazonaws.com/usgs-lidar-public/USGS_LPC_CA_Central_Valley_2017_LAS_2019" target="_blank">https://s3-us-west-2.amazonaws.com/usgs-lidar-public/USGS_LPC_CA_Central_Valley_2017_LAS_2019</a> is a 404, but <a href="https://s3-us-west-2.amazonaws.com/usgs-lidar-public/USGS_LPC_CA_Central_Valley_2017_LAS_2019/ept.json" target="_blank">https://s3-us-west-2.amazonaws.com/usgs-lidar-public/USGS_LPC_CA_Central_Valley_2017_LAS_2019/ept.json</a> is an actual file.</div><div><br></div><div>I wouldn't worry much about the file size difference here since your point counts match: since the EPT reader runs in a multi-threaded fashion, the order of points may vary between runs, which leads to slight differences in the compression. You could add a "filters.sort" after the EPT reader to counteract this (for LAZ data I'd recommend sorting by GpsTime and maybe secondarily by ReturnNumber).</div><div><br></div><div>I'm not sure why your filesource_id would be changing, so maybe open a Github issue on that one.</div><div><br></div><div>- Connor<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Dec 15, 2020 at 12:04 PM Matt Beckley <<a href="mailto:beckley@unavco.org" target="_blank">beckley@unavco.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hello,</div><div><br></div>It seems like when reading the ept data from the AWS 3DEP entwine bucket the reader will not work unless I add the prefix, "ept://" to the URL (see examples below). This applies only to PDAL v2.2, and it is not clear if this is a dataset-specific issue. PDAL 2.1 will run with or without the ept:// prefix, but it has the odd result that the filesizes will differ slightly if using ept:// in the prefix or not. Point counts are the same whether or not you use ept:// with PDAL 2.1, but the "filesource_id" parameter differs, so the filesize differences are probably due to slight header differences. In regards to the PDAL2.2 EPT issue, so far this seems to happen on the following AWS 3DEP Entwine datasets:<br><br>USGS LPC CA Central Valley 2017 LAS 2019<br>CO_Southwest_NRCS_B2_2018<br>TX WestTexas B1 2018<br>NM SouthCentral B8 2018<br><br><u>My question:</u> For PDAL v2.2, should I always use the EPT:// prefix when using readers.ept? (seems related to: <a href="https://github.com/PDAL/PDAL/pull/3174" target="_blank">https://github.com/PDAL/PDAL/pull/3174</a>). Also, as an aside, is streaming available for readers.ept? Documentation doesn't indicate it is, but this issue: <a href="https://github.com/PDAL/PDAL/issues/2439" target="_blank">https://github.com/PDAL/PDAL/issues/2439</a> makes it seem that maybe it is? I'm uncertain how to test this.<div><br></div><div>Any info you could provide would be most appreciated.<br><br>Test1: PDAL 2.2 WITHOUT EPT:// Prefix (PDAL installed via isolated conda environment):<br><br>{ <br> "pipeline": [{ <br> "type": "readers.ept", <br> "filename": "<a href="https://s3-us-west-2.amazonaws.com/usgs-lidar-public/USGS_LPC_CA_Central_Valley_2017_LAS_2019" target="_blank">https://s3-us-west-2.amazonaws.com/usgs-lidar-public/USGS_LPC_CA_Central_Valley_2017_LAS_2019</a>", <br> "bounds": "([-13484500, -13484200], [4653000,4654200])" <br> }, <br> "points_CA_noept.laz"]} <br><br>pdal pipeline pipeline.json gives error: <br><br>PDAL: readers.ept: Could not read from <a href="http://s3-us-west-2.amazonaws.com/usgs-lidar-public/USGS_LPC_CA_Central_Valley_2017_LAS_2019" target="_blank">s3-us-west-2.amazonaws.com/usgs-lidar-public/USGS_LPC_CA_Central_Valley_2017_LAS_2019</a><br><br>Test2: PDAL 2.2 WITH EPT:// Prefix (PDAL installed via isolated conda environment):<br>{ <br> "pipeline": [{ <br> "type": "readers.ept", <br> "filename": "ept://<a href="https://s3-us-west-2.amazonaws.com/usgs-lidar-public/USGS_LPC_CA_Central_Valley_2017_LAS_2019" target="_blank">https://s3-us-west-2.amazonaws.com/usgs-lidar-public/USGS_LPC_CA_Central_Valley_2017_LAS_2019</a>", <br> "bounds": "([-13484500, -13484200], [4653000,4654200])" <br> }, <br> "points_CA_wept.laz"]} <br><br>pdal pipeline pipeline.json runs successfully<br><br><br>Test3: PDAL 2.1 WITHOUT EPT:// Prefix (PDAL installed via isolated conda environment):<br>{<br> "pipeline": [{<br> "type": "readers.ept",<br> "filename": "ept://<a href="https://s3-us-west-2.amazonaws.com/usgs-lidar-public/USGS_LPC_CA_Central_Valley_2017_LAS_2019" target="_blank">https://s3-us-west-2.amazonaws.com/usgs-lidar-public/USGS_LPC_CA_Central_Valley_2017_LAS_2019</a>",<br> "bounds": "([-13484500, -13484200], [4653000,4654200])"<br> },<br> "points_CA_wept_v21.laz"]}<br><br>pdal pipeline pipeline.json runs successfully, filesize is: 3453289 bytes. "count": 956938<br><br><br>Test4: PDAL 2.1 WITH EPT:// Prefix (PDAL installed via isolated conda environment):<br>{ <br> "pipeline": [{ <br> "type": "readers.ept", <br> "filename": "<a href="https://s3-us-west-2.amazonaws.com/usgs-lidar-public/USGS_LPC_CA_Central_Valley_2017_LAS_2019" target="_blank">https://s3-us-west-2.amazonaws.com/usgs-lidar-public/USGS_LPC_CA_Central_Valley_2017_LAS_2019</a>", <br> "bounds": "([-13484500, -13484200], [4653000,4654200])" <br> }, <br> "points_CA_NOept_v21.laz"]} <br><br>pdal pipeline pipeline.json runs successfully, but filesize is different: 3479637 bytes. "count": 956938<br><br><br>Point counts for results from PDAL 2.1 run match. Only difference is "filesource_id". Version without EPT:// prefix has filesource_id=0, while with EPT:// prefix "filesource_id": 26982.<br clear="all"><div><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">---------------------------<div>Matthew Beckley</div><div>Data Engineer</div><div>UNAVCO/OpenTopography</div><div><a href="mailto:beckley@unavco.org" target="_blank">beckley@unavco.org</a></div>cell: 301-982-9819</div></div></div></div></div></div></div></div></div></div></div>
_______________________________________________<br>
pdal mailing list<br>
<a href="mailto:pdal@lists.osgeo.org" target="_blank">pdal@lists.osgeo.org</a><br>
<a href="https://lists.osgeo.org/mailman/listinfo/pdal" rel="noreferrer" target="_blank">https://lists.osgeo.org/mailman/listinfo/pdal</a><br>
</blockquote></div>
</blockquote></div>