<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Wed, May 24, 2017 at 8:31 AM, GUIMMARA, Sébastien (External) <span dir="ltr"><<a href="mailto:sebastien.guimmara.external@airbus.com" target="_blank">sebastien.guimmara.external@airbus.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="FR">
<p class="gmail-m_390808002701058225367050b90-a467-4d62-be36-e5ff8cdc9d4b"></p>
<div class="gmail-m_3908080027010582253WordSection1">
<p class="MsoNormal"><span lang="EN-US">Hello,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">I am trying to process a 400 MB LAS file of Mount Rainier downloaded from Open Topography:<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><a href="https://cloud.sdsc.edu/v1/AUTH_opentopography/PC_Bulk/Rainier/Q46122G12.laz" target="_blank">https://cloud.sdsc.edu/v1/<wbr>AUTH_opentopography/PC_Bulk/<wbr>Rainier/Q46122G12.laz</a><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">The dataset contains 82M points.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Firstly, I come across this warning: “Found invalid value of ‘6’ for points’ return number”. Is this a serious issue ?</span></p></div></div></blockquote><div><br></div><div>No. It's just telling you that the value is invalid, according to the spec.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="FR"><div class="gmail-m_3908080027010582253WordSection1"><p class="MsoNormal"><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> </span>Secondly, processing this dataset through a pipeline that contains filter.colorinterp, filter.normal, and filter.programmable consumes huge amounts of memory (15 GB).</p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Is it possible to strip the PointView of all but the essential dimensions (X, Y, Z) in the pipeline to save memory ?</span></p></div></div></blockquote><div><br></div><div><div class="gmail_quote">This has come up from time. We may be able to provide this functionality from the command-line, but I don't know that we've given it much thought lately. The issue is for you exacerbated because a) you use filters.normal, which creates an additional X/Y/Z dimension for each point to store the normal data and b) using Python currently forces data to get copied because the internal format of data in PDAL won't easily map directly to numpy arrays (it's not always stored as contiguous data). There may be a way around this, but I don't know that we've spent much time investigating. </div></div><div><br></div><div>There are a couple of issues/possibilities.</div><div><br></div><div>1) PDAL supports "stream" mode, which only loads a certain number of points at a time and processes them through the pipeline in batches. However, some filters don't support stream mode because they need all the data in order to work (think sorting). In your case, neither filters.normal nor filters.programmable supports stream mode.</div><div><br></div><div>2) If you're a programmer, you can create a custom point layout that rejects dimension registration other than the ones that you want. All you really need to do is to create a subclass of PointLayout and reimplement the update() function to return "false" for all dimensions that you're not interested in. I can provide an example, but if you're not programming C++, this isn't really an option for you.</div><div><br></div><div>3) You could convert the data to another format that doesn't have a set data layout (LAS point layouts are fixed). You could try something like:</div><div><br></div><div>$ pdal pipeline --stream <mypipeline></div><div><br></div><div>mypipeline:</div><div><br></div><div>{</div><div> "pipeline":[</div><div> "Q46122g12.laz",</div><div> {</div><div> "type":"writers.bpf",</div><div> "filename":"Q46122g12.bpf",</div><div> "output_dims":"X,Y,Z"</div><div> }<br></div><div> ]</div><div>}</div><div><br></div><div>This *untested* example should create a BPF file with only X,Y and Z in it. You can then use that as input to your pipeline. Note that internally X,Y and Z are still 8 bytes internally in PDAL, as are the normal vector X, Y and Z, so you've got a minimum of 48 bytes per point no matter what, plus the copies for Python.</div><div><br></div><div>Hope this helps a little.</div><div><br></div></div>-- <br><div class="gmail_signature">Andrew Bell<br><a href="mailto:andrew.bell.ia@gmail.com" target="_blank">andrew.bell.ia@gmail.com</a></div>
</div></div>