<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<!-- Template generated by Exclaimer Mail Disclaimers on 04:58:44 mercredi, 24 mai 2017 -->
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css">P.67050b90-a467-4d62-be36-e5ff8cdc9d4b {
MARGIN: 0cm 0cm 0pt
}
LI.67050b90-a467-4d62-be36-e5ff8cdc9d4b {
MARGIN: 0cm 0cm 0pt
}
DIV.67050b90-a467-4d62-be36-e5ff8cdc9d4b {
MARGIN: 0cm 0cm 0pt
}
TABLE.67050b90-a467-4d62-be36-e5ff8cdc9d4bTable {
MARGIN: 0cm 0cm 0pt
}
DIV.Section1 {
page: Section1
}
</style>
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.gmail-m390808002701058225367050b90-a467-4d62-be36-e5ff8cdc9d4b, li.gmail-m390808002701058225367050b90-a467-4d62-be36-e5ff8cdc9d4b, div.gmail-m390808002701058225367050b90-a467-4d62-be36-e5ff8cdc9d4b
{mso-style-name:gmail-m_390808002701058225367050b90-a467-4d62-be36-e5ff8cdc9d4b;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="FR" link="blue" vlink="purple">
<p class="67050b90-a467-4d62-be36-e5ff8cdc9d4b"></p>
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I think the third option (converting to a more flexible format first) is the best, I’m going to test it.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Thanks very much,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Sébastien<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">De :</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Andrew Bell [mailto:andrew.bell.ia@gmail.com]
<br>
<b>Envoyé :</b> mercredi 24 mai 2017 16:16<br>
<b>À :</b> GUIMMARA, Sébastien (External)<br>
<b>Cc :</b> pdal@lists.osgeo.org<br>
<b>Objet :</b> Re: [pdal] Can we drop dimensions to save memory ?<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">On Wed, May 24, 2017 at 8:31 AM, GUIMMARA, Sébastien (External) <<a href="mailto:sebastien.guimmara.external@airbus.com" target="_blank">sebastien.guimmara.external@airbus.com</a>> wrote:<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Hello,</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">I am trying to process a 400 MB LAS file of Mount Rainier downloaded from Open Topography:</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"><a href="https://cloud.sdsc.edu/v1/AUTH_opentopography/PC_Bulk/Rainier/Q46122G12.laz" target="_blank">https://cloud.sdsc.edu/v1/AUTH_opentopography/PC_Bulk/Rainier/Q46122G12.laz</a></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">The dataset contains 82M points.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Firstly, I come across this warning: “Found invalid value of ‘6’ for points’ return number”. Is this a serious issue ?</span><o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">No. It's just telling you that the value is invalid, according to the spec.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span>Secondly, processing this dataset through a pipeline that contains filter.colorinterp, filter.normal, and filter.programmable consumes huge amounts of
memory (15 GB).<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Is it possible to strip the PointView of all but the essential dimensions (X, Y, Z) in the pipeline to save memory ?</span><o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">This has come up from time. We may be able to provide this functionality from the command-line, but I don't know that we've given it much thought lately. The issue is for you exacerbated because a) you use filters.normal, which creates
an additional X/Y/Z dimension for each point to store the normal data and b) using Python currently forces data to get copied because the internal format of data in PDAL won't easily map directly to numpy arrays (it's not always stored as contiguous data).
There may be a way around this, but I don't know that we've spent much time investigating. <o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">There are a couple of issues/possibilities.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">1) PDAL supports "stream" mode, which only loads a certain number of points at a time and processes them through the pipeline in batches. However, some filters don't support stream mode because they need all the data in order to work (think
sorting). In your case, neither filters.normal nor filters.programmable supports stream mode.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">2) If you're a programmer, you can create a custom point layout that rejects dimension registration other than the ones that you want. All you really need to do is to create a subclass of PointLayout and reimplement the update() function
to return "false" for all dimensions that you're not interested in. I can provide an example, but if you're not programming C++, this isn't really an option for you.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">3) You could convert the data to another format that doesn't have a set data layout (LAS point layouts are fixed). You could try something like:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">$ pdal pipeline --stream <mypipeline><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">mypipeline:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">{<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> "pipeline":[<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> "Q46122g12.laz",<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> {<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> "type":"writers.bpf",<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> "filename":"Q46122g12.bpf",<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> "output_dims":"X,Y,Z"<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> }<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> ]<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">This *untested* example should create a BPF file with only X,Y and Z in it. You can then use that as input to your pipeline. Note that internally X,Y and Z are still 8 bytes internally in PDAL, as are the normal vector X, Y and Z, so
you've got a minimum of 48 bytes per point no matter what, plus the copies for Python.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Hope this helps a little.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<p class="MsoNormal">Andrew Bell<br>
<a href="mailto:andrew.bell.ia@gmail.com" target="_blank">andrew.bell.ia@gmail.com</a><o:p></o:p></p>
</div>
</div>
</div>
</div>
<p></p>
<p class="67050b90-a467-4d62-be36-e5ff8cdc9d4b"> </p>
<font color="#008000" size="1" face="Verdana"><img border="0" hspace="0" alt="" align="baseline" src="cid:image006e10.JPG@00aafa0a.47ac9e91"> Please consider the environment before printing this email message.</font>
<p></p>
<hr>
<p style="MARGIN-RIGHT: 0px" dir="ltr" class="{Imprint.UniqueID}"><font color="#999999" size="2" face="Verdana"><em>Ce courriel (incluant ses éventuelles pièces jointes) peut contenir des informations confidentielles et/ou protégées ou dont la diffusion est
restreinte. Si vous avez reçu ce courriel par erreur, vous ne devez ni le copier, ni l'utiliser, ni en divulguer le contenu à quiconque. Merci d'en avertir immédiatement l'expéditeur et d'effacer ce courriel de votre système. Airbus DS Geo décline toute responsabilité
en cas de corruption par virus, d'altération ou de falsification de ce courriel lors de sa transmission par voie électronique.<br>
<br>
This email (including any attachments) may contain confidential and/or privileged information or information otherwise protected from disclosure. If you are not the intended recipient, please notify the sender immediately, do not copy this message or any attachments
and do not use it for any purpose or disclose its content to any person, but delete this message and any attachments from your system. Airbus DS Geo disclaims any and all liability if this email transmission was virus corrupted, altered or falsified.</em></font></p>
<hr>
<p style="MARGIN-RIGHT: 0px" dir="ltr" class="{Imprint.UniqueID}"><font color="#999999" size="2" face="Verdana"><em>Airbus DS Geo SA (325 089 589 RCS Toulouse) - Siege social: 5, rue des Satellites, 31400 Toulouse, France.</em></font></p>
<p class="67050b90-a467-4d62-be36-e5ff8cdc9d4b"> </p>
<p class="67050b90-a467-4d62-be36-e5ff8cdc9d4b"></p>
</body>
</html>