<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<!-- Template generated by Exclaimer Mail Disclaimers on 09:47:46 lundi, 19 juin 2017 -->
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css">P.67050b90-a467-4d62-be36-e5ff8cdc9d4b {
MARGIN: 0cm 0cm 0pt
}
LI.67050b90-a467-4d62-be36-e5ff8cdc9d4b {
MARGIN: 0cm 0cm 0pt
}
DIV.67050b90-a467-4d62-be36-e5ff8cdc9d4b {
MARGIN: 0cm 0cm 0pt
}
TABLE.67050b90-a467-4d62-be36-e5ff8cdc9d4bTable {
MARGIN: 0cm 0cm 0pt
}
DIV.Section1 {
page: Section1
}
</style>
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="FR" link="blue" vlink="purple">
<p class="67050b90-a467-4d62-be36-e5ff8cdc9d4b"></p>
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Indeed, all the stages in my typical pipelines are not compatible with the --stream option (outlier, normal…).<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">My constraint is that I should be able to process any dataset between 50 and 1000MB (in uncompressed form) on a moderately powerful desktop machine,
not because we don’t have the processing power in servers, but essentially because the target machines could be used anywhere without any network connection.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I’m aiming to take this constraint into account by decimating the datasets until they fall into range of acceptable sizes. This decimation should
be done in a preliminary pass with the --stream option.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Sébastien<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">De :</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Andrew Bell [mailto:andrew.bell.ia@gmail.com]
<br>
<b>Envoyé :</b> vendredi 16 juin 2017 17:29<br>
<b>À :</b> Albert Godfrind<br>
<b>Cc :</b> GUIMMARA, Sébastien (External); pdal<br>
<b>Objet :</b> Re: [pdal] [Filters] Decimation built into a reader<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">On Fri, Jun 16, 2017 at 7:19 AM, Albert Godfrind <<a href="mailto:albert.godfrind@oracle.com" target="_blank">albert.godfrind@oracle.com</a>> wrote:<o:p></o:p></p>
<div>
<p class="MsoNormal">Ah OK. 6GB of LAZ would probably be equivalent to 75GB of LAS files. Still not a lot of data. Definitely something I would expect to process on a regular laptop without effort ...<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I’ve witnessed RAM usage of more than 38 GB (intensive swap usage) on my own dev box (8 cores, 16 GB RAM, SSD), and the process could not finish
because it ran out of memory. So extrapolating to the target machine, this is unbearable.</span><o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Looks like one of the processes in the pipeline requires loading the entire (uncompressed) dataset in memory ? Someone in the PDAL dev team may comment here. If so, then that is a bit of a worry regarding scalability ...<o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">There are many algorithms that require all the points to be loaded because they care about point locality. If you need to know the nearest neighbors of point X, and you have no idea where in the input dataset(s) the neighbor points are,
you have to load all the points or read the data multiple times. If you're doing heavy processing like this, it's not unreasonable to purchase hardware to support it. You can get a server with 512GB of ram for less than $2500, I think. Many people tile
their data in order to work around algorithmic limitations when working with very large datasets.<br>
<br>
If points are organized spatially, algorithms can take advantage to minimize memory usage and speed processing, but these algorithms are specialized for particular tasks and the arrangement of the data. This is a lot of work ($), and when hardware is cheap,
I'm not sure how valuable it is unless you're doing lots and lots of this kind of thing. Also, each algorithm may have differing requirements on data arrangement for optimal handling. LAStools triangulation, for example, makes assumptions about the data
and does at least three passes of the input in order to reduce memory requirements. It runs quickly, but it's very specific code for a particlar purpose. Even then, the benefits decay when moving from 2D to 3D.<br>
<br>
PDAL aims to be generic. If you're interested in pursuing algorithm development that reduces memory requirements, it's something that we can work on with funding, but I'm not sure where it sits on our priority list otherwise.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<p class="MsoNormal">Andrew Bell<br>
<a href="mailto:andrew.bell.ia@gmail.com" target="_blank">andrew.bell.ia@gmail.com</a><o:p></o:p></p>
</div>
</div>
</div>
</div>
<p></p>
<p class="67050b90-a467-4d62-be36-e5ff8cdc9d4b"> </p>
<font color="#008000" size="1" face="Verdana"><img border="0" hspace="0" alt="" align="baseline" src="cid:imageb48402.JPG@9d85f537.4590f770"> Please consider the environment before printing this email message.</font>
<p></p>
<hr>
<p style="MARGIN-RIGHT: 0px" dir="ltr" class="{Imprint.UniqueID}"><font color="#999999" size="2" face="Verdana"><em>Ce courriel (incluant ses éventuelles pièces jointes) peut contenir des informations confidentielles et/ou protégées ou dont la diffusion est
restreinte. Si vous avez reçu ce courriel par erreur, vous ne devez ni le copier, ni l'utiliser, ni en divulguer le contenu à quiconque. Merci d'en avertir immédiatement l'expéditeur et d'effacer ce courriel de votre système. Airbus DS Geo décline toute responsabilité
en cas de corruption par virus, d'altération ou de falsification de ce courriel lors de sa transmission par voie électronique.<br>
<br>
This email (including any attachments) may contain confidential and/or privileged information or information otherwise protected from disclosure. If you are not the intended recipient, please notify the sender immediately, do not copy this message or any attachments
and do not use it for any purpose or disclose its content to any person, but delete this message and any attachments from your system. Airbus DS Geo disclaims any and all liability if this email transmission was virus corrupted, altered or falsified.</em></font></p>
<hr>
<p style="MARGIN-RIGHT: 0px" dir="ltr" class="{Imprint.UniqueID}"><font color="#999999" size="2" face="Verdana"><em>Airbus DS Geo SA (325 089 589 RCS Toulouse) - Siege social: 5, rue des Satellites, 31400 Toulouse, France.</em></font></p>
<p class="67050b90-a467-4d62-be36-e5ff8cdc9d4b"> </p>
<p class="67050b90-a467-4d62-be36-e5ff8cdc9d4b"></p>
</body>
</html>