[pdal] [Filters] Decimation built into a reader

Fri Jun 16 05:19:44 PDT 2017

Ah OK. 6GB of LAZ would probably be equivalent to 75GB of LAS files. Still not a lot of data. Definitely something I would expect to process on a regular laptop without effort ...

> I’ve witnessed RAM usage of more than 38 GB (intensive swap usage) on my own dev box (8 cores, 16 GB RAM, SSD), and the process could not finish because it ran out of memory. So extrapolating to the target machine, this is unbearable.

Looks like one of the processes in the pipeline requires loading the entire (uncompressed) dataset in memory ? Someone in the PDAL dev team may comment here. If so, then that is a bit of a worry regarding scalability ...

Albert

--
 <http://www.oracle.com/>
Albert Godfrind | Geospatial and Graph technologies | Tel: +33 4 93 00 80 67 | Mobile: +33 6 09 97 27 23 | Twitter: @agodfrin
Oracle Server Technologies
Centre Régus - Les Algorithmes - Bâtiment ARISTOTE A | 2000 Route des Lucioles | 06410 Sophia Antipolis | France
Everything you ever wanted to know about Oracle Spatial <http://www.apress.com/9781590598993>
> On 16-Jun-2017, at 12:18, GUIMMARA, Sébastien (External) <sebastien.guimmara.external at airbus.com> wrote:
> 
> 
> Albert,
>  
> My goal is to be able to convert datasets on average desktop computers (4 core/8 GB RAM, spinning drives) without blowing memory requirements, so that they can be quickly displayed in a virtual globe.
>  
> The dataset is indeed 6 GB of LAZ files. The goal is to process it with PDAL (outlier, range, colorinterp/colorize, normal), then tile it to 3D Tiles with Entwine.
>  
> I’ve witnessed RAM usage of more than 38 GB (intensive swap usage) on my own dev box (8 cores, 16 GB RAM, SSD), and the process could not finish because it ran out of memory. So extrapolating to the target machine, this is unbearable.
>  
> For now, my workflow to circumvent this would  be to decimate each LAZ file in a separate pipeline prior to merge them and go on with the filters.
>  
> Sébastien
> 
>  
> <imagee88352.JPG> Please consider the environment before printing this email message.
> 
> Ce courriel (incluant ses éventuelles pièces jointes) peut contenir des informations confidentielles et/ou protégées ou dont la diffusion est restreinte. Si vous avez reçu ce courriel par erreur, vous ne devez ni le copier, ni l'utiliser, ni en divulguer le contenu à quiconque. Merci d'en avertir immédiatement l'expéditeur et d'effacer ce courriel de votre système. Airbus DS Geo décline toute responsabilité en cas de corruption par virus, d'altération ou de falsification de ce courriel lors de sa transmission par voie électronique.
> 
> This email (including any attachments) may contain confidential and/or privileged information or information otherwise protected from disclosure. If you are not the intended recipient, please notify the sender immediately, do not copy this message or any attachments and do not use it for any purpose or disclose its content to any person, but delete this message and any attachments from your system. Airbus DS Geo disclaims any and all liability if this email transmission was virus corrupted, altered or falsified.
> 
> Airbus DS Geo SA (325 089 589 RCS Toulouse) - Siege social: 5, rue des Satellites, 31400 Toulouse, France.
> 
>  
> 
> De : Albert Godfrind [mailto:albert.godfrind at oracle.com <mailto:albert.godfrind at oracle.com>] 
> Envoyé : vendredi 16 juin 2017 11:40
> À : GUIMMARA, Sébastien (External)
> Cc : pdal
> Objet : Re: [pdal] [Filters] Decimation built into a reader
>  
> Sébastien,
>  
> I would not exactly qualify 6GB of LAS data as massive. Do you really mean 6GB ? Or do you mean 6TB ?
>  
> When you say you want to import the data, what are you importing into ? A database ? Another file structure ?
>  
> my desktop computer simply couldn’t handle the memory footprint of such a massive dataset.
>  
> What kind of desktop are you using that cannot cope with such a small file ? (unless of course you really mean 6TB).
>  
> Albert
>  
> --
> <~WRD000.jpg> <http://www.oracle.com/>
> Albert Godfrind | Geospatial and Graph technologies | Tel: +33 4 93 00 80 67 | Mobile: +33 6 09 97 27 23 | Twitter: @agodfrin
> Oracle Server Technologies
> Centre Régus - Les Algorithmes - Bâtiment ARISTOTE A | 2000 Route des Lucioles | 06410 Sophia Antipolis | France
> Everything you ever wanted to know about Oracle Spatial <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.apress.com_9781590598993&d=DwMGaQ&c=RoP1YumCXCgaWHvlZYR8PQcxBKCX5YTpkKY057SbK10&r=1qY34HIjMdHQKENpnLMMP4yLfmc3rd-OMDuH2zkUKUk&m=ouipdry-XnkgLWUH6l0Sxec3Oe19MR_IJdXFdvNJXnQ&s=F9wB40TvYdZ5RS04es9Qg8B2HoM8ChmIb2AqQsXZ-Oc&e=>
>  
> On 15-Jun-2017, at 17:18, GUIMMARA, Sébastien (External) <sebastien.guimmara.external at airbus.com <mailto:sebastien.guimmara.external at airbus.com>> wrote:
>  
>  
> Hello everyone,
>  
> I’m trying to import a massive point cloud as a collection of LAS files (6GB, 27 files), and I realized that my desktop computer simply couldn’t handle the memory footprint of such a massive dataset.
>  
> So I’m putting a decimation filter to reduce the size, but I’m having a doubt: is the filter useful here or is the dataset entirely loaded into memory before even applying the filter ? In other words, does the downstream filters controls how the upstream reader loads data into memory ?
>  
> If not, would it be a nice feature to include to some (all ?) readers ? Like a builtin decimation/sampling filter.
>  
> Sébastien
>  
>  
> <imageed0be7.JPG> Please consider the environment before printing this email message.
>  
> Ce courriel (incluant ses éventuelles pièces jointes) peut contenir des informations confidentielles et/ou protégées ou dont la diffusion est restreinte. Si vous avez reçu ce courriel par erreur, vous ne devez ni le copier, ni l'utiliser, ni en divulguer le contenu à quiconque. Merci d'en avertir immédiatement l'expéditeur et d'effacer ce courriel de votre système. Airbus DS Geo décline toute responsabilité en cas de corruption par virus, d'altération ou de falsification de ce courriel lors de sa transmission par voie électronique.
> 
> This email (including any attachments) may contain confidential and/or privileged information or information otherwise protected from disclosure. If you are not the intended recipient, please notify the sender immediately, do not copy this message or any attachments and do not use it for any purpose or disclose its content to any person, but delete this message and any attachments from your system. Airbus DS Geo disclaims any and all liability if this email transmission was virus corrupted, altered or falsified.
> Airbus DS Geo SA (325 089 589 RCS Toulouse) - Siege social: 5, rue des Satellites, 31400 Toulouse, France.
>  
>  
> _______________________________________________
> pdal mailing list
> pdal at lists.osgeo.org <mailto:pdal at lists.osgeo.org>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.osgeo.org_mailman_listinfo_pdal&d=DwIGaQ&c=RoP1YumCXCgaWHvlZYR8PQcxBKCX5YTpkKY057SbK10&r=1qY34HIjMdHQKENpnLMMP4yLfmc3rd-OMDuH2zkUKUk&m=NWJCp7FhoWF0tKyD1cuiQ-1r9WWtvua_djBMq1U_COM&s=sMYfELWOL7r58JcBdgbpRVlMWJw201xHLGSr7_4JYpY&e= <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.osgeo.org_mailman_listinfo_pdal&d=DwIGaQ&c=RoP1YumCXCgaWHvlZYR8PQcxBKCX5YTpkKY057SbK10&r=1qY34HIjMdHQKENpnLMMP4yLfmc3rd-OMDuH2zkUKUk&m=NWJCp7FhoWF0tKyD1cuiQ-1r9WWtvua_djBMq1U_COM&s=sMYfELWOL7r58JcBdgbpRVlMWJw201xHLGSr7_4JYpY&e=>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pdal/attachments/20170616/01e2babe/attachment-0001.html>