[pdal] [Filters] Decimation built into a reader

Fri Jun 16 03:18:25 PDT 2017

Albert,

My goal is to be able to convert datasets on average desktop computers (4 core/8 GB RAM, spinning drives) without blowing memory requirements, so that they can be quickly displayed in a virtual globe.

The dataset is indeed 6 GB of LAZ files. The goal is to process it with PDAL (outlier, range, colorinterp/colorize, normal), then tile it to 3D Tiles with Entwine.

I’ve witnessed RAM usage of more than 38 GB (intensive swap usage) on my own dev box (8 cores, 16 GB RAM, SSD), and the process could not finish because it ran out of memory. So extrapolating to the target machine, this is unbearable.

For now, my workflow to circumvent this would  be to decimate each LAZ file in a separate pipeline prior to merge them and go on with the filters.

Sébastien

[cid:imagee88352.JPG at b307f4bb.488e45c1] Please consider the environment before printing this email message.

________________________________

Ce courriel (incluant ses éventuelles pièces jointes) peut contenir des informations confidentielles et/ou protégées ou dont la diffusion est restreinte. Si vous avez reçu ce courriel par erreur, vous ne devez ni le copier, ni l'utiliser, ni en divulguer le contenu à quiconque. Merci d'en avertir immédiatement l'expéditeur et d'effacer ce courriel de votre système. Airbus DS Geo décline toute responsabilité en cas de corruption par virus, d'altération ou de falsification de ce courriel lors de sa transmission par voie électronique.

This email (including any attachments) may contain confidential and/or privileged information or information otherwise protected from disclosure. If you are not the intended recipient, please notify the sender immediately, do not copy this message or any attachments and do not use it for any purpose or disclose its content to any person, but delete this message and any attachments from your system. Airbus DS Geo disclaims any and all liability if this email transmission was virus corrupted, altered or falsified.

________________________________

Airbus DS Geo SA (325 089 589 RCS Toulouse) - Siege social: 5, rue des Satellites, 31400 Toulouse, France.

De : Albert Godfrind [mailto:albert.godfrind at oracle.com]
Envoyé : vendredi 16 juin 2017 11:40
À : GUIMMARA, Sébastien (External)
Cc : pdal
Objet : Re: [pdal] [Filters] Decimation built into a reader

Sébastien,

I would not exactly qualify 6GB of LAS data as massive. Do you really mean 6GB ? Or do you mean 6TB ?

When you say you want to import the data, what are you importing into ? A database ? Another file structure ?

my desktop computer simply couldn’t handle the memory footprint of such a massive dataset.

What kind of desktop are you using that cannot cope with such a small file ? (unless of course you really mean 6TB).

Albert

--
[Image supprimée par l'expéditeur. ORACLE]<http://www.oracle.com>
Albert Godfrind | Geospatial and Graph technologies | Tel: +33 4 93 00 80 67 | Mobile: +33 6 09 97 27 23 | Twitter: @agodfrin
Oracle Server Technologies
Centre Régus - Les Algorithmes - Bâtiment ARISTOTE A | 2000 Route des Lucioles | 06410 Sophia Antipolis | France
Everything you ever wanted to know about Oracle Spatial<http://www.apress.com/9781590598993>

On 15-Jun-2017, at 17:18, GUIMMARA, Sébastien (External) <sebastien.guimmara.external at airbus.com<mailto:sebastien.guimmara.external at airbus.com>> wrote:

Hello everyone,

I’m trying to import a massive point cloud as a collection of LAS files (6GB, 27 files), and I realized that my desktop computer simply couldn’t handle the memory footprint of such a massive dataset.

So I’m putting a decimation filter to reduce the size, but I’m having a doubt: is the filter useful here or is the dataset entirely loaded into memory before even applying the filter ? In other words, does the downstream filters controls how the upstream reader loads data into memory ?

If not, would it be a nice feature to include to some (all ?) readers ? Like a builtin decimation/sampling filter.

Sébastien

<imageed0be7.JPG> Please consider the environment before printing this email message.

________________________________
Ce courriel (incluant ses éventuelles pièces jointes) peut contenir des informations confidentielles et/ou protégées ou dont la diffusion est restreinte. Si vous avez reçu ce courriel par erreur, vous ne devez ni le copier, ni l'utiliser, ni en divulguer le contenu à quiconque. Merci d'en avertir immédiatement l'expéditeur et d'effacer ce courriel de votre système. Airbus DS Geo décline toute responsabilité en cas de corruption par virus, d'altération ou de falsification de ce courriel lors de sa transmission par voie électronique.

This email (including any attachments) may contain confidential and/or privileged information or information otherwise protected from disclosure. If you are not the intended recipient, please notify the sender immediately, do not copy this message or any attachments and do not use it for any purpose or disclose its content to any person, but delete this message and any attachments from your system. Airbus DS Geo disclaims any and all liability if this email transmission was virus corrupted, altered or falsified.
________________________________
Airbus DS Geo SA (325 089 589 RCS Toulouse) - Siege social: 5, rue des Satellites, 31400 Toulouse, France.

_______________________________________________
pdal mailing list
pdal at lists.osgeo.org<mailto:pdal at lists.osgeo.org>
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.osgeo.org_mailman_listinfo_pdal&d=DwIGaQ&c=RoP1YumCXCgaWHvlZYR8PQcxBKCX5YTpkKY057SbK10&r=1qY34HIjMdHQKENpnLMMP4yLfmc3rd-OMDuH2zkUKUk&m=NWJCp7FhoWF0tKyD1cuiQ-1r9WWtvua_djBMq1U_COM&s=sMYfELWOL7r58JcBdgbpRVlMWJw201xHLGSr7_4JYpY&e=

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pdal/attachments/20170616/4d6e06f5/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ~WRD000.jpg
Type: image/jpeg
Size: 823 bytes
Desc: ~WRD000.jpg
URL: <http://lists.osgeo.org/pipermail/pdal/attachments/20170616/4d6e06f5/attachment-0001.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: imagee88352.JPG
Type: image/jpeg
Size: 625 bytes
Desc: imagee88352.JPG
URL: <http://lists.osgeo.org/pipermail/pdal/attachments/20170616/4d6e06f5/attachment-0001.jpe>