[pdal] Can we drop dimensions to save memory ?

Wed May 24 07:58:43 PDT 2017

I think the third option (converting to a more flexible format first) is the best, I’m going to test it.

Thanks very much,
Sébastien

De : Andrew Bell [mailto:andrew.bell.ia at gmail.com]
Envoyé : mercredi 24 mai 2017 16:16
À : GUIMMARA, Sébastien (External)
Cc : pdal at lists.osgeo.org
Objet : Re: [pdal] Can we drop dimensions to save memory ?

On Wed, May 24, 2017 at 8:31 AM, GUIMMARA, Sébastien (External) <sebastien.guimmara.external at airbus.com<mailto:sebastien.guimmara.external at airbus.com>> wrote:
Hello,

I am trying to process a 400 MB LAS file of Mount Rainier downloaded from Open Topography:

https://cloud.sdsc.edu/v1/AUTH_opentopography/PC_Bulk/Rainier/Q46122G12.laz

The dataset contains 82M points.

Firstly, I come across this warning: “Found invalid value of ‘6’ for points’ return number”. Is this a serious issue ?

No.  It's just telling you that the value is invalid, according to the spec.

 Secondly, processing this dataset through a pipeline that contains filter.colorinterp, filter.normal, and filter.programmable consumes huge amounts of memory (15 GB).

Is it possible to strip the PointView of all but the essential dimensions (X, Y, Z) in the pipeline to save memory ?

This has come up from time.  We may be able to provide this functionality from the command-line, but I don't know that we've given it much thought lately.  The issue is for you exacerbated because a) you use filters.normal, which creates an additional X/Y/Z dimension for each point to store the normal data and  b) using Python currently forces data to get copied because the internal format of data in PDAL won't easily map directly to numpy arrays (it's not always stored as contiguous data).  There may be a way around this, but I don't know that we've spent much time investigating.

There are a couple of issues/possibilities.

1) PDAL supports "stream" mode, which only loads a certain number of points at a time and processes them through the pipeline in batches.  However, some filters don't support stream mode because they need all the data in order to work (think sorting).  In your case, neither filters.normal nor filters.programmable supports stream mode.

2) If you're a programmer, you can create a custom point layout that rejects dimension registration other than the ones that you want.  All you really need to do is to create a subclass of PointLayout and reimplement the update() function to return "false" for all dimensions that you're not interested in.  I can provide an example, but if you're not programming C++, this isn't really an option for you.

3) You could convert the data to another format that doesn't have a set data layout (LAS point layouts are fixed).  You could try something like:

$ pdal pipeline --stream <mypipeline>

mypipeline:

{
  "pipeline":[
    "Q46122g12.laz",
    {
        "type":"writers.bpf",
        "filename":"Q46122g12.bpf",
        "output_dims":"X,Y,Z"
    }
  ]
}

This *untested* example should create a BPF file with only X,Y and Z in it.  You can then use that as input to your pipeline.  Note that internally X,Y and Z are still 8 bytes internally in PDAL, as are the normal vector X, Y and Z, so you've got a minimum of 48 bytes per point no matter what, plus the copies for Python.

Hope this helps a little.

--
Andrew Bell
andrew.bell.ia at gmail.com<mailto:andrew.bell.ia at gmail.com>

[cid:image006e10.JPG at 00aafa0a.47ac9e91] Please consider the environment before printing this email message.

________________________________

Ce courriel (incluant ses éventuelles pièces jointes) peut contenir des informations confidentielles et/ou protégées ou dont la diffusion est restreinte. Si vous avez reçu ce courriel par erreur, vous ne devez ni le copier, ni l'utiliser, ni en divulguer le contenu à quiconque. Merci d'en avertir immédiatement l'expéditeur et d'effacer ce courriel de votre système. Airbus DS Geo décline toute responsabilité en cas de corruption par virus, d'altération ou de falsification de ce courriel lors de sa transmission par voie électronique.

This email (including any attachments) may contain confidential and/or privileged information or information otherwise protected from disclosure. If you are not the intended recipient, please notify the sender immediately, do not copy this message or any attachments and do not use it for any purpose or disclose its content to any person, but delete this message and any attachments from your system. Airbus DS Geo disclaims any and all liability if this email transmission was virus corrupted, altered or falsified.

________________________________

Airbus DS Geo SA (325 089 589 RCS Toulouse) - Siege social: 5, rue des Satellites, 31400 Toulouse, France.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pdal/attachments/20170524/12b563d3/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image006e10.JPG
Type: image/jpeg
Size: 625 bytes
Desc: image006e10.JPG
URL: <http://lists.osgeo.org/pipermail/pdal/attachments/20170524/12b563d3/attachment-0001.jpe>