[pdal] [pipeline] How to use the --progress option ?

Mon Mar 20 01:43:15 PDT 2017

Very simply, we have an interactive visualization application that can load heavy files (notably LAS files) in a background task to be displayed,  and the user can monitor the progress via a progress bar. This can take up to five minutes for 200 MB files, and currently our progress bar is stuck to a fixed number until the pipeline finishes, giving the impression that he process has stalled.

I believe the writers.gdal stage of the pipeline (readers.las ->  filters.reprojection -> writers.gdal) is the longest of the three, so ideally, the GDAL writer would report progress in the form of a percentage with increments of 1% (or more if there is too much overhead) would be excellent to avoid the stalled progress bar syndrome.

We could mark the beginning and end of the main loop (the one that iterates over all the points in the point view) with a special token “START_MAINLOOP”/”END_MAINLOOP” to warn the listening application that it can start counting progress:

READYFILE
START_MAINLOOP
1
2
..
100
END_MAINLOOP

This could be a standard formatting for all stages, so we could reliably monitor the progress of any pipeline.

Regards,
Sébastien

De : Andrew Bell [mailto:andrew.bell.ia at gmail.com]
Envoyé : vendredi 17 mars 2017 22:08
À : GUIMMARA, Sébastien (External)
Cc : pdal at lists.osgeo.org
Objet : Re: [pdal] [pipeline] How to use the --progress option ?

I've fixed this and added a test to hopefully prevent its breakage in the future:

https://github.com/PDAL/PDAL/issues/1534

As you found, there isn't much current support for this feature.  If you tell me a bit about what you're doing and what you're looking for, we can probably add more functionality.

Best,

On Fri, Mar 17, 2017 at 9:05 AM, Andrew Bell <andrew.bell.ia at gmail.com<mailto:andrew.bell.ia at gmail.com>> wrote:
On Thu, Mar 16, 2017 at 10:38 AM, GUIMMARA, Sébastien (External) <sebastien.guimmara.external at airbus.com<mailto:sebastien.guimmara.external at airbus.com>> wrote:
I am struggling with the --progress flag of the pipeline application. Namely, whether I create an actual file or a FIFO, nothing is ever written to it:

pdal pipeline pipeline.json --progress progress.txt

If I’m reading the source correctly, currently only the NITF and LAS writers use the Utils::writeProgress function. But even if I switch to a LAS output, the progress file is still empty.

I'll test this and write back as it hasn't been used much and I'm not sure we have good test coverage.

--
Andrew Bell
andrew.bell.ia at gmail.com<mailto:andrew.bell.ia at gmail.com>

--
Andrew Bell
andrew.bell.ia at gmail.com<mailto:andrew.bell.ia at gmail.com>

[cid:imagee71a82.JPG at 47ff91d7.459a58ed] Please consider the environment before printing this email message.

________________________________

Ce courriel (incluant ses éventuelles pièces jointes) peut contenir des informations confidentielles et/ou protégées ou dont la diffusion est restreinte. Si vous avez reçu ce courriel par erreur, vous ne devez ni le copier, ni l'utiliser, ni en divulguer le contenu à quiconque. Merci d'en avertir immédiatement l'expéditeur et d'effacer ce courriel de votre système. Airbus DS Geo décline toute responsabilité en cas de corruption par virus, d'altération ou de falsification de ce courriel lors de sa transmission par voie électronique.

This email (including any attachments) may contain confidential and/or privileged information or information otherwise protected from disclosure. If you are not the intended recipient, please notify the sender immediately, do not copy this message or any attachments and do not use it for any purpose or disclose its content to any person, but delete this message and any attachments from your system. Airbus DS Geo disclaims any and all liability if this email transmission was virus corrupted, altered or falsified.

________________________________

Airbus DS Geo SA (325 089 589 RCS Toulouse) - Siege social: 5, rue des Satellites, 31400 Toulouse, France.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pdal/attachments/20170320/2decc876/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: imagee71a82.JPG
Type: image/jpeg
Size: 625 bytes
Desc: imagee71a82.JPG
URL: <http://lists.osgeo.org/pipermail/pdal/attachments/20170320/2decc876/attachment.jpe>