[pdal] Riegl extrabytes stripped

[Kristian Evers]

Andrew,

I think, in the case of version 1.1-1.3 LAS files, that the format of the extrabytes are not specified in the VLR header. It is simply not possible in the format specification, but it is still allowed to attach extra stuff at the end of a data record. How much extra stuff can be calculated from the information in the header about the record size, and the size of an un-altered record of a specific data format. I.e. knowing from the header that a record is 41 bytes and the data record point format 1 is 37 bytes we can infer that there must be 4 extra bytes in a record. What those bytes are is not specified in LAS files before version 1.4. And I guess that makes it hard for PDAL to understand how to treat those bytes, since there’s not a predefined dimension to put it in. Sorry if I am stating the obvious here, this stuff can be a bit confusing I think, so just trying make sure we agree on how to interpret the different LAS specifications.

The file in question has point format 1 so PDAL should be okay with it. Also this is not waveform data as specified in the LAS 1.3, it is just Riegl utilizing the extra-bytes feature to store some information related to waveforms. It could just as well have been different information stored.

You should be able to download the file from https://filkassen.statens-it.dk/mobile/links/public/home?token=sD240hC9uNeiPbjR&name=150327_103118.laz. Notice that it is a rather big file!
Output from pdal info --metadata for both input and output files can be seen here: https://gist.github.com/kbevers/f98f588c4fe67773b9a75aa7fce53bf0 

For the record, pdal --version says "pdal 1.3.0 (git-version: 258504)" build on windows with the latest version of visual studio.

Pete,

I notice the same description as you. That was indeed confusing. The same for pdal info -p 0. Which makes it hard to check if the extrabytes data is actually there. My input file is 1.1GB and the output is 650MB. Either PDAL compresses the data a lot better than Riegls software, or something is missing...

Thank you both for your help. I am less convinced know than before that there actually is a problem here. It might just be that I am trying to do something that PDAL was never meant to be able to do.

/Kristian

> -----Oprindelig meddelelse-----
> Fra: Pete Gadomski [mailto:pete.gadomski at gmail.com]
> Sendt: 3. oktober 2016 14:40
> Til: Andrew Bell
> Cc: Kristian Evers; pdal at lists.osgeo.org
> Emne: Re: [pdal] Riegl extrabytes stripped
> 
> Andrew,
> 
> Here's a 43 point las file w/ Riegl's extra bytes that I use for
> developing against this stuff. I was *not* able to reproduce the OP's
> issue — `pdal translate -i 1.2_1_extra_bytes.las -o out.laz
> --writers.las.forward=vlr --writers.las.extra_dims=all && pdal info
> --metadata out.laz` reported the presence of the extra bytes vlr, and
> the extra bytes were present on each point. However, `pdal info -p 0
> out.laz` did not report the values stored in those extra bytes
> (Reflectance, Amplitude, and Deviation) — not sure if that's supported
> ATM.
> 
> The VLR description is also changed from "RIEGL Extra Bytes" to "Extra
> Bytes Record", not a big deal but may surprise some end-users who
> expect a "forward" option to forward without change.
> 
> HTH,
> Pete
> 
> On Mon, Oct 3, 2016 at 6:25 AM, Andrew Bell <andrew.bell.ia at gmail.com>
> wrote:
> > On Mon, Oct 3, 2016 at 6:28 AM, Kristian Evers <kreve at sdfe.dk> wrote:
> >>
> >> Hi.
> >>
> >>
> >>
> >> I have some laz-files that incoorporates Riegl’s extrabytes waveform
> >> extension [0]. I am running the files through a PDAL pipeline but
> >> unfortunately the VLR header and extrabytes are stripped from the
> resulting
> >> files. This behavior of writers.las is documented on the webpage [1]. The
> >> forward options description ends with:
> >>
> >>
> >>
> >> “VLRs can be forwarded by using the special value ‘vlr’. VLRs containing
> >> the following User IDs are NOT forwarded: ‘LASF_Projection’,
> ‘LASF_Spec’,
> >> ‘liblas’, ‘laszip encoded’. These VLRs are known to contain information
> >> regarding the formatting of the data and will be rebuilt properly in the
> >> output file as necessary. Unlike header values, VLRs from multiple input
> >> files are accumulated and each is written to the output file. Forwarded
> VLRs
> >> may contain duplicate User ID/Record ID pairs.”
> >>
> >>
> >>
> >> The Riegl extrabytes are stored with user ID “LASF_Spec” and record ID 4.
> >> I accept that it is necessary to treat some VLR’s differently than others,
> >> but I don’t think this is one of them.
> >
> >
> > The LASF_Spec is defined in the LAS 1.4 specification and needs special
> > handling by PDAL.  In order to support LAS 1.4, PDAL needs to write
> > arbitrary dimensions to the output as extra bytes and also needs to be able
> > to construct the LASF_Spec VLR to reflect the data in the extra bytes.
> >
> > Without a sample data file, I really don't know what's going on or why the
> > data isn't being forwarded as expected.
> >
> >>
> >> The extra-bytes in this case contains data that PDAL has no way of
> knowing
> >> how to reconstruct properly.
> >
> >
> > I don't believe this is true.  The VLR should describe the data on input,
> > which should allow it to also properly reconstructed on output.
> >
> >>
> >> I believe something similar happens when storing waveform data as
> >> specified in LAS v. 1.3 since it is also defined as VLR’s with user ID
> >> “LASF_Spec” (and record ID in the interval 100-355).
> >
> >
> > Waveform data is explicitly not supported by PDAL (we don't support point
> > formats 4, 5, 9 and 10).
> >
> >> I would much prefer the extrabytes attached to each record just be
> >> forwarded like any other VLR and extrabytes data. A way to do this would
> be
> >> to only ignore VLRs with certain user ID’s that matches data that PDAL
> knows
> >> how to construct properly at write-time.
> >
> >
> > PDAL is handling formats other than LAS.  The current handling allows data
> > read from some other format to be properly written to a LAS 1.4 file, using
> > the LASF_Spec VLR and extra bytes to store data not supported by the
> native
> > LAS point formats.  In order to diagnose your issue, I need to examine the
> > data in more detail.
> >
> > --
> > Andrew Bell
> > andrew.bell.ia at gmail.com
> >
> > _______________________________________________
> > pdal mailing list
> > pdal at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/pdal