[pdal] 'pdal info' VLR error troubleshooting

[Jim Klassen]

On 5/8/23 13:08, Howard Butler wrote:
>
>> On Apr 27, 2023, at 11:49 AM, Speed, Cole M <cole.speed at beg.utexas.edu> wrote:
>>
>> Hello all,
>>
>> I am using 'pdal info' to view the metadata for a few laz files. For several of the files, pdal info returns the error:
>>
>> PDAL: readers.las: VLR 1(SF/0) size too large -- flows into point data.
>>
>> I believe this may indicate that the header/VLR s longer than is expected, causing the metadata information to be misinterpreted. Attempting to troubleshoot the issue by having a look at the lasreader source code (https://github.com/PDAL/PDAL/blob/master/io/LasReader.cpp), I see the below code snippet:
>>          if ((uint64_t)stream->tellg() + vlr.promisedDataSize > d->header.pointOffset)
>>              throwError("VLR " + std::to_string(i + 1) +
>>                  "(" + vlr.userId + "/" + std::to_string(vlr.recordId) + ") "
>>                  "size too large -- flows into point data.");
>>
>> I interpret this to indicate an underlying issue related to the laz header/VLR, but I am unsure what specifically this means or how to resolve it. The files are also unreadable by QGIS (Error: 'readers.las: Invalid VLR - exceeds specified file range'). Insight from anyone who has encountered these errors or has any thoughts as to what specifically they imply about the header/VLR formatting of these laz files would be greatly appreciated. Thanks in advance!
> I cycled this issue with Cole and a couple things are going on...
>
> 1) His pipeline was writing a LAS 1.2 file, which is the default minor_version for writers.las
> 2) His input data had scan_angle values that were too big for LAS 1.2 even though it says it is a 1.2 file
> 3) PDAL was throwing an error when it encountered a scan angle that was too large and trying to write it in the output in stream mode
> 4) The resulting file was partially written and not flushed.
>
> His workaround is to use filters.assign to strip off the offending scan angles and everything works as expected.
>
> I'm not sure there's a fix for us though. We can't guarantee a properly written file in stream mode in this scenario. If you were to write data in --nostream mode, you would get a valid but partial file, but that takes a lot of memory and is slow for pipelines like this.
>
> Any ideas on how things should behave from anyone?
>
>

My preference is that PDAL exits with an error (causing $? <> 0) allowing the calling script/user to detect this and cleanup.  I suspect this is the current state of things.

If PDAL creates the output file and doesn't finish cleanly before the file is complete, maybe it would be safe for PDAL to delete the file.  Then the user wouldn't be left with a corrupt file, which might be a more obvious sign of an error.  I'm not sure how this would work for operations like tile that make many files or tindex that can append to an existing file, or to operations started from other programs using libpdal_*, or how this works if the output isn't to a local filesystem.

Just my 2¢.