[pdal] Entwine build failure

[Kirk Waters - NOAA Federal]

While I've been able to build most of my datasets with Entwine, I've run
into issues with one in particular. I don't know if it's related to the
number of input files, the geographic extent, or just the total volume.
After making it through 80% of the files, it starts getting an exception
that it can't open a file for output. This is after running for 128 hours
with 10 threads. The logs look like this where things start to go bad:

Adding 31830 -
/san1/reef/lidar4_z/geoid18/data/4800/Block_010/20140122_166500e_3749500n.laz
        Done 31828
Adding 31831 -
/san1/reef/lidar4_z/geoid18/data/4800/Block_010/20140122_166000e_3749000n.laz
128:20:20 - 80% - 201,421,974,813 - 1,569(1,131)M/h - 216W - 46R - 396A
128:20:30 - 80% - 201,426,046,237 - 1,569(1,465)M/h - 0W - 87R - 605A
128:20:40 - 80% - 201,430,682,909 - 1,569(1,669)M/h - 30W - 43R - 700A
Exception in pool task: writers.las: Couldn't open file
'/san1/dem1_z/entwine/geoid18/4800/ept-data/16-24378-21598-65505.laz' for
output.
Exception in pool task: writers.las: Couldn't open file
'/san1/dem1_z/entwine/geoid18/4800/ept-data/16-24379-21589-65505.laz' for
output.
Exception in pool task: writers.las: Couldn't open file
'/san1/dem1_z/entwine/geoid18/4800/ept-data/16-24378-21597-65505.laz' for
output.

After that, it's all exceptions and exit with a -9 return code. It might be
an issue of the output directory size. The ept-data directory has a size of
over 335Mb and contains over 4 million files. It's possible that it's
taking so long to work with that directory that something times out.
Another possibility is that I'm running out of inodes on the output drive
(that directory is taking over 20% of the total inodes possible on the
drive). There's definitely space on the drive. This may not be an issue for
the software to solve. Is there a troubleshooting guide somewhere that I
missed?

Kirk Waters, PhD, BJCP          | NOAA Office for Coastal Management
Applied Sciences Program      | 2234 South Hobson Ave
843-740-1227 (empty office)   | Charleston, SC 29405
843-324-2203 (cell during COVID)
coast.noaa.gov/digitalcoast
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pdal/attachments/20210414/82af7a6b/attachment.html>