[pdal] [EXTERNAL] Entwine build failure

[Newcomb, Doug]

Kirk,
Ran across this. Assuming you are using a linux box to store.
https://www.ctrl.blog/entry/how-to-all-out-of-inodes.html#:~:text=That%20default%20depends%20on%20your,16%20KiB%20of%20disk%20space.&text=If%20you've%20a%20small,3%20million%20files%20and%20directories.

Doug
________________________________
From: pdal <pdal-bounces at lists.osgeo.org> on behalf of Kirk Waters - NOAA Federal <kirk.waters at noaa.gov>
Sent: Wednesday, April 14, 2021 7:48 AM
To: pdal <pdal at lists.osgeo.org>
Subject: [EXTERNAL] [pdal] Entwine build failure




 This email has been received from outside of DOI - Use caution before clicking on links, opening attachments, or responding.



While I've been able to build most of my datasets with Entwine, I've run into issues with one in particular. I don't know if it's related to the number of input files, the geographic extent, or just the total volume. After making it through 80% of the files, it starts getting an exception that it can't open a file for output. This is after running for 128 hours with 10 threads. The logs look like this where things start to go bad:

Adding 31830 - /san1/reef/lidar4_z/geoid18/data/4800/Block_010/20140122_166500e_3749500n.laz
        Done 31828
Adding 31831 - /san1/reef/lidar4_z/geoid18/data/4800/Block_010/20140122_166000e_3749000n.laz
128:20:20 - 80% - 201,421,974,813 - 1,569(1,131)M/h - 216W - 46R - 396A
128:20:30 - 80% - 201,426,046,237 - 1,569(1,465)M/h - 0W - 87R - 605A
128:20:40 - 80% - 201,430,682,909 - 1,569(1,669)M/h - 30W - 43R - 700A
Exception in pool task: writers.las: Couldn't open file '/san1/dem1_z/entwine/geoid18/4800/ept-data/16-24378-21598-65505.laz' for output.
Exception in pool task: writers.las: Couldn't open file '/san1/dem1_z/entwine/geoid18/4800/ept-data/16-24379-21589-65505.laz' for output.
Exception in pool task: writers.las: Couldn't open file '/san1/dem1_z/entwine/geoid18/4800/ept-data/16-24378-21597-65505.laz' for output.

After that, it's all exceptions and exit with a -9 return code. It might be an issue of the output directory size. The ept-data directory has a size of over 335Mb and contains over 4 million files. It's possible that it's taking so long to work with that directory that something times out. Another possibility is that I'm running out of inodes on the output drive (that directory is taking over 20% of the total inodes possible on the drive). There's definitely space on the drive. This may not be an issue for the software to solve. Is there a troubleshooting guide somewhere that I missed?

Kirk Waters, PhD, BJCP          | NOAA Office for Coastal Management
Applied Sciences Program      | 2234 South Hobson Ave
843-740-1227 (empty office)   | Charleston, SC 29405
843-324-2203 (cell during COVID)
coast.noaa.gov/digitalcoast<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fcoast.noaa.gov%2Fdigitalcoast&data=04%7C01%7Cdoug_newcomb%40fws.gov%7C6ad450199b9c44e2df5908d8ff3d4b89%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637539986664301396%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=4EJy6GLJWxgnVTw4dXqqAOneTShNQaPzwRV5I7VTVFw%3D&reserved=0>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pdal/attachments/20210414/1c4f6373/attachment.html>