[pdal] [EXTERNAL] Entwine build failure

[Kirk Waters - NOAA Federal]

Dear PDAL,
Are there any suggestions regarding what I might check or alternative
approaches? I tried building the EPT set again and it again died at 80%
done. This time it was outputting to a different disk and definitely did
not run out of inodes. Same errors that look like: Exception in pool task:
writers.las: Couldn't open file
'/san1/lidar4_p/entwine/geoid18/4800/ept-data/16-24448-21619-65505.laz' for
output. The input dataset has over 40K files. The docs for entwine build
suggest that hierarchyStep might be useful, but also says it shouldn't be
necessary. Would subset be a useful option? It's a little unclear how to
make that work.

Kirk


On Wed, Apr 14, 2021 at 9:04 AM Kirk Waters - NOAA Federal <
kirk.waters at noaa.gov> wrote:

> Doug,
> Linux box is doing the writing. Output disk in a storage area network
> device, so all I get for disk type is nfs. The inode limit is 21 million
> and I suspect a significant portion of that is already taken up with the
> more permanent disk contents.
>
> Kirk
>
>
> On Wed, Apr 14, 2021 at 8:16 AM Newcomb, Doug <doug_newcomb at fws.gov>
> wrote:
>
>> Kirk,
>> Ran across this. Assuming you are using a linux box to store.
>>
>> https://www.ctrl.blog/entry/how-to-all-out-of-inodes.html#:~:text=That%20default%20depends%20on%20your,16%20KiB%20of%20disk%20space.&text=If%20you've%20a%20small,3%20million%20files%20and%20directories.
>>
>> Doug
>> ------------------------------
>> *From:* pdal <pdal-bounces at lists.osgeo.org> on behalf of Kirk Waters -
>> NOAA Federal <kirk.waters at noaa.gov>
>> *Sent:* Wednesday, April 14, 2021 7:48 AM
>> *To:* pdal <pdal at lists.osgeo.org>
>> *Subject:* [EXTERNAL] [pdal] Entwine build failure
>>
>>
>>
>>
>> * This email has been received from outside of DOI - Use caution before
>> clicking on links, opening attachments, or responding.  *
>>
>>
>>
>> While I've been able to build most of my datasets with Entwine, I've run
>> into issues with one in particular. I don't know if it's related to the
>> number of input files, the geographic extent, or just the total volume.
>> After making it through 80% of the files, it starts getting an exception
>> that it can't open a file for output. This is after running for 128 hours
>> with 10 threads. The logs look like this where things start to go bad:
>>
>> Adding 31830 -
>> /san1/reef/lidar4_z/geoid18/data/4800/Block_010/20140122_166500e_3749500n.laz
>>         Done 31828
>> Adding 31831 -
>> /san1/reef/lidar4_z/geoid18/data/4800/Block_010/20140122_166000e_3749000n.laz
>> 128:20:20 - 80% - 201,421,974,813 - 1,569(1,131)M/h - 216W - 46R - 396A
>> 128:20:30 - 80% - 201,426,046,237 - 1,569(1,465)M/h - 0W - 87R - 605A
>> 128:20:40 - 80% - 201,430,682,909 - 1,569(1,669)M/h - 30W - 43R - 700A
>> Exception in pool task: writers.las: Couldn't open file
>> '/san1/dem1_z/entwine/geoid18/4800/ept-data/16-24378-21598-65505.laz' for
>> output.
>> Exception in pool task: writers.las: Couldn't open file
>> '/san1/dem1_z/entwine/geoid18/4800/ept-data/16-24379-21589-65505.laz' for
>> output.
>> Exception in pool task: writers.las: Couldn't open file
>> '/san1/dem1_z/entwine/geoid18/4800/ept-data/16-24378-21597-65505.laz' for
>> output.
>>
>> After that, it's all exceptions and exit with a -9 return code. It might
>> be an issue of the output directory size. The ept-data directory has a size
>> of over 335Mb and contains over 4 million files. It's possible that it's
>> taking so long to work with that directory that something times out.
>> Another possibility is that I'm running out of inodes on the output drive
>> (that directory is taking over 20% of the total inodes possible on the
>> drive). There's definitely space on the drive. This may not be an issue for
>> the software to solve. Is there a troubleshooting guide somewhere that I
>> missed?
>>
>> Kirk Waters, PhD, BJCP          | NOAA Office for Coastal Management
>> Applied Sciences Program      | 2234 South Hobson Ave
>> 843-740-1227 (empty office)   | Charleston, SC 29405
>> 843-324-2203 (cell during COVID)
>> coast.noaa.gov/digitalcoast
>> <https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fcoast.noaa.gov%2Fdigitalcoast&data=04%7C01%7Cdoug_newcomb%40fws.gov%7C6ad450199b9c44e2df5908d8ff3d4b89%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637539986664301396%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=4EJy6GLJWxgnVTw4dXqqAOneTShNQaPzwRV5I7VTVFw%3D&reserved=0>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pdal/attachments/20210426/ec5b8957/attachment.html>