<div dir="ltr">Dear PDAL,<div>Are there any suggestions regarding what I might check or alternative approaches? I tried building the EPT set again and it again died at 80% done. This time it was outputting to a different disk and definitely did not run out of inodes. Same errors that look like: Exception in pool task: writers.las: Couldn't open file '/san1/lidar4_p/entwine/geoid18/4800/ept-data/16-24448-21619-65505.laz' for output. The input dataset has over 40K files. The docs for entwine build suggest that hierarchyStep might be useful, but also says it shouldn't be necessary. Would subset be a useful option? It's a little unclear how to make that work. </div><div><br clear="all"><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Kirk</div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 14, 2021 at 9:04 AM Kirk Waters - NOAA Federal <<a href="mailto:kirk.waters@noaa.gov">kirk.waters@noaa.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Doug,<div>Linux box is doing the writing. Output disk in a storage area network device, so all I get for disk type is nfs. The inode limit is 21 million and I suspect a significant portion of that is already taken up with the more permanent disk contents.</div><div><br clear="all"><div><div dir="ltr"><div dir="ltr">Kirk</div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 14, 2021 at 8:16 AM Newcomb, Doug <<a href="mailto:doug_newcomb@fws.gov" target="_blank">doug_newcomb@fws.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
Kirk,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
Ran across this. Assuming you are using a linux box to store.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<a href="https://www.ctrl.blog/entry/how-to-all-out-of-inodes.html#:~:text=That%20default%20depends%20on%20your,16%20KiB%20of%20disk%20space.&text=If%20you've%20a%20small,3%20million%20files%20and%20directories." id="gmail-m_4102558017931943525gmail-m_1055422551637759559LPlnk392862" target="_blank">https://www.ctrl.blog/entry/how-to-all-out-of-inodes.html#:~:text=That%20default%20depends%20on%20your,16%20KiB%20of%20disk%20space.&text=If%20you've%20a%20small,3%20million%20files%20and%20directories.</a><br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
Doug</div>
<div id="gmail-m_4102558017931943525gmail-m_1055422551637759559appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_4102558017931943525gmail-m_1055422551637759559divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> pdal <<a href="mailto:pdal-bounces@lists.osgeo.org" target="_blank">pdal-bounces@lists.osgeo.org</a>> on behalf of Kirk Waters - NOAA Federal <<a href="mailto:kirk.waters@noaa.gov" target="_blank">kirk.waters@noaa.gov</a>><br>
<b>Sent:</b> Wednesday, April 14, 2021 7:48 AM<br>
<b>To:</b> pdal <<a href="mailto:pdal@lists.osgeo.org" target="_blank">pdal@lists.osgeo.org</a>><br>
<b>Subject:</b> [EXTERNAL] [pdal] Entwine build failure</font>
<div> </div>
</div>
<div>
<p> </p>
<div>
<table border="2" cellspacing="4" cellpadding="0" style="border-color:black;background-color:yellow;margin-left:auto;margin-right:auto">
<tbody>
<tr>
<td>
<p style="text-align:center"><strong> This email has been received from outside of DOI - Use caution before clicking on links, opening attachments, or responding. </strong></p>
</td>
</tr>
</tbody>
</table>
</div>
<p><br>
<br>
</p>
<div>
<div dir="ltr">While I've been able to build most of my datasets with Entwine, I've run into issues with one in particular. I don't know if it's related to the number of input files, the geographic extent, or just the total volume. After making it through 80%
of the files, it starts getting an exception that it can't open a file for output. This is after running for 128 hours with 10 threads. The logs look like this where things start to go bad:<br>
<br>
Adding 31830 - /san1/reef/lidar4_z/geoid18/data/4800/Block_010/20140122_166500e_3749500n.laz<br>
Done 31828<br>
Adding 31831 - /san1/reef/lidar4_z/geoid18/data/4800/Block_010/20140122_166000e_3749000n.laz<br>
128:20:20 - 80% - 201,421,974,813 - 1,569(1,131)M/h - 216W - 46R - 396A<br>
128:20:30 - 80% - 201,426,046,237 - 1,569(1,465)M/h - 0W - 87R - 605A<br>
128:20:40 - 80% - 201,430,682,909 - 1,569(1,669)M/h - 30W - 43R - 700A<br>
Exception in pool task: writers.las: Couldn't open file '/san1/dem1_z/entwine/geoid18/4800/ept-data/16-24378-21598-65505.laz' for output.<br>
Exception in pool task: writers.las: Couldn't open file '/san1/dem1_z/entwine/geoid18/4800/ept-data/16-24379-21589-65505.laz' for output.<br>
Exception in pool task: writers.las: Couldn't open file '/san1/dem1_z/entwine/geoid18/4800/ept-data/16-24378-21597-65505.laz' for output.<br>
<br>
After that, it's all exceptions and exit with a -9 return code. It might be an issue of the output directory size. The ept-data directory has a size of over 335Mb and contains over 4 million files. It's possible that it's taking so long to work with that directory
that something times out. Another possibility is that I'm running out of inodes on the output drive (that directory is taking over 20% of the total inodes possible on the drive). There's definitely space on the drive. This may not be an issue for the software
to solve. Is there a troubleshooting guide somewhere that I missed?
<div><br clear="all">
<div>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div><font face="arial, helvetica, sans-serif">Kirk Waters, PhD, BJCP | NOAA Office for Coastal Management<br>
</font></div>
<div><font face="arial, helvetica, sans-serif">Applied Sciences Program | 2234 South Hobson Ave<br>
843-740-1227 (empty office) | Charleston, SC 29405</font></div>
<div><font face="arial, helvetica, sans-serif">843-324-2203 (cell during COVID) </font>
<div><font face="arial, helvetica, sans-serif"><a href="https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fcoast.noaa.gov%2Fdigitalcoast&data=04%7C01%7Cdoug_newcomb%40fws.gov%7C6ad450199b9c44e2df5908d8ff3d4b89%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637539986664301396%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=4EJy6GLJWxgnVTw4dXqqAOneTShNQaPzwRV5I7VTVFw%3D&reserved=0" target="_blank">coast.noaa.gov/digitalcoast</a></font><br>
</div>
</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote></div>
</blockquote></div>