<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
I use the following general structure in many of my scripts:<br>
<br>
#!/bin/bash<br>
set -euo pipefail # exit the script if a command doesn't return
success<br>
pdal .... --writers.las.filename=$out.tmp.laz<br>
mv $out.tmp.laz $out.laz<br>
<br>
<br>
This has an advantage that I can place the temporary file either
next to where the final result should be, or I can put it in a
faster scratch space (usually a local SSD or ramdisk instead of
network storage) which can help because it is usually most efficient
to write to the network in large contiguous blocks instead of
piecemeal. Which way is optimal depends on the system and
processing task at hand. To me this is much less surprising
behavior than pdal writing to some other file behind my back.<br>
<br>
<br>
Alternatively, I have also used GNU make to drive the processing as
it can automatically remove the output file if the command that was
supposed to generate it fails. For this to work the output file has
to be called for by a make rule and having set ".DELETE_ON_ERROR".<br>
<br>
<div class="moz-cite-prefix">On 11/10/20 6:22 AM, Andrew Bell wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CACJ51z3dZC4X7BXcArdtxa716zgjbf8Esj=wkh9qyoaqubL1rQ@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div><br>
</div>
My take on this is that it's expensive to write to a temporary
file and then copy it. Some output files very are large. Since
you, the user, are the one interrupting the process, it seems
that it's up to you to clean up. This behavior is also
consistent with most other programs.</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, Nov 10, 2020 at 5:50
AM Peder Axensten <<a href="mailto:Peder.Axensten@slu.se"
moz-do-not-send="true">Peder.Axensten@slu.se</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi!<br>
<br>
If pdal is interrupted while saving to the destination file,
it might result in a corrupt file. This is not unreasonable,
but could be avoided.<br>
<br>
We use a make script to process large amounts of files and
sometimes we have to interrupt processing for different
reasons. We then risk to have corrupt files that make will
consider final when rerunning the script, so the files are
left in a corrupt state.<br>
<br>
Would it be a good idea to make pdal by default save the
contents to a temporary file and then move the temporary file
to the destination file? This way either a correct file is
produced or nothing. I'm implementing this in the make script
– it is somewhat cumbersome but will work ok, I guess.<br>
<br>
Isn’t it a very attractive and useful guarantee: if the
destination file is produced, then it is correct?<br>
<br>
Best regards,<br>
<br>
Peder Axensten<br>
Research engineer<br>
<br>
Remote Sensing<br>
Department of Forest Resource Management<br>
Swedish University of Agricultural Sciences<br>
SE-901 83 Umeå<br>
Visiting address: Skogsmarksgränd<br>
Phone: +46 90 786 85 00<br>
<a href="mailto:peder.axensten@slu.se" target="_blank"
moz-do-not-send="true">peder.axensten@slu.se</a>, <a
href="http://www.slu.se/srh" rel="noreferrer"
target="_blank" moz-do-not-send="true">www.slu.se/srh</a><br>
<br>
The Department of Forest Resource Management is
environmentally certified in accordance with ISO 14001.<br>
<br>
---<br>
När du skickar e-post till SLU så innebär detta att SLU
behandlar dina personuppgifter. För att läsa mer om hur detta
går till, klicka här <<a
href="https://www.slu.se/om-slu/kontakta-slu/personuppgifter/"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://www.slu.se/om-slu/kontakta-slu/personuppgifter/</a>><br>
E-mailing SLU will result in SLU processing your personal
data. For more information on how this is done, click here
<<a
href="https://www.slu.se/en/about-slu/contact-slu/personal-data/"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://www.slu.se/en/about-slu/contact-slu/personal-data/</a>><br>
_______________________________________________<br>
pdal mailing list<br>
<a href="mailto:pdal@lists.osgeo.org" target="_blank"
moz-do-not-send="true">pdal@lists.osgeo.org</a><br>
<a href="https://lists.osgeo.org/mailman/listinfo/pdal"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://lists.osgeo.org/mailman/listinfo/pdal</a></blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="gmail_signature">Andrew Bell<br>
<a href="mailto:andrew.bell.ia@gmail.com" target="_blank"
moz-do-not-send="true">andrew.bell.ia@gmail.com</a></div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
pdal mailing list
<a class="moz-txt-link-abbreviated" href="mailto:pdal@lists.osgeo.org">pdal@lists.osgeo.org</a>
<a class="moz-txt-link-freetext" href="https://lists.osgeo.org/mailman/listinfo/pdal">https://lists.osgeo.org/mailman/listinfo/pdal</a></pre>
</blockquote>
<br>
</body>
</html>