<div dir="ltr"><div><div><div>Hi Piero<br><br></div>I'm watching your questions with interest - many have been on my mind also! <br><br></div>...did your second proposal (run 400 times) work?<br><br>that would, on the surface, use less memory since you're reading from one las file at a time rather than (400/64) las files (potentially, assuming a lot about how the data are distributed in space). ...but would also mean partial writing of each entwine chunk, which will eventually contain data from potentially (400/64) of your files... <br><br></div><div>...so the question there is 'can entwine support partial writing of subsets'?<br></div><div><br><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, 14 Jun 2019 at 01:57, Piero Toffanin <<a href="mailto:pt@masseranolabs.com">pt@masseranolabs.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Thanks, suspected that was the case but wanted to confirm.</p>
<p>In regard to building subsets, is there an advantage to using
"entwine scan" vs. the input files directly to "entwine build" in
terms of performance (or is scan a simple utility to simplify
finding datasets within a folder)?</p>
<p>Are there any tips or tricks that I should be aware of in terms
of memory usage when building using subset? For example, is it
memory efficient to do:</p>
<p>entwine build -i 1.las 2.las [...] 399.las 400.las --subset 1 64
-o out1</p>
<p>?</p>
<p>As compared to perhaps running 400 times:</p>
<p>entwine build -i 1.las 2.las [...] 399.las 400.las --subset 1 64
-o out1 --run 1</p>
<p>?<br>
</p>
<p>Sorry for all the questions!<br>
</p>
<div class="gmail-m_-7620134373879682099moz-cite-prefix">On 6/13/19 11:39 AM, Connor Manning
wrote:<br>
</div>
<blockquote type="cite" id="gmail-m_-7620134373879682099mid_CAO_FyjJ8b8onyeebg2uhDkeOXu4azuH_05ZTLzxZ0Um4mKTs0g_mail_gmail_com" class="gmail-m_-7620134373879682099cite">
<div dir="ltr">Correct - that is not possible.</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, Jun 13, 2019 at 10:16
AM Piero Toffanin <<a href="mailto:pt@masseranolabs.com" target="_blank">pt@masseranolabs.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote gmail-m_-7620134373879682099cite" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" id="gmail-m_-7620134373879682099Cite_7749447">
<div bgcolor="#FFFFFF">
<p>Hey Connor,</p>
<p>thanks for the reply. I have looked at the subset option
and I think it would work well for the case where I have
already computed all the models. For example if I have a
folder with:</p>
<p>1.las<br>
2.las<br>
...</p>
<p>Then I could spin four machines and do:</p>
<p>1] entwine build -i 1.las 2.las --subset 1 4 -o out1<br>
2] entwine build -i 1.las 2.las --subset 2 4 -o out2<br>
3] entwine build -i 1.las 2.las --subset 3 4 -o out3<br>
4] entwine build -i 1.las 2.las --subset 4 4 -o out4<br>
<br>
Then merge the results. I've noticed two things with this.
It seemed that as the number of input files increased, the
memory and time required to create each subset seemed
increased also (that's why I opted to use scan + build
--run 1). The second is that I need to wait for all point
clouds to be available (both 1.las and 2.las need to be
available before I can start processing them).</p>
<p>I wanted to rule out whether it was possible to do
something like (on two separate machines):</p>
<p>1] entwine build -i 1.las -o out1<br>
2] entwine build -i 2.las -o out2<br>
</p>
<p>And then merge the resulting EPT indexes into a "global"
one:</p>
<p>entwine merge -i out1 out2 -o merged</p>
<p>But I don't think it's possible, correct?</p>
<p>-Piero<br>
</p>
<br>
<br>
<div class="gmail-m_-7620134373879682099gmail-m_8800756632665200235moz-cite-prefix">On
6/13/19 10:43 AM, Connor Manning wrote:<br>
</div>
<blockquote type="cite" id="gmail-m_-7620134373879682099gmail_m_8800756632665200235mid_CAO_FyjLPi99vi4T02gaCUexSiWhc6UadmD0HZqeNAhadwQeCSQ_mail_gmail_com" class="gmail-m_-7620134373879682099gmail-m_8800756632665200235cite gmail-m_-7620134373879682099cite">
<div dir="ltr">The `subset` option lets each iteration of
the build run a spatially distinct region, which can be
trivially merged afterward, which sounds like what
you're after. Another option could be to simply use
multiple indexes - potree can accept multiple input EPT
sources, and a PDAL pipeline may have multiple EPT
readers.</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, Jun 13, 2019
at 6:46 AM Piero Toffanin <<a href="mailto:pt@masseranolabs.com" target="_blank">pt@masseranolabs.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote gmail-m_-7620134373879682099gmail-m_8800756632665200235cite gmail-m_-7620134373879682099cite" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" id="gmail-m_-7620134373879682099gmail_m_8800756632665200235Cite_7070813">
<div bgcolor="#FFFFFF">
<p>Hi there,</p>
<p>I have a question regarding the usage of Entwine
and was hoping somebody could help me? The use
case is merging point clouds that have been
generated on different machines. Each of these
point clouds is part to the same final dataset.
Entwine works great with the current workflow:</p>
<p>entwine scan -i a.las b.las ... -o output/</p>
<p>for i in {a, b, ... }<br>
</p>
<p> entwine build -i output/scan.json -o output/
--run 1</p>
<p>The "--run 1" is done to lower the memory usage.
On small datasets runtime is excellent, but with
more models the runtime starts to increase quite a
bit. I'm looking specifically to see if there are
ways to speed the generation of the EPT index. In
particular, since I generate the various LAS files
on different machines, I was wondering if there
was a way to let each machine contribute its part
of the index from the individual LAS files (such
index mapped to a network location) or if a
workflow is supported in which each machine can
build its own EPT index and then merge all EPT
indexes into one? I don't think this is possible,
but wanted to check.</p>
<p>Thank you for any help,</p>
<p>-Piero<br>
</p>
<p><br>
</p>
</div>
_______________________________________________<br>
pdal mailing list<br>
<a href="mailto:pdal@lists.osgeo.org" target="_blank">pdal@lists.osgeo.org</a><br>
<a href="https://lists.osgeo.org/mailman/listinfo/pdal" rel="noreferrer" target="_blank">https://lists.osgeo.org/mailman/listinfo/pdal</a></blockquote>
</div>
</blockquote>
<div class="gmail-m_-7620134373879682099gmail-m_8800756632665200235moz-signature">-- <br>
<p><font face="Helvetica, Arial, sans-serif"><b>Piero
Toffanin</b><br>
Drone Solutions Engineer<br>
</font></p>
<p><font face="Helvetica, Arial, sans-serif"> <font color="#5599ff"><a href="https://www.masseranolabs.com" target="_blank">masseranolabs.com</a></font><br>
</font> <a href="https://www.piero.dev" target="_blank">piero.dev</a><br>
</p>
<p><br>
</p>
</div>
</div>
</blockquote>
</div>
</blockquote>
<div class="gmail-m_-7620134373879682099moz-signature">-- <br>
<p><font face="Helvetica, Arial, sans-serif"><b>Piero Toffanin</b><br>
Drone Solutions Engineer<br>
</font></p>
<p><font face="Helvetica, Arial, sans-serif"> <font color="#5599ff"><a href="https://www.masseranolabs.com" target="_blank">masseranolabs.com</a></font><br>
</font> <a href="https://www.piero.dev" target="_blank">piero.dev</a><br>
</p>
<p><br>
</p>
</div>
</div>
_______________________________________________<br>
pdal mailing list<br>
<a href="mailto:pdal@lists.osgeo.org" target="_blank">pdal@lists.osgeo.org</a><br>
<a href="https://lists.osgeo.org/mailman/listinfo/pdal" rel="noreferrer" target="_blank">https://lists.osgeo.org/mailman/listinfo/pdal</a></blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div>Dr. Adam Steer</div><div><a href="http://spatialised.net" target="_blank">http://spatialised.net</a></div><div><a href="https://www.researchgate.net/profile/Adam_Steer" target="_blank">https://www.researchgate.net/profile/Adam_Steer</a><br><a href="http://au.linkedin.com/in/adamsteer" target="_blank">http://au.linkedin.com/in/adamsteer</a></div><div><a href="http://orcid.org/0000-0003-0046-7236" target="_blank">http://orcid.org/0000-0003-0046-7236</a><br>+61 427 091 712 :: @adamdsteer</div><div><br></div><div>Suits are bad for business: <a href="http://www.spatialised.net/business-penguins/" target="_blank">http://www.spatialised.net/business-penguins/</a></div></div></div></div></div></div></div></div>