<div dir="ltr">I've opened a ticket for this issue: <a href="https://github.com/PDAL/PDAL/issues/1621">https://github.com/PDAL/PDAL/issues/1621</a></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jul 18, 2017 at 12:59 PM, Jason Beverage <span dir="ltr"><<a href="mailto:jasonbeverage@gmail.com" target="_blank">jasonbeverage@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hey Andrew,<div><br></div><div>Thanks for the feedback. I'm not sure I have a great answer to the question either :) My hope was just to completely replace my usage of LASTools with PDAL, but I'll have to do something different to support streamed writing.</div><div><br></div><div>I also looked through the Entwine code a little bit to see if I could get some insight into how you're handing processing large datasets there, but I didn't dig deep enough yet :)</div><div><br></div><div>Thanks again for the help, I'll figure out something.</div><span class="HOEnZb"><font color="#888888"><div><br></div><div>Jason</div></font></span></div><div class="HOEnZb"><div class="h5"><br><div class="gmail_quote"><div dir="ltr">On Tue, Jul 18, 2017 at 12:54 PM Andrew Bell <<a href="mailto:andrew.bell.ia@gmail.com" target="_blank">andrew.bell.ia@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">If you're using LAStools to write, then it doesn't seem like it really makes any difference. Your code can contain/invoke any function you want. You're also welcome to modify the PDAL code to support your needs, but I'm not sure that we'd integrate it into the distribution without some discussion.<div><br></div><div>The functions are private because the calls necessary to ensure proper functioning can be complicated and disallowing direct access eliminates problems. You need to invoke a bunch of private virtuals in order to make things work, not just processOne(). It's also all that's necessary to support the standard workflow. Private virtuals provide a clear implementation interface that would be broken if changed.</div><div><br></div><div>I suppose we could have a #define for the private virtual functions for people who want to do what you're asking for. I'm not sure I love it, but perhaps I don't hate it, either. I'm thinking about it.</div><div><br></div><div>This has come up before, and I think the model that was desired was the same as yours. I'd like to find a generic way to deal with it, but I don't think that means making all functions public. Writers can already be kind of complicated, and I'm not sure how best to deal with it. It's hard to come up with a solution until you understand the problem. Here are a couple of things to think about:<br></div><div><br></div><div>- The current LAS writer can create multiple output files. Do we want to feed points to a single (complicated) writer and have it multiplex points or do we want to attach multiple writers to an input stage?</div><div><br></div><div>- If you're creating multiple downstream stages, at what point do they get created? Are they created dynamically? When?</div><div><br></div><div>- How do you decide how to route points to downstream stages? Is there code already in place for non-streaming mode that might simplify this?</div><div><br></div><div>Anyway, your question is a good one, but I'm not sure that I have an answer that I love right now. If you want to get going, just modify the PDAL code to make the functions you need public. Or better yet, make your own class a friend of LasWriter (or even Stage) and then invoke whatever private functions you please from the class you create. That's only a single line change to the existing PDAL code.</div><div><br></div><div>Hope that helps for now,</div><div><br></div></div><div class="gmail_extra"></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jul 18, 2017 at 11:46 AM, Jason Beverage <span dir="ltr"><<a href="mailto:jasonbeverage@gmail.com" target="_blank">jasonbeverage@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hey Andrew,<div><br></div><div>That's exactly what I want to do, and I think it could work, but some of the methods I want to get to are private in PDAL so I can't get to them. Basically something this is what I'm after:</div><div><br></div><div><div><span style="white-space:pre-wrap"> </span>class SplitWriter : public Writer</div><div><span style="white-space:pre-wrap"> </span>{</div><div><span style="white-space:pre-wrap"> </span>public:<span style="white-space:pre-wrap"> </span></div><div><span style="white-space:pre-wrap"> </span>SplitWriter()</div><div><span style="white-space:pre-wrap"> </span>{</div><div><span style="white-space:pre-wrap"> </span>}</div><div><br></div><div><span style="white-space:pre-wrap"> </span>private:</div><div><br></div><div><span style="white-space:pre-wrap"> </span>Writer* getWriter(PointRef& point)</div><div><span style="white-space:pre-wrap"> </span>{</div><div><span style="white-space:pre-wrap"> </span>// Return the correct writer for the point based on location.</div><div><span style="white-space:pre-wrap"> </span>}</div><div><br></div><div><span style="white-space:pre-wrap"> </span>virtual bool processOne(PointRef& point)</div><div><span style="white-space:pre-wrap"> </span>{</div><div><span style="white-space:pre-wrap"> </span>// Get the appropriate writer.</div><div><span style="white-space:pre-wrap"> </span>Writer* writer = getWriter(point);</div><div><span style="white-space:pre-wrap"> </span>writer->processOne(point);</div><div><span style="white-space:pre-wrap"> </span>return true;</div><div><span style="white-space:pre-wrap"> </span>}</div><div><span style="white-space:pre-wrap"> </span>};</div></div><div><br></div><div>My process one function is called, but I can't forward the point to an internal writer b/c processOne is private. </div><div><br></div><div>Your idea about either tagging the points or having the splitter do the writing itself is fine, but I don't see a way to use a class like the LASWriter directly since almost every method is private.</div><div><br></div><div>Just making those protected or public would actually probably solve the issue.</div><span class="m_-5333646350219677803m_8337259876339473354HOEnZb"><font color="#888888"><div><br></div><div>Jason</div><div><br></div></font></span></div><div class="m_-5333646350219677803m_8337259876339473354HOEnZb"><div class="m_-5333646350219677803m_8337259876339473354h5"><br><div class="gmail_quote"><div dir="ltr">On Tue, Jul 18, 2017 at 10:40 AM Jason Beverage <<a href="mailto:jasonbeverage@gmail.com" target="_blank">jasonbeverage@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hey Andrew!<div><br></div><div>Thanks for getting back to me. I think part of the issue is that I don't quite understand how to convert my current cursor based LASTools code to a streaming Stage based system.</div><div><br></div><div>I've attached an image of what I'm trying to do.</div><div><br></div><div>Basically I want to have a reader that can stream a bunch of points in, then a splitter stage that determines which of 9 potential writers to add the point to based on the location of the point and some other criteria, but I want to do it all without storing all of the points in memory.</div><div><br></div><div>The existing ChipperFilter is similar to what I want to do, but it doesn't appear to support streaming. It looks like it does the partitioning in memory and then returns a PointViewSet with all of the outputs.</div></div><div dir="ltr"><div><br></div><div>Jason</div></div><div dir="ltr"><div><br></div><div><br></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Jul 18, 2017 at 10:18 AM Andrew Bell <<a href="mailto:andrew.bell.ia@gmail.com" target="_blank">andrew.bell.ia@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">On Mon, Jul 17, 2017 at 10:29 PM, Jason Beverage <span dir="ltr"><<a href="mailto:jasonbeverage@gmail.com" target="_blank">jasonbeverage@gmail.com</a>></span> wrote:</div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi all,<div><br></div><div>I'm new to PDAL and I'm currently porting an existing point cloud tookit we have at Pelican Mapping to use PDAL instead of LASTools and I had a couple of questions.</div><div><br></div><div>Our toolkit is similar to Entwine in that it takes a large number of input points from LAS/LAZ files and chops them up into tiled LAZ files with the goal of displaying them in OpenSceneGraph and osgEarth based applications.</div><div><br></div><div>Since we're dealing with a huge number of points, I'm making heavy use of streaming so I can avoid keeping the entire dataset in memory.</div><div><br></div><div>For reading, I found the StreamCallbackFilter so I can intercept the points coming from a stage and process them one by one as I was doing before, so I think that will work for my case.</div><div><br></div><div>For writing however, I'm running into a bit of snag probably due to my lack of understanding of the streaming API.</div><div><br></div><div>We basically have an algorithm that does something like this:</div><div><br></div><div>for each point in all the input points:<br></div><div> // Figure out which output file to write the point to based on it's location.</div><div> Writer* writer = getWriter(point);</div><div> writer->writePoint(writer);</div><div><br></div><div>So we create a bunch of output writers (using LASTools) based on an octree split, and for each point we determine which writer to write to and write the incoming point to the appropriate output file.</div><div><br></div><div>Is there a simple way to write points interactively to a Writer? It seems like if the write functions of Writer were public that it would be fairly easy to implement something like that.</div></div></blockquote><div><br></div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>I'm not sure I understand exactly. You can create your own stage that implements processOne() and do anything you like in it. There is nothing particularly special about a writer. It's just a stage. There's no reason you can't chain a bunch of StreamCallbackFilters together if that's helpful -- the first one could do whatever you're doing now, the second one could do the writing you want to do. Or they could be combined.</div><div><br></div><div>Is part of what you are wanting is to use PDAL's LAS writer instead of LAStools?</div><div><br></div></div></div></div><div dir="ltr"><div class="gmail_extra">-- <br><div class="m_-5333646350219677803m_8337259876339473354m_-1631922014764760578m_-1482031444919401928m_1821188321884791268m_-7008113717705040360gmail_signature" data-smartmail="gmail_signature">Andrew Bell<br><a href="mailto:andrew.bell.ia@gmail.com" target="_blank">andrew.bell.ia@gmail.com</a></div>
</div></div></blockquote></div></div></blockquote></div>
</div></div></blockquote></div><br><br clear="all"><div><br></div></div><div class="gmail_extra">-- <br><div class="m_-5333646350219677803m_8337259876339473354gmail_signature" data-smartmail="gmail_signature">Andrew Bell<br><a href="mailto:andrew.bell.ia@gmail.com" target="_blank">andrew.bell.ia@gmail.com</a></div>
</div></blockquote></div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">Andrew Bell<br><a href="mailto:andrew.bell.ia@gmail.com" target="_blank">andrew.bell.ia@gmail.com</a></div>
</div>