[pdal] Reading and Writing outside of a stage

[Jason Beverage]

Hey Andrew,

Thanks for the feedback.  I'm not sure I have a great answer to the
question either :)  My hope was just to completely replace my usage of
LASTools with PDAL, but I'll have to do something different to support
streamed writing.

I also looked through the Entwine code a little bit to see if I could get
some insight into how you're handing processing large datasets there, but I
didn't dig deep enough yet :)

Thanks again for the help, I'll figure out something.

Jason

On Tue, Jul 18, 2017 at 12:54 PM Andrew Bell <andrew.bell.ia at gmail.com>
wrote:

> If you're using LAStools to write, then it doesn't seem like it really
> makes any difference.  Your code can contain/invoke any function you want.
> You're also welcome to modify the PDAL code to support your needs, but I'm
> not sure that we'd integrate it into the distribution without some
> discussion.
>
> The functions are private because the calls necessary to ensure proper
> functioning can be complicated and disallowing direct access eliminates
> problems.  You need to invoke a bunch of private virtuals in order to make
> things work, not just processOne().  It's also all that's necessary to
> support the standard workflow.  Private virtuals provide a clear
> implementation interface that would be broken if changed.
>
> I suppose we could have a #define for the private virtual functions for
> people who want to do what you're asking for.  I'm not sure I love it, but
> perhaps I don't hate it, either.  I'm thinking about it.
>
> This has come up before, and I think the model that was desired was the
> same as yours.  I'd like to find a generic way to deal with it, but I don't
> think that means making all functions public.  Writers can already be kind
> of complicated, and I'm not sure how best to deal with it.  It's hard to
> come up with a solution until you understand the problem.  Here are a
> couple of things to think about:
>
> - The current LAS writer can create multiple output files.  Do we want to
> feed points to a single (complicated) writer and have it multiplex points
> or do we want to attach multiple writers to an input stage?
>
> - If you're creating multiple downstream stages, at what point do they get
> created?  Are they created dynamically?  When?
>
> - How do you decide how to route points to downstream stages?  Is there
> code already in place for non-streaming mode that might simplify this?
>
> Anyway, your question is a good one, but I'm not sure that I have an
> answer that I love right now.  If you want to get going, just modify the
> PDAL code to make the functions you need public.  Or better yet, make your
> own class a friend of LasWriter (or even Stage) and then invoke whatever
> private functions you please from the class you create.  That's only a
> single line change to the existing PDAL code.
>
> Hope that helps for now,
>
>
> On Tue, Jul 18, 2017 at 11:46 AM, Jason Beverage <jasonbeverage at gmail.com>
> wrote:
>
>> Hey Andrew,
>>
>> That's exactly what I want to do, and I think it could work, but some of
>> the methods I want to get to are private in PDAL so I can't get to them.
>> Basically something this is what I'm after:
>>
>> class SplitWriter : public Writer
>> {
>> public:
>> SplitWriter()
>> {
>> }
>>
>> private:
>>
>> Writer* getWriter(PointRef& point)
>> {
>> // Return the correct writer for the point based on location.
>> }
>>
>> virtual bool processOne(PointRef& point)
>> {
>> // Get the appropriate writer.
>> Writer* writer = getWriter(point);
>> writer->processOne(point);
>> return true;
>> }
>> };
>>
>> My process one function is called, but I can't forward the point to an
>> internal writer b/c processOne is private.
>>
>> Your idea about either tagging the points or having the splitter do the
>> writing itself is fine, but I don't see a way to use a class like the
>> LASWriter directly since almost every method is private.
>>
>> Just making those protected or public would actually probably solve the
>> issue.
>>
>> Jason
>>
>>
>> On Tue, Jul 18, 2017 at 10:40 AM Jason Beverage <jasonbeverage at gmail.com>
>> wrote:
>>
>>> Hey Andrew!
>>>
>>> Thanks for getting back to me.  I think part of the issue is that I
>>> don't quite understand how to convert my current cursor based LASTools code
>>> to a streaming Stage based system.
>>>
>>> I've attached an image of what I'm trying to do.
>>>
>>> Basically I want to have a reader that can stream a bunch of points in,
>>> then a splitter stage that determines which of 9 potential writers to add
>>> the point to based on the location of the point and some other criteria,
>>> but I want to do it all without storing all of the points in memory.
>>>
>>> The existing ChipperFilter is similar to what I want to do, but it
>>> doesn't appear to support streaming.  It looks like it does the
>>> partitioning in memory and then returns a PointViewSet with all of the
>>> outputs.
>>>
>>> Jason
>>>
>>>
>>>
>>> On Tue, Jul 18, 2017 at 10:18 AM Andrew Bell <andrew.bell.ia at gmail.com>
>>> wrote:
>>>
>>>> On Mon, Jul 17, 2017 at 10:29 PM, Jason Beverage <
>>>> jasonbeverage at gmail.com> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I'm new to PDAL and I'm currently porting an existing point cloud
>>>>> tookit we have at Pelican Mapping to use PDAL instead of LASTools and I had
>>>>> a couple of questions.
>>>>>
>>>>> Our toolkit is similar to Entwine in that it takes a large number of
>>>>> input points from LAS/LAZ files and chops them up into tiled LAZ files with
>>>>> the goal of displaying them in OpenSceneGraph and osgEarth based
>>>>> applications.
>>>>>
>>>>> Since we're dealing with a huge number of points, I'm making heavy use
>>>>> of streaming so I can avoid keeping the entire dataset in memory.
>>>>>
>>>>> For reading, I found the StreamCallbackFilter so I can intercept the
>>>>> points coming from a stage and process them one by one as I was doing
>>>>> before, so I think that will work for my case.
>>>>>
>>>>> For writing however, I'm running into a bit of snag probably due to my
>>>>> lack of understanding of the streaming API.
>>>>>
>>>>> We basically have an algorithm that does something like this:
>>>>>
>>>>> for each point in all the input points:
>>>>>     // Figure out which output file to write the point to based on
>>>>> it's location.
>>>>>     Writer* writer = getWriter(point);
>>>>>     writer->writePoint(writer);
>>>>>
>>>>> So we create a bunch of output writers (using LASTools) based on an
>>>>> octree split, and for each point we determine which writer to write to and
>>>>> write the incoming point to the appropriate output file.
>>>>>
>>>>> Is there a simple way to write points interactively to a Writer?  It
>>>>> seems like if the write functions of Writer were public that it would be
>>>>> fairly easy to implement something like that.
>>>>>
>>>>
>>>> I'm not sure I understand exactly.  You can create your own stage that
>>>> implements processOne() and do anything you like in it.  There is nothing
>>>> particularly special about a writer.  It's just a stage.  There's no reason
>>>> you can't chain a bunch of StreamCallbackFilters together if that's helpful
>>>> -- the first one could do whatever you're doing now, the second one could
>>>> do the writing you want to do.  Or they could be combined.
>>>>
>>>> Is part of what you are wanting is to use PDAL's LAS writer instead of
>>>> LAStools?
>>>>
>>>> --
>>>> Andrew Bell
>>>> andrew.bell.ia at gmail.com
>>>>
>>>
>
>
> --
> Andrew Bell
> andrew.bell.ia at gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pdal/attachments/20170718/8ea4fee8/attachment.html>