[pdal] Reading and Writing outside of a stage

Tue Jul 18 10:40:38 PDT 2017

I've opened a ticket for this issue:
https://github.com/PDAL/PDAL/issues/1621

On Tue, Jul 18, 2017 at 12:59 PM, Jason Beverage <jasonbeverage at gmail.com>
wrote:

> Hey Andrew,
>
> Thanks for the feedback.  I'm not sure I have a great answer to the
> question either :)  My hope was just to completely replace my usage of
> LASTools with PDAL, but I'll have to do something different to support
> streamed writing.
>
> I also looked through the Entwine code a little bit to see if I could get
> some insight into how you're handing processing large datasets there, but I
> didn't dig deep enough yet :)
>
> Thanks again for the help, I'll figure out something.
>
> Jason
>
> On Tue, Jul 18, 2017 at 12:54 PM Andrew Bell <andrew.bell.ia at gmail.com>
> wrote:
>
>> If you're using LAStools to write, then it doesn't seem like it really
>> makes any difference.  Your code can contain/invoke any function you want.
>> You're also welcome to modify the PDAL code to support your needs, but I'm
>> not sure that we'd integrate it into the distribution without some
>> discussion.
>>
>> The functions are private because the calls necessary to ensure proper
>> functioning can be complicated and disallowing direct access eliminates
>> problems.  You need to invoke a bunch of private virtuals in order to make
>> things work, not just processOne().  It's also all that's necessary to
>> support the standard workflow.  Private virtuals provide a clear
>> implementation interface that would be broken if changed.
>>
>> I suppose we could have a #define for the private virtual functions for
>> people who want to do what you're asking for.  I'm not sure I love it, but
>> perhaps I don't hate it, either.  I'm thinking about it.
>>
>> This has come up before, and I think the model that was desired was the
>> same as yours.  I'd like to find a generic way to deal with it, but I don't
>> think that means making all functions public.  Writers can already be kind
>> of complicated, and I'm not sure how best to deal with it.  It's hard to
>> come up with a solution until you understand the problem.  Here are a
>> couple of things to think about:
>>
>> - The current LAS writer can create multiple output files.  Do we want to
>> feed points to a single (complicated) writer and have it multiplex points
>> or do we want to attach multiple writers to an input stage?
>>
>> - If you're creating multiple downstream stages, at what point do they
>> get created?  Are they created dynamically?  When?
>>
>> - How do you decide how to route points to downstream stages?  Is there
>> code already in place for non-streaming mode that might simplify this?
>>
>> Anyway, your question is a good one, but I'm not sure that I have an
>> answer that I love right now.  If you want to get going, just modify the
>> PDAL code to make the functions you need public.  Or better yet, make your
>> own class a friend of LasWriter (or even Stage) and then invoke whatever
>> private functions you please from the class you create.  That's only a
>> single line change to the existing PDAL code.
>>
>> Hope that helps for now,
>>
>>
>> On Tue, Jul 18, 2017 at 11:46 AM, Jason Beverage <jasonbeverage at gmail.com
>> > wrote:
>>
>>> Hey Andrew,
>>>
>>> That's exactly what I want to do, and I think it could work, but some of
>>> the methods I want to get to are private in PDAL so I can't get to them.
>>> Basically something this is what I'm after:
>>>
>>> class SplitWriter : public Writer
>>> {
>>> public:
>>> SplitWriter()
>>> {
>>> }
>>>
>>> private:
>>>
>>> Writer* getWriter(PointRef& point)
>>> {
>>> // Return the correct writer for the point based on location.
>>> }
>>>
>>> virtual bool processOne(PointRef& point)
>>> {
>>> // Get the appropriate writer.
>>> Writer* writer = getWriter(point);
>>> writer->processOne(point);
>>> return true;
>>> }
>>> };
>>>
>>> My process one function is called, but I can't forward the point to an
>>> internal writer b/c processOne is private.
>>>
>>> Your idea about either tagging the points or having the splitter do the
>>> writing itself is fine, but I don't see a way to use a class like the
>>> LASWriter directly since almost every method is private.
>>>
>>> Just making those protected or public would actually probably solve the
>>> issue.
>>>
>>> Jason
>>>
>>>
>>> On Tue, Jul 18, 2017 at 10:40 AM Jason Beverage <jasonbeverage at gmail.com>
>>> wrote:
>>>
>>>> Hey Andrew!
>>>>
>>>> Thanks for getting back to me.  I think part of the issue is that I
>>>> don't quite understand how to convert my current cursor based LASTools code
>>>> to a streaming Stage based system.
>>>>
>>>> I've attached an image of what I'm trying to do.
>>>>
>>>> Basically I want to have a reader that can stream a bunch of points in,
>>>> then a splitter stage that determines which of 9 potential writers to add
>>>> the point to based on the location of the point and some other criteria,
>>>> but I want to do it all without storing all of the points in memory.
>>>>
>>>> The existing ChipperFilter is similar to what I want to do, but it
>>>> doesn't appear to support streaming.  It looks like it does the
>>>> partitioning in memory and then returns a PointViewSet with all of the
>>>> outputs.
>>>>
>>>> Jason
>>>>
>>>>
>>>>
>>>> On Tue, Jul 18, 2017 at 10:18 AM Andrew Bell <andrew.bell.ia at gmail.com>
>>>> wrote:
>>>>
>>>>> On Mon, Jul 17, 2017 at 10:29 PM, Jason Beverage <
>>>>> jasonbeverage at gmail.com> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I'm new to PDAL and I'm currently porting an existing point cloud
>>>>>> tookit we have at Pelican Mapping to use PDAL instead of LASTools and I had
>>>>>> a couple of questions.
>>>>>>
>>>>>> Our toolkit is similar to Entwine in that it takes a large number of
>>>>>> input points from LAS/LAZ files and chops them up into tiled LAZ files with
>>>>>> the goal of displaying them in OpenSceneGraph and osgEarth based
>>>>>> applications.
>>>>>>
>>>>>> Since we're dealing with a huge number of points, I'm making heavy
>>>>>> use of streaming so I can avoid keeping the entire dataset in memory.
>>>>>>
>>>>>> For reading, I found the StreamCallbackFilter so I can intercept the
>>>>>> points coming from a stage and process them one by one as I was doing
>>>>>> before, so I think that will work for my case.
>>>>>>
>>>>>> For writing however, I'm running into a bit of snag probably due to
>>>>>> my lack of understanding of the streaming API.
>>>>>>
>>>>>> We basically have an algorithm that does something like this:
>>>>>>
>>>>>> for each point in all the input points:
>>>>>>     // Figure out which output file to write the point to based on
>>>>>> it's location.
>>>>>>     Writer* writer = getWriter(point);
>>>>>>     writer->writePoint(writer);
>>>>>>
>>>>>> So we create a bunch of output writers (using LASTools) based on an
>>>>>> octree split, and for each point we determine which writer to write to and
>>>>>> write the incoming point to the appropriate output file.
>>>>>>
>>>>>> Is there a simple way to write points interactively to a Writer?  It
>>>>>> seems like if the write functions of Writer were public that it would be
>>>>>> fairly easy to implement something like that.
>>>>>>
>>>>>
>>>>> I'm not sure I understand exactly.  You can create your own stage that
>>>>> implements processOne() and do anything you like in it.  There is nothing
>>>>> particularly special about a writer.  It's just a stage.  There's no reason
>>>>> you can't chain a bunch of StreamCallbackFilters together if that's helpful
>>>>> -- the first one could do whatever you're doing now, the second one could
>>>>> do the writing you want to do.  Or they could be combined.
>>>>>
>>>>> Is part of what you are wanting is to use PDAL's LAS writer instead of
>>>>> LAStools?
>>>>>
>>>>> --
>>>>> Andrew Bell
>>>>> andrew.bell.ia at gmail.com
>>>>>
>>>>
>>
>>
>> --
>> Andrew Bell
>> andrew.bell.ia at gmail.com
>>
>

-- 
Andrew Bell
andrew.bell.ia at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pdal/attachments/20170718/eb6a416d/attachment-0001.html>