[pdal] Reading and Writing outside of a stage

[Andrew Bell]

If you're using LAStools to write, then it doesn't seem like it really
makes any difference.  Your code can contain/invoke any function you want.
You're also welcome to modify the PDAL code to support your needs, but I'm
not sure that we'd integrate it into the distribution without some
discussion.

The functions are private because the calls necessary to ensure proper
functioning can be complicated and disallowing direct access eliminates
problems.  You need to invoke a bunch of private virtuals in order to make
things work, not just processOne().  It's also all that's necessary to
support the standard workflow.  Private virtuals provide a clear
implementation interface that would be broken if changed.

I suppose we could have a #define for the private virtual functions for
people who want to do what you're asking for.  I'm not sure I love it, but
perhaps I don't hate it, either.  I'm thinking about it.

This has come up before, and I think the model that was desired was the
same as yours.  I'd like to find a generic way to deal with it, but I don't
think that means making all functions public.  Writers can already be kind
of complicated, and I'm not sure how best to deal with it.  It's hard to
come up with a solution until you understand the problem.  Here are a
couple of things to think about:

- The current LAS writer can create multiple output files.  Do we want to
feed points to a single (complicated) writer and have it multiplex points
or do we want to attach multiple writers to an input stage?

- If you're creating multiple downstream stages, at what point do they get
created?  Are they created dynamically?  When?

- How do you decide how to route points to downstream stages?  Is there
code already in place for non-streaming mode that might simplify this?

Anyway, your question is a good one, but I'm not sure that I have an answer
that I love right now.  If you want to get going, just modify the PDAL code
to make the functions you need public.  Or better yet, make your own class
a friend of LasWriter (or even Stage) and then invoke whatever private
functions you please from the class you create.  That's only a single line
change to the existing PDAL code.

Hope that helps for now,


On Tue, Jul 18, 2017 at 11:46 AM, Jason Beverage <jasonbeverage at gmail.com>
wrote:

> Hey Andrew,
>
> That's exactly what I want to do, and I think it could work, but some of
> the methods I want to get to are private in PDAL so I can't get to them.
> Basically something this is what I'm after:
>
> class SplitWriter : public Writer
> {
> public:
> SplitWriter()
> {
> }
>
> private:
>
> Writer* getWriter(PointRef& point)
> {
> // Return the correct writer for the point based on location.
> }
>
> virtual bool processOne(PointRef& point)
> {
> // Get the appropriate writer.
> Writer* writer = getWriter(point);
> writer->processOne(point);
> return true;
> }
> };
>
> My process one function is called, but I can't forward the point to an
> internal writer b/c processOne is private.
>
> Your idea about either tagging the points or having the splitter do the
> writing itself is fine, but I don't see a way to use a class like the
> LASWriter directly since almost every method is private.
>
> Just making those protected or public would actually probably solve the
> issue.
>
> Jason
>
>
> On Tue, Jul 18, 2017 at 10:40 AM Jason Beverage <jasonbeverage at gmail.com>
> wrote:
>
>> Hey Andrew!
>>
>> Thanks for getting back to me.  I think part of the issue is that I don't
>> quite understand how to convert my current cursor based LASTools code to a
>> streaming Stage based system.
>>
>> I've attached an image of what I'm trying to do.
>>
>> Basically I want to have a reader that can stream a bunch of points in,
>> then a splitter stage that determines which of 9 potential writers to add
>> the point to based on the location of the point and some other criteria,
>> but I want to do it all without storing all of the points in memory.
>>
>> The existing ChipperFilter is similar to what I want to do, but it
>> doesn't appear to support streaming.  It looks like it does the
>> partitioning in memory and then returns a PointViewSet with all of the
>> outputs.
>>
>> Jason
>>
>>
>>
>> On Tue, Jul 18, 2017 at 10:18 AM Andrew Bell <andrew.bell.ia at gmail.com>
>> wrote:
>>
>>> On Mon, Jul 17, 2017 at 10:29 PM, Jason Beverage <
>>> jasonbeverage at gmail.com> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I'm new to PDAL and I'm currently porting an existing point cloud
>>>> tookit we have at Pelican Mapping to use PDAL instead of LASTools and I had
>>>> a couple of questions.
>>>>
>>>> Our toolkit is similar to Entwine in that it takes a large number of
>>>> input points from LAS/LAZ files and chops them up into tiled LAZ files with
>>>> the goal of displaying them in OpenSceneGraph and osgEarth based
>>>> applications.
>>>>
>>>> Since we're dealing with a huge number of points, I'm making heavy use
>>>> of streaming so I can avoid keeping the entire dataset in memory.
>>>>
>>>> For reading, I found the StreamCallbackFilter so I can intercept the
>>>> points coming from a stage and process them one by one as I was doing
>>>> before, so I think that will work for my case.
>>>>
>>>> For writing however, I'm running into a bit of snag probably due to my
>>>> lack of understanding of the streaming API.
>>>>
>>>> We basically have an algorithm that does something like this:
>>>>
>>>> for each point in all the input points:
>>>>     // Figure out which output file to write the point to based on it's
>>>> location.
>>>>     Writer* writer = getWriter(point);
>>>>     writer->writePoint(writer);
>>>>
>>>> So we create a bunch of output writers (using LASTools) based on an
>>>> octree split, and for each point we determine which writer to write to and
>>>> write the incoming point to the appropriate output file.
>>>>
>>>> Is there a simple way to write points interactively to a Writer?  It
>>>> seems like if the write functions of Writer were public that it would be
>>>> fairly easy to implement something like that.
>>>>
>>>
>>> I'm not sure I understand exactly.  You can create your own stage that
>>> implements processOne() and do anything you like in it.  There is nothing
>>> particularly special about a writer.  It's just a stage.  There's no reason
>>> you can't chain a bunch of StreamCallbackFilters together if that's helpful
>>> -- the first one could do whatever you're doing now, the second one could
>>> do the writing you want to do.  Or they could be combined.
>>>
>>> Is part of what you are wanting is to use PDAL's LAS writer instead of
>>> LAStools?
>>>
>>> --
>>> Andrew Bell
>>> andrew.bell.ia at gmail.com
>>>
>>


-- 
Andrew Bell
andrew.bell.ia at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pdal/attachments/20170718/30a687fc/attachment-0001.html>