[pdal] Reading and Writing outside of a stage

[Andrew Bell]

Perhaps it would help if you posted some code.

I think I understand what you're generally trying to achieve.

The chipper filter can't use streaming mode because it requires sorting of
the points in order to determine cells.  The splitter filter is a more
traditional tiling filter, and it could, theoretically, use streaming mode
(since points are binned by nothing than their location).  The issue is
that there is no "routing" in the pipeline when in streaming mode.  There
is no way to say "send this point on to stage X".  Or any generic way to
have a downstream stage X realize that the point needs to be processed in
such and such a way.  Perhaps we could think about doing something like
this, but it's not generally doable right now.

I'm not sure what your splitter is doing, but if it were to tag the points
with some sort of cell ID, you could make a single writer stage accepts
points from your splitter that essentially multiplexes the points based on
that cell ID.  Or you could have your splitter simply do the writing based
on the cell for each point.


On Tue, Jul 18, 2017 at 10:40 AM, Jason Beverage <jasonbeverage at gmail.com>
wrote:

> Hey Andrew!
>
> Thanks for getting back to me.  I think part of the issue is that I don't
> quite understand how to convert my current cursor based LASTools code to a
> streaming Stage based system.
>
> I've attached an image of what I'm trying to do.
>
> Basically I want to have a reader that can stream a bunch of points in,
> then a splitter stage that determines which of 9 potential writers to add
> the point to based on the location of the point and some other criteria,
> but I want to do it all without storing all of the points in memory.
>
> The existing ChipperFilter is similar to what I want to do, but it doesn't
> appear to support streaming.  It looks like it does the partitioning in
> memory and then returns a PointViewSet with all of the outputs.
>
> Jason
>
>
>
> On Tue, Jul 18, 2017 at 10:18 AM Andrew Bell <andrew.bell.ia at gmail.com>
> wrote:
>
>> On Mon, Jul 17, 2017 at 10:29 PM, Jason Beverage <jasonbeverage at gmail.com
>> > wrote:
>>
>>> Hi all,
>>>
>>> I'm new to PDAL and I'm currently porting an existing point cloud tookit
>>> we have at Pelican Mapping to use PDAL instead of LASTools and I had a
>>> couple of questions.
>>>
>>> Our toolkit is similar to Entwine in that it takes a large number of
>>> input points from LAS/LAZ files and chops them up into tiled LAZ files with
>>> the goal of displaying them in OpenSceneGraph and osgEarth based
>>> applications.
>>>
>>> Since we're dealing with a huge number of points, I'm making heavy use
>>> of streaming so I can avoid keeping the entire dataset in memory.
>>>
>>> For reading, I found the StreamCallbackFilter so I can intercept the
>>> points coming from a stage and process them one by one as I was doing
>>> before, so I think that will work for my case.
>>>
>>> For writing however, I'm running into a bit of snag probably due to my
>>> lack of understanding of the streaming API.
>>>
>>> We basically have an algorithm that does something like this:
>>>
>>> for each point in all the input points:
>>>     // Figure out which output file to write the point to based on it's
>>> location.
>>>     Writer* writer = getWriter(point);
>>>     writer->writePoint(writer);
>>>
>>> So we create a bunch of output writers (using LASTools) based on an
>>> octree split, and for each point we determine which writer to write to and
>>> write the incoming point to the appropriate output file.
>>>
>>> Is there a simple way to write points interactively to a Writer?  It
>>> seems like if the write functions of Writer were public that it would be
>>> fairly easy to implement something like that.
>>>
>>
>> I'm not sure I understand exactly.  You can create your own stage that
>> implements processOne() and do anything you like in it.  There is nothing
>> particularly special about a writer.  It's just a stage.  There's no reason
>> you can't chain a bunch of StreamCallbackFilters together if that's helpful
>> -- the first one could do whatever you're doing now, the second one could
>> do the writing you want to do.  Or they could be combined.
>>
>> Is part of what you are wanting is to use PDAL's LAS writer instead of
>> LAStools?
>>
>> --
>> Andrew Bell
>> andrew.bell.ia at gmail.com
>>
>


-- 
Andrew Bell
andrew.bell.ia at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pdal/attachments/20170718/3f6725dc/attachment.html>