[pdal] Threading and PDAL

[Connor Manning]

You'll need a lock around anywhere a `pdal::SpatialReference` is created.
For Entwine, prepare and preview are the main places that happens, but
you'll also need to lock if you're explicitly going to create one.

On Thu, Jul 20, 2017 at 2:21 PM, Jason Beverage <jasonbeverage at gmail.com>
wrote:

> Thanks Connor, I just took a look at the executor.cpp.
>
> I see locking around stage creation which I already have.  I also see
> explicit locking around the prepare call and the preview, which I'm not
> doing so I will try adding that.
>
> Is there anything else that you can think of I might be missing?
>
> Jason
>
> On Thu, Jul 20, 2017 at 3:08 PM Connor Manning <connor at hobu.co> wrote:
>
>> Entwine is multi-threaded.  Most of the PDAL-locking in Entwine happens
>> in `entwine/util/executor.cpp` and `hpp`.
>>
>> On Thu, Jul 20, 2017 at 2:02 PM, Jason Beverage <jasonbeverage at gmail.com>
>> wrote:
>>
>>> Hi Andrew,
>>>
>>> Basically I have a bunch of jobs that read from input files using PDAL,
>>> do something on them, and then write some resulting points out to an LAZ
>>> file.  I've got all Stage creation wrapped in a mutex and I was hoping that
>>> the stage pipelines for each job could run independently without a lock.
>>> I'm getting random access violations in PDAL.  I'm going to try to build
>>> pdal in debug and see if I can figure out what is going on.  If I globally
>>> mutex any access at all to PDAL things run just fine (although not in
>>> parallel :) ).
>>>
>>> Does Entwine use threading or is it a strictly multiprocess setup?
>>>
>>> Jason
>>>
>>>
>>> On Thu, Jul 20, 2017 at 10:17 AM Andrew Bell <andrew.bell.ia at gmail.com>
>>> wrote:
>>>
>>>> The stage creation code through the PluginManager and StageManager is
>>>> believed to be threadsafe.  Nothing else is guaranteed.  We rely on many
>>>> libraries which may or may not be threadsafe.  If you need thread safety,
>>>> you need to implement your own locking.
>>>>
>>>> If you're getting thread-related crashes in PluginManager or
>>>> StageManager, please open a ticket.
>>>>
>>>> That said, there is probably a lot you can get away with -- we don't
>>>> rely much on constructs that are inherently non-threadsafe.  I'm willing to
>>>> look at issues up to a point, but we're not going to work hard to make
>>>> things threadsafe.  I'd need to know more about specific problems to be
>>>> helpful.
>>>>
>>>>
>>>> On Thu, Jul 20, 2017 at 10:06 AM, Jason Beverage <
>>>> jasonbeverage at gmail.com> wrote:
>>>>
>>>>> I'm running into some threading issues with PDAL that I could use some
>>>>> help with.  I'm running into some random crashing using PDAL in a
>>>>> multithreaded manner.  I understand that PDAL isn't entirely threadsafe,
>>>>> but do you have some info on what is and isn't threadsafe?
>>>>>
>>>>> Basically what I'm trying to do is multiple threads are creating
>>>>> Stages using a shared StageFactory and each thread is running it's own
>>>>> independent pipeline.  The only thing that is shared is the StageFactory
>>>>> which is static for the application.  I've got access to the StageFactory
>>>>> protected by my own mutex since I was getting crashes accessing it from
>>>>> multiple threads.  I tried creating a StageFactory per thread but that
>>>>> caused other issues like stages randomly not being able to be able to be
>>>>> created and it not loading my plugins all the time.
>>>>>
>>>>> Right now I'm using the OSGEO4W 64 bit package to get PDAL and all of
>>>>> it's dependencies and building my application against that.  It looks like
>>>>> the pdal version is 1.5.0 which is the latest release.
>>>>>
>>>>> Thanks!
>>>>>
>>>>> Jason
>>>>>
>>>>> _______________________________________________
>>>>> pdal mailing list
>>>>> pdal at lists.osgeo.org
>>>>> https://lists.osgeo.org/mailman/listinfo/pdal
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Andrew Bell
>>>> andrew.bell.ia at gmail.com
>>>>
>>>
>>> _______________________________________________
>>> pdal mailing list
>>> pdal at lists.osgeo.org
>>> https://lists.osgeo.org/mailman/listinfo/pdal
>>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pdal/attachments/20170720/155dbaf8/attachment.html>