[pdal] Threading and PDAL

[Jason Beverage]

Thanks Connor, I just took a look at the executor.cpp.

I see locking around stage creation which I already have.  I also see
explicit locking around the prepare call and the preview, which I'm not
doing so I will try adding that.

Is there anything else that you can think of I might be missing?

Jason

On Thu, Jul 20, 2017 at 3:08 PM Connor Manning <connor at hobu.co> wrote:

> Entwine is multi-threaded.  Most of the PDAL-locking in Entwine happens in
> `entwine/util/executor.cpp` and `hpp`.
>
> On Thu, Jul 20, 2017 at 2:02 PM, Jason Beverage <jasonbeverage at gmail.com>
> wrote:
>
>> Hi Andrew,
>>
>> Basically I have a bunch of jobs that read from input files using PDAL,
>> do something on them, and then write some resulting points out to an LAZ
>> file.  I've got all Stage creation wrapped in a mutex and I was hoping that
>> the stage pipelines for each job could run independently without a lock.
>> I'm getting random access violations in PDAL.  I'm going to try to build
>> pdal in debug and see if I can figure out what is going on.  If I globally
>> mutex any access at all to PDAL things run just fine (although not in
>> parallel :) ).
>>
>> Does Entwine use threading or is it a strictly multiprocess setup?
>>
>> Jason
>>
>>
>> On Thu, Jul 20, 2017 at 10:17 AM Andrew Bell <andrew.bell.ia at gmail.com>
>> wrote:
>>
>>> The stage creation code through the PluginManager and StageManager is
>>> believed to be threadsafe.  Nothing else is guaranteed.  We rely on many
>>> libraries which may or may not be threadsafe.  If you need thread safety,
>>> you need to implement your own locking.
>>>
>>> If you're getting thread-related crashes in PluginManager or
>>> StageManager, please open a ticket.
>>>
>>> That said, there is probably a lot you can get away with -- we don't
>>> rely much on constructs that are inherently non-threadsafe.  I'm willing to
>>> look at issues up to a point, but we're not going to work hard to make
>>> things threadsafe.  I'd need to know more about specific problems to be
>>> helpful.
>>>
>>>
>>> On Thu, Jul 20, 2017 at 10:06 AM, Jason Beverage <
>>> jasonbeverage at gmail.com> wrote:
>>>
>>>> I'm running into some threading issues with PDAL that I could use some
>>>> help with.  I'm running into some random crashing using PDAL in a
>>>> multithreaded manner.  I understand that PDAL isn't entirely threadsafe,
>>>> but do you have some info on what is and isn't threadsafe?
>>>>
>>>> Basically what I'm trying to do is multiple threads are creating Stages
>>>> using a shared StageFactory and each thread is running it's own independent
>>>> pipeline.  The only thing that is shared is the StageFactory which is
>>>> static for the application.  I've got access to the StageFactory protected
>>>> by my own mutex since I was getting crashes accessing it from multiple
>>>> threads.  I tried creating a StageFactory per thread but that caused other
>>>> issues like stages randomly not being able to be able to be created and it
>>>> not loading my plugins all the time.
>>>>
>>>> Right now I'm using the OSGEO4W 64 bit package to get PDAL and all of
>>>> it's dependencies and building my application against that.  It looks like
>>>> the pdal version is 1.5.0 which is the latest release.
>>>>
>>>> Thanks!
>>>>
>>>> Jason
>>>>
>>>> _______________________________________________
>>>> pdal mailing list
>>>> pdal at lists.osgeo.org
>>>> https://lists.osgeo.org/mailman/listinfo/pdal
>>>>
>>>
>>>
>>>
>>> --
>>> Andrew Bell
>>> andrew.bell.ia at gmail.com
>>>
>>
>> _______________________________________________
>> pdal mailing list
>> pdal at lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/pdal
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pdal/attachments/20170720/c569ecb7/attachment-0001.html>