[pdal] Threading and PDAL

[Jason Beverage]

Thanks Andrew!   I'll check it out next week and let you know how it goes.

On Fri, Aug 4, 2017, 5:13 PM Andrew Bell <andrew.bell.ia at gmail.com> wrote:

> Hi Jason,
>
> I pushed a change today that should eliminate the corruption that you were
> seeing when using preview/prepare with stages in multiple threads.  You
> should be able to remove the locks that you added in your application.
> That said, we've never advertised PDAL as being threadsafe.  But the
> problem exhibited by your test program (thank you) has been addressed.
>
> On Thu, Jul 20, 2017 at 4:03 PM, Jason Beverage <jasonbeverage at gmail.com>
> wrote:
>
>> You're awesome!  Locking around prepare and preview appear to work.  I'm
>> going to see if I can take some of the other locks out as well.
>>
>> I'm assuming that the SpatialReference locks are due to GDAL's lack of
>> thread safety.  Would it be possible to just have an internal lock in PDAL
>> around the SpatialReference class?
>>
>> We have to do the same thing in our osgEarth project with respect to GDAL.
>>
>> Jason
>>
>>
>>
>> On Thu, Jul 20, 2017 at 3:36 PM Connor Manning <connor at hobu.co> wrote:
>>
>>> You'll need a lock around anywhere a `pdal::SpatialReference` is
>>> created.  For Entwine, prepare and preview are the main places that
>>> happens, but you'll also need to lock if you're explicitly going to create
>>> one.
>>>
>>> On Thu, Jul 20, 2017 at 2:21 PM, Jason Beverage <jasonbeverage at gmail.com
>>> > wrote:
>>>
>>>> Thanks Connor, I just took a look at the executor.cpp.
>>>>
>>>> I see locking around stage creation which I already have.  I also see
>>>> explicit locking around the prepare call and the preview, which I'm not
>>>> doing so I will try adding that.
>>>>
>>>> Is there anything else that you can think of I might be missing?
>>>>
>>>> Jason
>>>>
>>>> On Thu, Jul 20, 2017 at 3:08 PM Connor Manning <connor at hobu.co> wrote:
>>>>
>>>>> Entwine is multi-threaded.  Most of the PDAL-locking in Entwine
>>>>> happens in `entwine/util/executor.cpp` and `hpp`.
>>>>>
>>>>> On Thu, Jul 20, 2017 at 2:02 PM, Jason Beverage <
>>>>> jasonbeverage at gmail.com> wrote:
>>>>>
>>>>>> Hi Andrew,
>>>>>>
>>>>>> Basically I have a bunch of jobs that read from input files using
>>>>>> PDAL, do something on them, and then write some resulting points out to an
>>>>>> LAZ file.  I've got all Stage creation wrapped in a mutex and I was hoping
>>>>>> that the stage pipelines for each job could run independently without a
>>>>>> lock.  I'm getting random access violations in PDAL.  I'm going to try to
>>>>>> build pdal in debug and see if I can figure out what is going on.  If I
>>>>>> globally mutex any access at all to PDAL things run just fine (although not
>>>>>> in parallel :) ).
>>>>>>
>>>>>> Does Entwine use threading or is it a strictly multiprocess setup?
>>>>>>
>>>>>> Jason
>>>>>>
>>>>>>
>>>>>> On Thu, Jul 20, 2017 at 10:17 AM Andrew Bell <
>>>>>> andrew.bell.ia at gmail.com> wrote:
>>>>>>
>>>>>>> The stage creation code through the PluginManager and StageManager
>>>>>>> is believed to be threadsafe.  Nothing else is guaranteed.  We rely on many
>>>>>>> libraries which may or may not be threadsafe.  If you need thread safety,
>>>>>>> you need to implement your own locking.
>>>>>>>
>>>>>>> If you're getting thread-related crashes in PluginManager or
>>>>>>> StageManager, please open a ticket.
>>>>>>>
>>>>>>> That said, there is probably a lot you can get away with -- we don't
>>>>>>> rely much on constructs that are inherently non-threadsafe.  I'm willing to
>>>>>>> look at issues up to a point, but we're not going to work hard to make
>>>>>>> things threadsafe.  I'd need to know more about specific problems to be
>>>>>>> helpful.
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jul 20, 2017 at 10:06 AM, Jason Beverage <
>>>>>>> jasonbeverage at gmail.com> wrote:
>>>>>>>
>>>>>>>> I'm running into some threading issues with PDAL that I could use
>>>>>>>> some help with.  I'm running into some random crashing using PDAL in a
>>>>>>>> multithreaded manner.  I understand that PDAL isn't entirely threadsafe,
>>>>>>>> but do you have some info on what is and isn't threadsafe?
>>>>>>>>
>>>>>>>> Basically what I'm trying to do is multiple threads are creating
>>>>>>>> Stages using a shared StageFactory and each thread is running it's own
>>>>>>>> independent pipeline.  The only thing that is shared is the StageFactory
>>>>>>>> which is static for the application.  I've got access to the StageFactory
>>>>>>>> protected by my own mutex since I was getting crashes accessing it from
>>>>>>>> multiple threads.  I tried creating a StageFactory per thread but that
>>>>>>>> caused other issues like stages randomly not being able to be able to be
>>>>>>>> created and it not loading my plugins all the time.
>>>>>>>>
>>>>>>>> Right now I'm using the OSGEO4W 64 bit package to get PDAL and all
>>>>>>>> of it's dependencies and building my application against that.  It looks
>>>>>>>> like the pdal version is 1.5.0 which is the latest release.
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>> Jason
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> pdal mailing list
>>>>>>>> pdal at lists.osgeo.org
>>>>>>>> https://lists.osgeo.org/mailman/listinfo/pdal
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Andrew Bell
>>>>>>> andrew.bell.ia at gmail.com
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> pdal mailing list
>>>>>> pdal at lists.osgeo.org
>>>>>> https://lists.osgeo.org/mailman/listinfo/pdal
>>>>>>
>>>>>
>>>>>
>>>
>
>
> --
> Andrew Bell
> andrew.bell.ia at gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pdal/attachments/20170804/5c7fa226/attachment-0001.html>