[pdal] Threading and PDAL

[Andrew Bell]

Hi Jason,

I pushed a change today that should eliminate the corruption that you were
seeing when using preview/prepare with stages in multiple threads.  You
should be able to remove the locks that you added in your application.
That said, we've never advertised PDAL as being threadsafe.  But the
problem exhibited by your test program (thank you) has been addressed.

On Thu, Jul 20, 2017 at 4:03 PM, Jason Beverage <jasonbeverage at gmail.com>
wrote:

> You're awesome!  Locking around prepare and preview appear to work.  I'm
> going to see if I can take some of the other locks out as well.
>
> I'm assuming that the SpatialReference locks are due to GDAL's lack of
> thread safety.  Would it be possible to just have an internal lock in PDAL
> around the SpatialReference class?
>
> We have to do the same thing in our osgEarth project with respect to GDAL.
>
> Jason
>
>
>
> On Thu, Jul 20, 2017 at 3:36 PM Connor Manning <connor at hobu.co> wrote:
>
>> You'll need a lock around anywhere a `pdal::SpatialReference` is
>> created.  For Entwine, prepare and preview are the main places that
>> happens, but you'll also need to lock if you're explicitly going to create
>> one.
>>
>> On Thu, Jul 20, 2017 at 2:21 PM, Jason Beverage <jasonbeverage at gmail.com>
>> wrote:
>>
>>> Thanks Connor, I just took a look at the executor.cpp.
>>>
>>> I see locking around stage creation which I already have.  I also see
>>> explicit locking around the prepare call and the preview, which I'm not
>>> doing so I will try adding that.
>>>
>>> Is there anything else that you can think of I might be missing?
>>>
>>> Jason
>>>
>>> On Thu, Jul 20, 2017 at 3:08 PM Connor Manning <connor at hobu.co> wrote:
>>>
>>>> Entwine is multi-threaded.  Most of the PDAL-locking in Entwine happens
>>>> in `entwine/util/executor.cpp` and `hpp`.
>>>>
>>>> On Thu, Jul 20, 2017 at 2:02 PM, Jason Beverage <
>>>> jasonbeverage at gmail.com> wrote:
>>>>
>>>>> Hi Andrew,
>>>>>
>>>>> Basically I have a bunch of jobs that read from input files using
>>>>> PDAL, do something on them, and then write some resulting points out to an
>>>>> LAZ file.  I've got all Stage creation wrapped in a mutex and I was hoping
>>>>> that the stage pipelines for each job could run independently without a
>>>>> lock.  I'm getting random access violations in PDAL.  I'm going to try to
>>>>> build pdal in debug and see if I can figure out what is going on.  If I
>>>>> globally mutex any access at all to PDAL things run just fine (although not
>>>>> in parallel :) ).
>>>>>
>>>>> Does Entwine use threading or is it a strictly multiprocess setup?
>>>>>
>>>>> Jason
>>>>>
>>>>>
>>>>> On Thu, Jul 20, 2017 at 10:17 AM Andrew Bell <andrew.bell.ia at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> The stage creation code through the PluginManager and StageManager is
>>>>>> believed to be threadsafe.  Nothing else is guaranteed.  We rely on many
>>>>>> libraries which may or may not be threadsafe.  If you need thread safety,
>>>>>> you need to implement your own locking.
>>>>>>
>>>>>> If you're getting thread-related crashes in PluginManager or
>>>>>> StageManager, please open a ticket.
>>>>>>
>>>>>> That said, there is probably a lot you can get away with -- we don't
>>>>>> rely much on constructs that are inherently non-threadsafe.  I'm willing to
>>>>>> look at issues up to a point, but we're not going to work hard to make
>>>>>> things threadsafe.  I'd need to know more about specific problems to be
>>>>>> helpful.
>>>>>>
>>>>>>
>>>>>> On Thu, Jul 20, 2017 at 10:06 AM, Jason Beverage <
>>>>>> jasonbeverage at gmail.com> wrote:
>>>>>>
>>>>>>> I'm running into some threading issues with PDAL that I could use
>>>>>>> some help with.  I'm running into some random crashing using PDAL in a
>>>>>>> multithreaded manner.  I understand that PDAL isn't entirely threadsafe,
>>>>>>> but do you have some info on what is and isn't threadsafe?
>>>>>>>
>>>>>>> Basically what I'm trying to do is multiple threads are creating
>>>>>>> Stages using a shared StageFactory and each thread is running it's own
>>>>>>> independent pipeline.  The only thing that is shared is the StageFactory
>>>>>>> which is static for the application.  I've got access to the StageFactory
>>>>>>> protected by my own mutex since I was getting crashes accessing it from
>>>>>>> multiple threads.  I tried creating a StageFactory per thread but that
>>>>>>> caused other issues like stages randomly not being able to be able to be
>>>>>>> created and it not loading my plugins all the time.
>>>>>>>
>>>>>>> Right now I'm using the OSGEO4W 64 bit package to get PDAL and all
>>>>>>> of it's dependencies and building my application against that.  It looks
>>>>>>> like the pdal version is 1.5.0 which is the latest release.
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> Jason
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> pdal mailing list
>>>>>>> pdal at lists.osgeo.org
>>>>>>> https://lists.osgeo.org/mailman/listinfo/pdal
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Andrew Bell
>>>>>> andrew.bell.ia at gmail.com
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> pdal mailing list
>>>>> pdal at lists.osgeo.org
>>>>> https://lists.osgeo.org/mailman/listinfo/pdal
>>>>>
>>>>
>>>>
>>


-- 
Andrew Bell
andrew.bell.ia at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pdal/attachments/20170804/76904019/attachment.html>