[gdal-dev] four threads are huh, five are buh? Regressions w/ GDAL 3.5.2/PROJ_9.1.0

[Christian Brock]

Thanks, I made a workaround using Java's ThreadLocal and it works 😉

-----Original Message-----
From: Even Rouault <even.rouault at spatialys.com> 
Sent: Wednesday, October 12, 2022 10:31
To: Christian Brock <Christian.Brock at amdocs.com>; gdal-dev at lists.osgeo.org
Subject: Re: [gdal-dev] four threads are huh, five are buh? Regressions w/ GDAL 3.5.2/PROJ_9.1.0

CAUTION: This email is from an external source. Please do not open any unknown links or attachments.

Le 12/10/2022 à 08:59, Christian Brock a écrit :
> Hi Even,
>
> Thanks for the fast answer.
>
> We do not share any gdal datasets, BUT we share all spatial references and all coordinate transformations between threads. It used to work in the past, hmm ...
>
> I assume this is a problem?

yes, this was never guaranteed to be safe, even if it was mostly in practice before PROJ 6. But with PROJ >= 6, you can't use the same SpatialReference or CoordinateTranfsormation concurrently from several threads, even for read-only operations, since some internal state might be changed.

Even

>
> Cheers --Chr.
>
> -----Original Message-----
> From: Even Rouault <even.rouault at spatialys.com>
>
> Christian,
>
> I suspect the issue is not that much jumping from 4 to 5 threads, but some latent issue in your code or in GDAL/PROJ that becomes more visible. First thing to check: are you sure you are not using a given GDAL / OGR object simultaneously in more than one thread ?
>
> If that's not the issue, if you want effective help, I'm afraid you'll 
> have to go through the effort of writing a minimal standalone 
> reproducer
> (code+data) that you can share with us.
>
> Even
>
> Le 11/10/2022 à 14:56, Christian Brock via gdal-dev a écrit :
>> I have some special issue and would be glad for suggestions.
>>
>> We run complex regression tests using environmental files -- too 
>> complex for a bug report ;-(
>>
>> Moving from GDAL_2.4.2/PROJ_5.2.0 to GDAL 3.5.2/PROJ_9.1.0 we run into regressions.
>>
>> The most stunning observation is that the issue only occurs when USING MORE THAN FOUR THREADS!
>> Can anyone think of something that is true with five or more threads but not below?
>>
>> Besides this I should mention, that we use Java bindings
>>
>> The issue is
>> - file format independent,
>> - were observed on ubuntu 20.04 and 22.04,
>> - affects a sample with the following coordinate system:
>> PROJCS["Transverse Mercator",GEOGCS["Potsdam_IST_V2_0",DATUM["Potsdam_IST_V2_0",SPHEROID["Bessel",6377397.155,299.152813106079],TOWGS84[583,68,399.5,0,0,1.36E-05,1.13E-05]],PRIMEM["Greenwich",0],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]]],PROJECTION["Transverse_Mercator"],PARAMETER["latitude_of_origin",0],PARAMETER["central_meridian",9],PARAMETER["scale_factor",1],PARAMETER["false_easting",3500000],PARAMETER["false_northing",0],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["Easting",EAST],AXIS["Northing",NORTH]]"
>> - WGS84 and all UTM systems are good.
>>
>> I should also mention, that this issue affects only the second sample, when I run more than one sample with the PROJCS above.
>> When I change the sample order, it is the second sample again!
>>
>> And as a final point, there are random memory corruptions every couple of hours.
>>
>> So again, please step forward when you know of anything inside GDAL, PROJ or Java bindings that cares about the number of threads.
>>
>> Many thanks
>> --Chr.
>>
>>
>>
>> This email and the information contained herein is proprietary and 
>> confidential and subject to the Amdocs Email Terms of Service, which 
>> you may review at https://www.amdocs.com/about/email-terms-of-service
>> <https://www.amdocs.com/about/email-terms-of-service>
>>
>> _______________________________________________
>> gdal-dev mailing list
>> gdal-dev at lists.osgeo.org
>> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flis
>> t 
>> s.osgeo.org%2Fmailman%2Flistinfo%2Fgdal-dev&data=05%7C01%7CChrist
>> i
>> an.Brock%40amdocs.com%7C2ed3fa7f029246255d3808daab8a13c3%7Cc8eca3ca12
>> 7 
>> 646d59d9da0f2a028920f%7C0%7C0%7C638010906883663070%7CUnknown%7CTWFpbG
>> Z
>> sb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%
>> 3
>> D%7C3000%7C%7C%7C&sdata=nOmLEy8X%2FU%2FMlSh1PPvvwRFsYH17H%2F8Fpm1
>> 7
>> EWPtx5U%3D&reserved=0
> --
> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.s
> patialys.com%2F&data=05%7C01%7CChristian.Brock%40amdocs.com%7C8383
> ff98852c4c5b087c08daac2c2fa3%7Cc8eca3ca127646d59d9da0f2a028920f%7C0%7C
> 0%7C638011603036795920%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJ
> QIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=
> aRs8p7jleiUviFpxeNSI1KmIx21Ig0QrbEt%2Bs6QiY08%3D&reserved=0
> My software is free, but my time generally not.
>
> This email and the information contained herein is proprietary and 
> confidential and subject to the Amdocs Email Terms of Service, which 
> you may review at https://www.amdocs.com/about/email-terms-of-service 
> <https://www.amdocs.com/about/email-terms-of-service>
>
--
https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.spatialys.com%2F&data=05%7C01%7CChristian.Brock%40amdocs.com%7C8383ff98852c4c5b087c08daac2c2fa3%7Cc8eca3ca127646d59d9da0f2a028920f%7C0%7C0%7C638011603036795920%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=aRs8p7jleiUviFpxeNSI1KmIx21Ig0QrbEt%2Bs6QiY08%3D&reserved=0
My software is free, but my time generally not.

This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service <https://www.amdocs.com/about/email-terms-of-service>