[gdal-dev] gdal-dev Digest, Vol 244, Issue 46
Fox, Shawn D (US)
shawn.fox at baesystems.us
Fri Sep 20 08:05:32 PDT 2024
I believe I have fixed it by creating a simple struct that can be created in automatic scope of the main function and when it is destructed at the end of the function block it seems to work fine. The problem occurs when we have a static pointer that prevents the destruction of the resources until well after our main function exits. That's one of the downsides of having singleton classes with static instance pointers.
I think there is a typo in the GDALDestroy documentation. This statement, "Prior to GDAL 2.4.0, this function should normally be explicitly called by application code if GDAL is dynamically linked (but that does not hurt), since it was automatically called through the unregistration mechanisms of dynamic library loading." Should probably read that "this function should NOT normally be explicitly called.", otherwise the phrase but that does not hurt in parens doesn't make sense.
I was thinking that something related to TLS must have been destroyed and something called within GDALDestroy might be trying to double free a resource. There is an occasional error about a linked list being corrupted which is a potential bug. Regardless of the issues that can happen during static destruction I am not sure how a linked list could be corrupted. The body of that function where the crash occurs looks straightforward enough.
I just wanted to thank the community for responses and let you know that I have a solution that works for us. A struct with a constructor and destructor allowing me to creating something in a block scope to manage cleanup seems to fix the issue.
Shawn Fox
-----Original Message-----
From: gdal-dev <gdal-dev-bounces at lists.osgeo.org> On Behalf Of gdal-dev-request at lists.osgeo.org
Sent: Thursday, September 19, 2024 7:00 AM
To: gdal-dev at lists.osgeo.org
Subject: gdal-dev Digest, Vol 244, Issue 46
External Email Alert
This email has been sent from an account outside of the BAE Systems network.
Please treat the email with caution, especially if you are requested to click on a link, decrypt/open an attachment, or enable macros. For further information on how to spot phishing, access “Cybersecurity OneSpace Page” and report phishing by clicking the button “Report Phishing” on the Outlook toolbar.
Send gdal-dev mailing list submissions to
gdal-dev at lists.osgeo.org
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.osgeo.org/mailman/listinfo/gdal-dev
or, via email, send a message with subject or body 'help' to
gdal-dev-request at lists.osgeo.org
You can reach the person managing the list at
gdal-dev-owner at lists.osgeo.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of gdal-dev digest..."
Today's Topics:
1. Re: Call to GDALDestroy results in occasional core dump, GDAL
3.4.2 (Even Rouault)
2. Re: WMS supported SRS (Micha? Kowalczuk)
----------------------------------------------------------------------
Message: 1
Date: Thu, 19 Sep 2024 14:53:31 +0200
From: Even Rouault <even.rouault at spatialys.com>
To: gdal-dev at lists.osgeo.org
Subject: Re: [gdal-dev] Call to GDALDestroy results in occasional core
dump, GDAL 3.4.2
Message-ID: <2dbda9fa-6f34-4bd9-b491-cb51e1fbf5e8 at spatialys.com>
Content-Type: text/plain; charset="utf-8"; Format="flowed"
If your program uses threads, and you don't make sure to wait for their termination before calling GDALDestroy, this could explain memory corruption if they use GDAL TLS, while GDALDestroy() is cleaning the TLS structures. I'll stop random guessing here.
And you can just get rid of the GDALDestroy() call if it is an annoyance. It just frees memory, which process termination does anyway.
It is just there so that no still-reachable memory is reported when running under Valgrind or other similar analyzers.
Le 19/09/2024 ? 03:51, Fox, Shawn D (US) via gdal-dev a ?crit?:
>
> Just for fun, I changed by code back to cause the crash again.? Now
> it?s difficult to reproduce but here is the part of the callstack
> showing where the issue occurs.? The CPLCleanupTLSList function is the
> function of interest as it loops over some type of list.? Within a
> conditional it is adding two numbers together in an array offset and
> testing a pointer to determine whether to free some memory.? This is
> far down into the weeds of GDAL so I?ve no idea what this function is
> doing but it?s above my level of understanding of GDAL internals.? We
> don?t attempt to use any networking capabilities within GDAL as far as
> I know.? When I get it to crash again I?ll step into it some more to
> see if something jumps out at me.
>
> #0? 0x00007fc79b81e241 in unlink_chunk.isra () from /lib64/libc.so.6
>
> #1? 0x00007fc79b81e465 in malloc_consolidate () from /lib64/libc.so.6
>
> #2? 0x00007fc79b81fae0 in _int_free () from /lib64/libc.so.6
>
> #3? 0x00007fc792ab9708 in VSIFree (pData=0x9bfd40) at
> cpl_vsisimple.cpp:873
>
> #4? 0x00007fc792a78820 in CPLCleanupTLSList (papTLSList=0x7ab950) at
> cpl_multiproc.cpp:467
>
> #5? 0x00007fc792a799ec in CPLCleanupTLS () at cpl_multiproc.cpp:2247
>
> #6? 0x00007fc79252667c in GDALDriverManager::~GDALDriverManager
> (this=0x848ae0, __in_chrg=<optimized out>) at
> gdaldrivermanager.cpp:274
>
> #7? 0x00007fc792526746 in GDALDriverManager::~GDALDriverManager
> (this=0x848ae0, __in_chrg=<optimized out>) at
> gdaldrivermanager.cpp:335
>
> #8? 0x00007fc792527989 in GDALDestroyDriverManager () at
> gdaldrivermanager.cpp:915
>
> #9? 0x00007fc792520fce in GDALDestroy () at gdaldllmain.cpp:86
>
> Shawn Fox
>
> *From:* Andrew Bell <andrew.bell.ia at gmail.com>
> *Sent:* Wednesday, September 18, 2024 5:50 PM
> *To:* Fox, Shawn D (US) <shawn.fox at baesystems.us>
> *Cc:* gdal-dev at lists.osgeo.org
> *Subject:* Re: [gdal-dev] Call to GDALDestroy results in occasional
> core dump, GDAL 3.4.2
>
> *_External Email Alert_*
>
> *This email has been sent from an account outside of the BAE Systems
> network.*
>
> Please treat the email with caution, especially if you are requested
> to click on a link, decrypt/open an attachment, or enable macros.? For
> further information on how to spot phishing, access ?Cybersecurity
> OneSpace Page? and report phishing by clicking the button ?Report
> Phishing? on the Outlook toolbar.
>
> On Wed, Sep 18, 2024 at 8:32?PM Fox, Shawn D (US) via gdal-dev
> <gdal-dev at lists.osgeo.org> wrote:
>
> In our case we have a singleton class that acts as a fa?ade and
> all of our calls to GDAL Apis are done by the methods of this
> class.? The rest of our code base only interacts with the
> singleton so that we only have one project that actually depends
> directly on the GDAL library.? Since the _instance member is a
> static smart pointer the destructor of our class and the
> GDALDestroy function is being called after the main function exits.
>
> This is not a great plan unless you understand exactly the order of
> things being destroyed, which is not well-defined between compilation
> units unless you have done things to?guarantee it. It seems likely
> that GDALDestroy() is attempting to free things already destroyed
> during program tear-down. Since your program is exiting, I can't
> imagine you need to call GDALDestroy() at all. You could also
> eliminate this issue by instantiating your GDAL class-thingee as the
> first line of your program rather than as a static.
>
> --
>
> Andrew Bell
> andrew.bell.ia at gmail.com
>
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
--
http://www.spatialys.com
My software is free, but my time generally not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240919/728d20e8/attachment-0001.htm>
------------------------------
Message: 2
Date: Thu, 19 Sep 2024 15:59:56 +0200
From: Micha? Kowalczuk <michkowalczuk at gmail.com>
To: Rahkonen Jukka <jukka.rahkonen at maanmittauslaitos.fi>
Cc: "gdal-dev at lists.osgeo.org" <gdal-dev at lists.osgeo.org>
Subject: Re: [gdal-dev] WMS supported SRS
Message-ID:
<CAP+FqQCv6GzepM3fYkgPd-oRcmQRoiiFj5NL_OvJ7=ZhuRfuSQ at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Hi,
After re-thinking this topic, I also came to conclusion that using BoundingBox tags will not be a satisfactory solution for GDAL users. Maybe it will better than current, but still poor...
In my opinion the most flexible solution is to provide a list of supported SRSs by a server, just like in *OGR_L_GetSupportedSRSList for WFS.* Then add a helper function to generate a valid path for chosen layer (subdataset), SRS from the list, and providing reprojection of bounding box if needed (as Jukka suggested in last message in last sentence).
This ensures users to use the full capabilities of the server.
Best,
Micha? Kowalczuk
pt., 13 wrz 2024 o 15:15 Rahkonen Jukka <jukka.rahkonen at maanmittauslaitos.fi>
napisa?(a):
> Hi,
>
>
>
> GDAL is open source, of course you can open a feature request. It will
> then be implemented, closed, of left open eternally.
>
>
>
> What do you plan to write into the feature request? I think that using
> the CRS of the additional BoundingBox is a good default and best for
> the users when they want to get data in that CRS. But if they want to
> use data in some other CRS and their favorite CRS appears in some CRS
> list in the GetCapabilities, even without additional BoundingBox, then
> they should use that for communicating with the WMS server. That?s
> because the WMS server usually sends much better quality maps when it
> re-projects the source data for rendering, compared to the result that
> is achieved when GDAL warps the rasters that it if fetching with GetMaps in another CRS.
>
>
>
> So, after re-thinking your original idea about creating a subdataset
> from each supported CRS is good, but because of potentially thousands
> of subdatasets it may lead to some usability problems. I am not sure
> if there can be some problems in implementation as well. I see that
> GDAL shows the subdatasets with BBOX:
>
>
>
> SUBDATASET_26_NAME=WMS:
> http://localhost:8080/geoserver/ows?SERVICE=WMS&VERSION=1.3.0&REQUEST=
> GetMap&LAYERS=topp%3Atasmania_state_boundaries&CRS=CRS:84&BBOX=143.834
> 82400000003,-43.648056,148.47914100000003,-39.573891
>
> However, usually there are additional BoundingBoxes only for some of
> the supported CRSs, and for the rest GDAL should probably warp the
> <EX_GeographicBoundingBox>.
>
>
> -Jukka Rahkonen-
>
>
>
>
>
> *L?hett?j?:* Micha? Kowalczuk <michkowalczuk at gmail.com>
> *L?hetetty:* perjantai 13. syyskuuta 2024 15.12
> *Vastaanottaja:* Rahkonen Jukka <jukka.rahkonen at maanmittauslaitos.fi>
> *Kopio:* gdal-dev at lists.osgeo.org
> *Aihe:* Re: [gdal-dev] WMS supported SRS
>
>
>
> So, can I open a feature request?
>
>
>
> Best,
>
> Michal Kowalczuk
>
>
>
> W dniu czw., 12.09.2024 o 13:57 Rahkonen Jukka <
> jukka.rahkonen at maanmittauslaitos.fi> napisa?(a):
>
> Hi,
>
>
>
> Your suggestion to utilize the additional layer BBOXes if they exist
> feels good to me, and better than to use the first advertised SRS from
> the root level. Beyond that what can be done with parsing the
> GetCapabilities is limited in my mind, because all projections which
> are supported may not be useful in real life.
>
>
>
> -Jukka Rahkonen-
>
>
>
> *L?hett?j?:* Micha? Kowalczuk <michkowalczuk at gmail.com>
> *L?hetetty:* torstai 12. syyskuuta 2024 14.35
>
>
> *Vastaanottaja:* Rahkonen Jukka <jukka.rahkonen at maanmittauslaitos.fi>
> *Kopio:* gdal-dev at lists.osgeo.org
> *Aihe:* Re: [gdal-dev] WMS supported SRS
>
>
>
> So we remain at the point that there is no good method that, using
> only GDAL, will allow to fully use the WMS service without additional
> work related to XML parsing...?
>
>
>
> Best,
>
> Micha? Kowalczuk
>
>
>
> wt., 10 wrz 2024 o 17:06 Micha? Kowalczuk <michkowalczuk at gmail.com>
> napisa?(a):
>
> A sample from your link:
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *<Layer queryable="1" opaque="0">
> <Name>Rakennustietoruudukko_2017</Name>
> <Title>Rakennustietoruudukko 2017</Title> <Abstract>Tietoa
> rakennuksista
> 250 x 250m ruudukoissa p??kaupunkiseudulla</Abstract> <KeywordList>
> <Keyword>features</Keyword>
> <Keyword>Rakennustietoruudukko_2017</Keyword>
> </KeywordList> <CRS>EPSG:3879</CRS> <CRS>CRS:84</CRS>
> <EX_GeographicBoundingBox>
> <westBoundLongitude>24.50112802448554</westBoundLongitude>
> <eastBoundLongitude>25.25849178466466</eastBoundLongitude>
> <southBoundLatitude>60.052265324029165</southBoundLatitude>
> <northBoundLatitude>60.39874702231777</northBoundLatitude>
> </EX_GeographicBoundingBox> <BoundingBox CRS="CRS:84"
> minx="24.50112802448554" miny="60.052265324029165" maxx="25.25849178466466"
> maxy="60.39874702231777"/> <BoundingBox CRS="EPSG:3879" minx="6659998.5"
> miny="2.5472498E7" maxx="6698499.5" maxy="2.551425E7"/> <Style>
> <Name>polygon</Name> <Title>Default polygon style</Title> <Abstract>A
> sample style that just draws out a solid gray interior with a black
> 1px outline</Abstract> <LegendURL width="20" height="20">
> <Format>image/png</Format> <OnlineResource xmlns:xlink="*
> *http://www.w3.org/1999/xlink* <http://www.w3.org/1999/xlink>*"
> xlink:type="simple" xlink:href="*
> *https://kartta.hsy.fi/geoserver/asuminen_ja_maankaytto/ows?service=WM
> S&version=1.3.0&request=GetLegendGraphic&format=image%2Fpng&width=20&h
> eight=20&layer=Rakennustietoruudukko_2017*
> <https://kartta.hsy.fi/geoserver/asuminen_ja_maankaytto/ows?service=WM
> S&version=1.3.0&request=GetLegendGraphic&format=image%2Fpng&width=20&h
> eight=20&layer=Rakennustietoruudukko_2017>
>
>
> *"/> </LegendURL> </Style> </Layer>*
>
>
>
> wt., 10 wrz 2024 o 17:05 Micha? Kowalczuk <michkowalczuk at gmail.com>
> napisa?(a):
>
> Ok, now I see the problem. Thank you Jukka for the explanation with a
> sample.
>
> So maybe we can use information from *BoundingBox*?
>
> and generate SUBDATASETS for each combination of Layer and BoundingBox
> definition that is usually defined for "best"/dedicated projections
> for a layer.
>
>
>
> *(7.2.4.6.8 )*
>
> *A Layer may have multiple BoundingBox elements, but each one shall
> state a different CRS (...)*
>
> *A Layer shall not provide a BoundingBox for a CRS it does not support.
> Conversely, a Layer may support CRSs*
>
> *for which it does not provide a BoundingBox: a server that has the
> ability to transform data to different CRSs may*
>
> *choose not to provide an explicit BoundingBox for every possible CRS
> available for each Layer. The server*
>
> *should provide BoundingBox information for at least the native CRS of
> the Layer (that is, the CRS in which the*
>
> *Layer is stored in the server?s database).*
>
>
>
> Best,
>
> Micha?
>
>
>
> W dniu wt., 10.09.2024 o 16:07 Rahkonen Jukka <
> jukka.rahkonen at maanmittauslaitos.fi> napisa?(a):
>
> Hi
>
>
>
> - I meant to use CRSs only reported by GetCapabilities. I haven't seen
> that GetCapabilities returns so many (6782) possible projections...
>
>
>
> I was meaning the same, CRSs reported in GetCapabilities. I used a
> search engine with phrase ?service=WMS&version=1.3.0? and found in
> three minutes many examples, including this one
> kartta.hsy.fi/geoserver/asuminen_ja_maankaytto/ows?service=WMS&version=1.3.0&request=GetCapabilities.
> If you want to see more, install Geoserver on your computer and have a
> look at the default GetCapabilities.
>
>
>
> Finding the best CRS from WMS GetCapabilities is guesswork. The server
> above is a Finnish one so probably EPSG:3067 is the most native CRS
> because that is what we tend to use in Finland. However, there is also
> one and only one additional BBOX for EPSG:3879 in GetCapabilities, but
> that would be somewhat odd selection for a country-wide CRS. It is
> also possible that for example some CRS of New Zealand does not really
> work in GetMap of the Finnish data even GetCapabilities says it will.
> But when WMS is configured to show a short CRS list then I would trust that they are all good to use.
> Just be aware that the list can also be very long.
>
>
>
> If you start implementing the LIST_ALL_SRS option, it should be noted
> that the service level CRS list applies to all layers, and each layer
> can add more CRSs to the list. Maybe in theory there could be even
> ?crs-list-of-service-root + crs-list-of-group-layer +
> crs-list-of-child-layer?.
>
>
>
> ?7.2.4.6.7 CRS
>
> Every Layer is available in one or more layer coordinate reference
> systems. 6.7.3 discusses the Layer CRS. In
>
> order to indicate which Layer CRSs are available, every named Layer
> shall have at least one <CRS> element that
>
> is either stated explicitly or inherited from a parent Layer. The root
> <Layer> element shall include a sequence of
>
> zero or more CRS elements listing all CRSs that are common to all
> subsidiary layers. A child layer may optionally
>
> add to the list inherited from a parent layer. Any duplication shall
> be ignored by clients.
>
> When a Layer is available in several coordinate reference systems, the
> list of available CRS values shall be
>
> represented as a sequence of <CRS> elements, each of which contains
> only a single CRS name.?
>
>
>
> -Jukka Rahkonen-
>
>
>
>
>
>
>
>
>
> *L?hett?j?:* Micha? Kowalczuk <michkowalczuk at gmail.com>
> *L?hetetty:* tiistai 10. syyskuuta 2024 16.32
> *Vastaanottaja:* Rahkonen Jukka <jukka.rahkonen at maanmittauslaitos.fi>
> *Kopio:* gdal-dev at lists.osgeo.org
>
>
> *Aihe:* Re: [gdal-dev] WMS supported SRS
>
>
>
> Thank you for your feedback. My comments are below:
>
>
>
> *It means that there are tens of thousands WMS services which support
> 6782 different projections for each layer (checked from Geoserver
> version 2.25.3). I would not like them all to be reported as
> subdatasets.*
>
> I meant to use CRSs only reported by GetCapabilities. I haven't seen
> that GetCapabilities returns so many (6782) possible projections...
>
>
>
> *WMS standard also does not define any default CRS and the first one
> on the list in GetCapabilities does not need to be the best.*
>
> Therefore, the current form of the generating subdatasets names method
> for WMS driver is even more useless, if we have a non-default (not the
> best) projection for the data.
>
>
>
> And the only good way to work with it is to parse XML and generate
> paths manually...
>
>
>
> That's why I favor Even's option to introduce open option
> LIST_ALL_SRS=YES/NO.
>
>
>
> Any more thoughts?
>
>
>
> Micha? Kowalczuk
>
>
>
> wt., 10 wrz 2024 o 15:03 Rahkonen Jukka <
> jukka.rahkonen at maanmittauslaitos.fi> napisa?(a):
>
> Hi,
>
>
>
> WMTS typically supports a rather small number of tilematrices and
> tiles are usually cached, so it makes a lot of sense to advertise the
> available matrices and utilize them. On the other hand WMS maps are
> created on-the-fly and there is very low technical cost on the server
> side to support however many projections. WMS standard also does not
> define any default CRS and the first one on the list in
> GetCapabilities does not need to be the best.
>
>
>
> I guess that Geoserver is the most common WMS server in the world and
> amazingly many Geoservers run with the default settings. It means that
> there are tens of thousands WMS services which support 6782 different
> projections for each layer (checked from Geoserver version 2.25.3). I
> would not like them all to be reported as subdatasets.
>
>
>
> I can see that ogrinfo and the OGC API Features driver report the list
> of supported projections. Test with ogrinfo OAPIF:
> https://ogc-api.nrw.de/inspire-us-feuerwehr -al -so shows ?Supported SRS:
> OGC:CRS84, EPSG:25832, EPSG:25833, EPSG:4258, EPSG:4326, EPSG:3395,
> EPSG:3857, EPSG:3034, EPSG:3035?
>
> Something similar might be an option for gdalinfo, but still about
> 6782 EPSG codes per layer is too much to be viewed by default. Maybe
> the CRS list could be reported under some metadata domain that user
> could read on demand?
>
>
>
> -Jukka Rahkonen-
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *L?hett?j?:* gdal-dev <gdal-dev-bounces at lists.osgeo.org> *Puolesta
> *Michal Kowalczuk via gdal-dev
> *L?hetetty:* tiistai 10. syyskuuta 2024 15.15
> *Vastaanottaja:* gdal-dev at lists.osgeo.org
> *Aihe:* Re: [gdal-dev] WMS supported SRS
>
>
>
> I found that there was a similar issue in 2021 without any specific answer:
>
> https://www.mail-archive.com/gdal-dev@lists.osgeo.org/msg35549.html
>
>
>
> I wonder if getting SUBDATASETS shouldn't return the result in a
> similar way as it does for WMTS service,
>
> where GDAL generates paths to all layer/tilematrix/tilematrixset
> combination.
>
> For example if layer is available in 3 SRS, GDAL produces 3
> subdatasets, so using only SUBDATASE_NAME property user has access to
> all map services shared by WMTS server.
>
> In my opinion, GDAL should use the same approach for WMS.
>
>
>
> Does anyone object?
>
> I will then submit a new feature request/bug report.
>
>
>
> Best regards and have a nice day
>
> Micha? Kowalczuk
>
>
>
> pon., 9 wrz 2024 o 16:17 Micha? Kowalczuk <michkowalczuk at gmail.com>
> napisa?(a):
>
> Hi,
>
> Does GDAL provides a method to get supported spatial reference systems
> by WMS service?
>
> *GDALGetMetadata(SUBDATASETS) *generate links only for the first
> (default) SRS from GetCapabilities, so this won't be helpful.
>
> Working with WFS I can get supported SRS using OGR
> *OGR_L_GetSupportedSRSList *function.
>
> Does GDAL offers something similar for WMS, or should I get this
> information from GetCapabilities XML by myself?
>
>
>
> Regards,
>
> Michal
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240919/cac61615/attachment.htm>
------------------------------
Subject: Digest Footer
_______________________________________________
gdal-dev mailing list
gdal-dev at lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
------------------------------
End of gdal-dev Digest, Vol 244, Issue 46
*****************************************
More information about the gdal-dev
mailing list