From mazin.marwan0 at gmail.com Wed Oct 1 11:54:40 2025 From: mazin.marwan0 at gmail.com (Mazin Marwan) Date: Wed, 1 Oct 2025 11:54:40 -0700 Subject: [gdal-dev] Generating Python Bindings for GDAL 3.11.4 In-Reply-To: References: Message-ID: Further on this, I've tried to generate the bindings myself by duplicating swig/python/setup.py.in and replacing the template variables. However I'm running into this error when I run C:\Users\ma005968\AppData\Local\Programs\Python\Python313\python.exe setup.py gdal_wrap.cpp c1xx: fatal error C1083: Cannot open source file: 'extensions/gdal_wrap.cpp': No such file or directory gnm_wrap.cpp c1xx: fatal error C1083: Cannot open source file: 'extensions/gnm_wrap.cpp': No such file or directory The readme.rst isn't incredibly clear on how to generate the python bindings myself and this was my understanding of how to do it as running the cmake command there wasn't working either. On Mon, Sep 29, 2025 at 3:08?PM Mazin Marwan wrote: > Hi all, > > I've installed GDAL 3.11.4 using VCPKG and am now trying to generate the > python bindings using: > > pip install gdal > > However I'm running into the following error: > > ERROR: Failed building wheel for gdal > Failed to build gdal > error: failed-wheel-build-for-install > > ? Failed to build installable wheels for some pyproject.toml based projects > ??> gdal > > A bit of research online has led me to believe that the pip version is > just broken and that the fix is installing a .whl file from > cgholke's geospatial wheels repo but that seems like a workaround instead > of a proper fix. Is there another way to fix this? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mazin.marwan0 at gmail.com Thu Oct 2 14:32:52 2025 From: mazin.marwan0 at gmail.com (Mazin Marwan) Date: Thu, 2 Oct 2025 14:32:52 -0700 Subject: [gdal-dev] Location of gdal.dll GDAL 3.11.4 Message-ID: Hello again, Sorry for all the questions lately. I'm looking for the .dll file produced when building and installing gdal from source and I found 4 gdald.dll files in different locations. Is there a particular one I should be using for my own projects? -------------- next part -------------- An HTML attachment was scrubbed... URL: From mazin.marwan0 at gmail.com Thu Oct 2 14:39:56 2025 From: mazin.marwan0 at gmail.com (Mazin Marwan) Date: Thu, 2 Oct 2025 14:39:56 -0700 Subject: [gdal-dev] Location of gdal.dll GDAL 3.11.4 In-Reply-To: References: Message-ID: Additionally, are the executables in the Debug folder the ones I should be using in my own project? On Thu, Oct 2, 2025 at 2:32?PM Mazin Marwan wrote: > Hello again, > > Sorry for all the questions lately. I'm looking for the .dll file produced > when building and installing gdal from source and I found 4 gdald.dll files > in different locations. Is there a particular one I should be using for my > own projects? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From afernandez at odyhpc.com Fri Oct 3 08:02:29 2025 From: afernandez at odyhpc.com (afernandez at odyhpc.com) Date: Fri, 03 Oct 2025 15:02:29 +0000 Subject: [gdal-dev] How to specify 'OUTPUT' when warping (reproject) Message-ID: Hello, A PyQGIS code needs to reproject one raster layer from a CRS to a new one. To begin with, I've tried to use only the minimal required options so the snippet reads: parameters = { 'INPUT': my_input_vector_layer, 'SOURCE_CRS': QgsCoordinateReferenceSystem(source_PROJ), 'TARGET_CRS': QgsCoordinateReferenceSystem(target_PROJ), 'RESAMPLING': 1, 'DATA_TYPE': 4, 'OUTPUT': '/home/ubuntu/my_new_raster.tif' } terrain = processing.run('gdal:warpreproject', parameters) However, this doesn't do much as terrain is simply a dictionary with the keyword 'OUTPUT' and my_new_raster.tif is never created. Obviously I must be doing something wrong and suspect that might be related to 'OUTPUT'. The documentation at https://docs.qgis.org/testing/en/docs/user_manual/processing/toolbox.html#output-parameter-widget didn't dissipate my doubts so I'm unsure how to proceed. I also tried (an alternative method) calling gdal.Warp directly (not sure if this method is preferable to using processing.run) but it keeps saying that it only takes 2 arguments. I found a few examples online but they all have more than 2 arguments. For example options = gdal.WarpOptions(dstSRS=target_PROJ) warp = gdal.Warp('/home/ubuntu/pepe.tif', my_input_vector_layer, options) warp = None complains about including 3 and not 2 arguments. Same if I try including all arguments at warp. Thanks. From jmckenna at gatewaygeomatics.com Fri Oct 3 09:35:42 2025 From: jmckenna at gatewaygeomatics.com (Jeff McKenna) Date: Fri, 3 Oct 2025 13:35:42 -0300 Subject: [gdal-dev] Location of gdal.dll GDAL 3.11.4 In-Reply-To: References: Message-ID: <0af707f3-ed9f-4b6e-95c9-aec45048bdf4@gatewaygeomatics.com> Try setting the "GDAL_LIB_OUTPUT_NAME" CMake variable, to something like "mazin-gdal" and then you can use that "mazin-gdal.dll" file in your projects. -jeff -- Jeff McKenna GatewayGeo: Developers of MS4W, & offering MapServer Consulting/Dev co-founder of FOSS4G http://gatewaygeo.com/ On 2025-10-02 6:32 p.m., Mazin Marwan via gdal-dev wrote: > Hello again, > > Sorry for all the questions lately. I'm looking for the .dll file > produced when building and installing gdal from source and I found 4 > gdald.dll files in different locations. Is there a particular one I > should be using for my own projects? > From rupestre.campos at gmail.com Fri Oct 3 09:35:37 2025 From: rupestre.campos at gmail.com (ky) Date: Fri, 3 Oct 2025 13:35:37 -0300 Subject: [gdal-dev] How to specify 'OUTPUT' when warping (reproject) In-Reply-To: References: Message-ID: Hi, I don't really know what you are trying to do with a vector layer as input for gdal.Warp as it is used more like a cutline to clip a raster rather as an input, as gdal.Warp expects a raster input/output but to make it work you must pass options as keyword argument, not as positional argument, like bellow: warp = gdal.Warp('/home/ubuntu/pepe.tif', my_input_vector_layer, options=options) Hope it helps! On Fri, Oct 3, 2025 at 1:01?PM AF via gdal-dev wrote: > Hello, > A PyQGIS code needs to reproject one raster layer from a CRS to a new > one. To begin with, I've tried to use only the minimal required options > so the snippet reads: > parameters = { > 'INPUT': my_input_vector_layer, > 'SOURCE_CRS': QgsCoordinateReferenceSystem(source_PROJ), > 'TARGET_CRS': QgsCoordinateReferenceSystem(target_PROJ), > 'RESAMPLING': 1, > 'DATA_TYPE': 4, > 'OUTPUT': '/home/ubuntu/my_new_raster.tif' > } > terrain = processing.run('gdal:warpreproject', parameters) > However, this doesn't do much as terrain is simply a dictionary with the > keyword 'OUTPUT' and my_new_raster.tif is never created. Obviously I > must be doing something wrong and suspect that might be related to > 'OUTPUT'. The documentation at > > https://docs.qgis.org/testing/en/docs/user_manual/processing/toolbox.html#output-parameter-widget > didn't dissipate my doubts so I'm unsure how to proceed. > I also tried (an alternative method) calling gdal.Warp directly (not > sure if this method is preferable to using processing.run) but it keeps > saying that it only takes 2 arguments. I found a few examples online but > they all have more than 2 arguments. For example > options = gdal.WarpOptions(dstSRS=target_PROJ) > warp = gdal.Warp('/home/ubuntu/pepe.tif', > my_input_vector_layer, options) > warp = None > complains about including 3 and not 2 arguments. Same if I try including > all arguments at warp. > Thanks. > _______________________________________________ > gdal-dev mailing list > gdal-dev at lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nyall.dawson at gmail.com Fri Oct 3 14:30:52 2025 From: nyall.dawson at gmail.com (Nyall Dawson) Date: Sat, 4 Oct 2025 07:30:52 +1000 Subject: [gdal-dev] How to specify 'OUTPUT' when warping (reproject) In-Reply-To: References: Message-ID: On Sat, 4 Oct 2025, 2:00?am AF via gdal-dev, wrote: > Hello, > A PyQGIS code needs to reproject one raster layer from a CRS to a new > one. To begin with, I've tried to use only the minimal required options > so the snippet reads: > parameters = { > 'INPUT': my_input_vector_layer, > Not sure if this is a typo (or a badly named variable, or ai slop code), but it looks like you're trying to warp a vector later to a raster layer? ? Nyall 'SOURCE_CRS': QgsCoordinateReferenceSystem(source_PROJ), > 'TARGET_CRS': QgsCoordinateReferenceSystem(target_PROJ), > 'RESAMPLING': 1, > 'DATA_TYPE': 4, > 'OUTPUT': '/home/ubuntu/my_new_raster.tif' > } > terrain = processing.run('gdal:warpreproject', parameters) > However, this doesn't do much as terrain is simply a dictionary with the > keyword 'OUTPUT' and my_new_raster.tif is never created. Obviously I > must be doing something wrong and suspect that might be related to > 'OUTPUT'. The documentation at > > https://docs.qgis.org/testing/en/docs/user_manual/processing/toolbox.html#output-parameter-widget > didn't dissipate my doubts so I'm unsure how to proceed. > I also tried (an alternative method) calling gdal.Warp directly (not > sure if this method is preferable to using processing.run) but it keeps > saying that it only takes 2 arguments. I found a few examples online but > they all have more than 2 arguments. For example > options = gdal.WarpOptions(dstSRS=target_PROJ) > warp = gdal.Warp('/home/ubuntu/pepe.tif', > my_input_vector_layer, options) > warp = None > complains about including 3 and not 2 arguments. Same if I try including > all arguments at warp. > Thanks. > _______________________________________________ > gdal-dev mailing list > gdal-dev at lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From afernandez at odyhpc.com Fri Oct 3 15:17:03 2025 From: afernandez at odyhpc.com (afernandez at odyhpc.com) Date: Fri, 03 Oct 2025 22:17:03 +0000 Subject: [gdal-dev] How to specify 'OUTPUT' when warping (reproject) In-Reply-To: References: Message-ID: Hi, I apologize for my clumsiness as the source layer is definitely of the raster type as mentioned in the initial paragraph (just made an error while substituting the layer name in the code with a more general label, so it should have read my_input_raster_layer where previously typed my_input_vector_layer). Anyway, I tried your suggestion and made 'options' a keyword argument rather than a positional one: options = gdal.WarpOptions(dstSRS=target_PROJ) warp = gdal.Warp('/home/ubuntu/my_new_raster.tif', my_input_raster_layer, options=options) warp = None This got over the previous error but generated a new one that I have been trying to figure out all day long: File "/usr/lib/python3/dist-packages/osgeo/gdal.py", line 1116, in Warp return wrapper_GDALWarpDestName(destNameOrDestDS, srcDSTab, opts, callback, callback_data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/osgeo/gdal.py", line 6110, in wrapper_GDALWarpDestName return _gdal.wrapper_GDALWarpDestName(*args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TypeError: object of wrong GDALDatasetShadow I don't understand if this new error is due to some syntax mishap at my end (have tried several variations) or if the source raster layer needs to meet some specific requirements. Thanks. On 2025-10-03 16:35, ky wrote: > Hi, > I don't really know what you are trying to do with a vector layer as > input for gdal.Warp as it is used more like a cutline to clip a raster > rather as an input, as gdal.Warp expects a raster input/output but > to make it work you must pass options as keyword argument, not as > positional argument, like bellow: > > warp = gdal.Warp('/home/ubuntu/pepe.tif', my_input_vector_layer, > options=options) > > Hope it helps! > > On Fri, Oct 3, 2025 at 1:01?PM AF via gdal-dev > wrote: > >> Hello, >> A PyQGIS code needs to reproject one raster layer from a CRS to a >> new >> one. To begin with, I've tried to use only the minimal required >> options >> so the snippet reads: >> parameters = { >> 'INPUT': my_input_vector_layer, >> 'SOURCE_CRS': >> QgsCoordinateReferenceSystem(source_PROJ), >> 'TARGET_CRS': >> QgsCoordinateReferenceSystem(target_PROJ), >> 'RESAMPLING': 1, >> 'DATA_TYPE': 4, >> 'OUTPUT': '/home/ubuntu/my_new_raster.tif' >> } >> terrain = processing.run('gdal:warpreproject', >> parameters) >> However, this doesn't do much as terrain is simply a dictionary with >> the >> keyword 'OUTPUT' and my_new_raster.tif is never created. Obviously I >> >> must be doing something wrong and suspect that might be related to >> 'OUTPUT'. The documentation at >> > https://docs.qgis.org/testing/en/docs/user_manual/processing/toolbox.html#output-parameter-widget >> >> didn't dissipate my doubts so I'm unsure how to proceed. >> I also tried (an alternative method) calling gdal.Warp directly (not >> >> sure if this method is preferable to using processing.run) but it >> keeps >> saying that it only takes 2 arguments. I found a few examples online >> but >> they all have more than 2 arguments. For example >> options = gdal.WarpOptions(dstSRS=target_PROJ) >> warp = gdal.Warp('/home/ubuntu/pepe.tif', >> my_input_vector_layer, options) >> warp = None >> complains about including 3 and not 2 arguments. Same if I try >> including >> all arguments at warp. >> Thanks. >> _______________________________________________ >> gdal-dev mailing list >> gdal-dev at lists.osgeo.org >> https://lists.osgeo.org/mailman/listinfo/gdal-dev From afernandez at odyhpc.com Fri Oct 3 15:43:45 2025 From: afernandez at odyhpc.com (afernandez at odyhpc.com) Date: Fri, 03 Oct 2025 22:43:45 +0000 Subject: [gdal-dev] How to specify 'OUTPUT' when warping (reproject) In-Reply-To: References: Message-ID: <514b3e50ddb0a31e7ab8e857cd212ec0@odyhpc.com> Hi Nyall, It was a typo (pretty clumsy at my end) as it should have read 'my_input_raster_layer' rather than 'my_input_vector_layer.' The source/input layer is a raster. Thanks. On 2025-10-03 21:30, Nyall Dawson wrote: > On Sat, 4 Oct 2025, 2:00?am AF via gdal-dev, > wrote: > >> Hello, >> A PyQGIS code needs to reproject one raster layer from a CRS to a >> new >> one. To begin with, I've tried to use only the minimal required >> options >> so the snippet reads: >> parameters = { >> 'INPUT': my_input_vector_layer, > > Not sure if this is a typo (or a badly named variable, or ai slop > code), but it looks like you're trying to warp a vector later to a > raster layer? ? > > Nyall > >> 'SOURCE_CRS': >> QgsCoordinateReferenceSystem(source_PROJ), >> 'TARGET_CRS': >> QgsCoordinateReferenceSystem(target_PROJ), >> 'RESAMPLING': 1, >> 'DATA_TYPE': 4, >> 'OUTPUT': '/home/ubuntu/my_new_raster.tif' >> } >> terrain = processing.run('gdal:warpreproject', >> parameters) >> However, this doesn't do much as terrain is simply a dictionary with >> the >> keyword 'OUTPUT' and my_new_raster.tif is never created. Obviously I >> >> must be doing something wrong and suspect that might be related to >> 'OUTPUT'. The documentation at >> > https://docs.qgis.org/testing/en/docs/user_manual/processing/toolbox.html#output-parameter-widget >> >> didn't dissipate my doubts so I'm unsure how to proceed. >> I also tried (an alternative method) calling gdal.Warp directly (not >> >> sure if this method is preferable to using processing.run) but it >> keeps >> saying that it only takes 2 arguments. I found a few examples online >> but >> they all have more than 2 arguments. For example >> options = gdal.WarpOptions(dstSRS=target_PROJ) >> warp = gdal.Warp('/home/ubuntu/pepe.tif', >> my_input_vector_layer, options) >> warp = None >> complains about including 3 and not 2 arguments. Same if I try >> including >> all arguments at warp. >> Thanks. >> _______________________________________________ >> gdal-dev mailing list >> gdal-dev at lists.osgeo.org >> https://lists.osgeo.org/mailman/listinfo/gdal-dev From rupestre.campos at gmail.com Fri Oct 3 17:29:21 2025 From: rupestre.campos at gmail.com (ky) Date: Fri, 3 Oct 2025 21:29:21 -0300 Subject: [gdal-dev] How to specify 'OUTPUT' when warping (reproject) In-Reply-To: <514b3e50ddb0a31e7ab8e857cd212ec0@odyhpc.com> References: <514b3e50ddb0a31e7ab8e857cd212ec0@odyhpc.com> Message-ID: I believe you should check what is inside my_input_raster_layer, there should be a path to a input raster file like gdal.Warp("output.tif", "input.tif", options=options) On Fri, Oct 3, 2025 at 7:44?PM AF via gdal-dev wrote: > Hi Nyall, > It was a typo (pretty clumsy at my end) as it should have read > 'my_input_raster_layer' rather than 'my_input_vector_layer.' The > source/input layer is a raster. > Thanks. > > On 2025-10-03 21:30, Nyall Dawson wrote: > > On Sat, 4 Oct 2025, 2:00?am AF via gdal-dev, > > wrote: > > > >> Hello, > >> A PyQGIS code needs to reproject one raster layer from a CRS to a > >> new > >> one. To begin with, I've tried to use only the minimal required > >> options > >> so the snippet reads: > >> parameters = { > >> 'INPUT': my_input_vector_layer, > > > > Not sure if this is a typo (or a badly named variable, or ai slop > > code), but it looks like you're trying to warp a vector later to a > > raster layer? ? > > > > Nyall > > > >> 'SOURCE_CRS': > >> QgsCoordinateReferenceSystem(source_PROJ), > >> 'TARGET_CRS': > >> QgsCoordinateReferenceSystem(target_PROJ), > >> 'RESAMPLING': 1, > >> 'DATA_TYPE': 4, > >> 'OUTPUT': '/home/ubuntu/my_new_raster.tif' > >> } > >> terrain = processing.run('gdal:warpreproject', > >> parameters) > >> However, this doesn't do much as terrain is simply a dictionary with > >> the > >> keyword 'OUTPUT' and my_new_raster.tif is never created. Obviously I > >> > >> must be doing something wrong and suspect that might be related to > >> 'OUTPUT'. The documentation at > >> > > > https://docs.qgis.org/testing/en/docs/user_manual/processing/toolbox.html#output-parameter-widget > >> > >> didn't dissipate my doubts so I'm unsure how to proceed. > >> I also tried (an alternative method) calling gdal.Warp directly (not > >> > >> sure if this method is preferable to using processing.run) but it > >> keeps > >> saying that it only takes 2 arguments. I found a few examples online > >> but > >> they all have more than 2 arguments. For example > >> options = gdal.WarpOptions(dstSRS=target_PROJ) > >> warp = gdal.Warp('/home/ubuntu/pepe.tif', > >> my_input_vector_layer, options) > >> warp = None > >> complains about including 3 and not 2 arguments. Same if I try > >> including > >> all arguments at warp. > >> Thanks. > >> _______________________________________________ > >> gdal-dev mailing list > >> gdal-dev at lists.osgeo.org > >> https://lists.osgeo.org/mailman/listinfo/gdal-dev > _______________________________________________ > gdal-dev mailing list > gdal-dev at lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From afernandez at odyhpc.com Sat Oct 4 05:51:52 2025 From: afernandez at odyhpc.com (afernandez at odyhpc.com) Date: Sat, 04 Oct 2025 12:51:52 +0000 Subject: [gdal-dev] How to specify 'OUTPUT' when warping (reproject) In-Reply-To: References: <514b3e50ddb0a31e7ab8e857cd212ec0@odyhpc.com> Message-ID: <1e0ba637e0c6f690cd6abb8011bda000@odyhpc.com> Hi Ky, I'm beginning to understand this a bit better. My original code actually passed the raster layer per se (rather than the tiff file) so that explains the previous error. After fixing it, QGIS is now reprojecting into the new CRS but, for some reason that I still need to troubleshoot, it's cropping the western side of the layer (the central and eastern side are fine) -this is hard to explain with words but easy to see with images-. Will let you know if any other issue comes up or if it finally works as anticipated. Thanks. On 2025-10-04 00:29, ky wrote: > I believe you should check what is inside my_input_raster_layer, there > should be a path to a input raster file like > > gdal.Warp("output.tif", "input.tif", options=options) > > On Fri, Oct 3, 2025 at 7:44?PM AF via gdal-dev > wrote: > >> Hi Nyall, >> It was a typo (pretty clumsy at my end) as it should have read >> 'my_input_raster_layer' rather than 'my_input_vector_layer.' The >> source/input layer is a raster. >> Thanks. >> >> On 2025-10-03 21:30, Nyall Dawson wrote: >>> On Sat, 4 Oct 2025, 2:00?am AF via gdal-dev, >>> wrote: >>> >>>> Hello, >>>> A PyQGIS code needs to reproject one raster layer from a CRS to a >>>> new >>>> one. To begin with, I've tried to use only the minimal required >>>> options >>>> so the snippet reads: >>>> parameters = { >>>> 'INPUT': my_input_vector_layer, >>> >>> Not sure if this is a typo (or a badly named variable, or ai slop >>> code), but it looks like you're trying to warp a vector later to a >>> raster layer? ? >>> >>> Nyall >>> >>>> 'SOURCE_CRS': >>>> QgsCoordinateReferenceSystem(source_PROJ), >>>> 'TARGET_CRS': >>>> QgsCoordinateReferenceSystem(target_PROJ), >>>> 'RESAMPLING': 1, >>>> 'DATA_TYPE': 4, >>>> 'OUTPUT': '/home/ubuntu/my_new_raster.tif' >>>> } >>>> terrain = processing.run('gdal:warpreproject', >>>> parameters) >>>> However, this doesn't do much as terrain is simply a dictionary >> with >>>> the >>>> keyword 'OUTPUT' and my_new_raster.tif is never created. >> Obviously I >>>> >>>> must be doing something wrong and suspect that might be related >> to >>>> 'OUTPUT'. The documentation at >>>> >>> >> > https://docs.qgis.org/testing/en/docs/user_manual/processing/toolbox.html#output-parameter-widget >>>> >>>> didn't dissipate my doubts so I'm unsure how to proceed. >>>> I also tried (an alternative method) calling gdal.Warp directly >> (not >>>> >>>> sure if this method is preferable to using processing.run) but it >>>> keeps >>>> saying that it only takes 2 arguments. I found a few examples >> online >>>> but >>>> they all have more than 2 arguments. For example >>>> options = gdal.WarpOptions(dstSRS=target_PROJ) >>>> warp = gdal.Warp('/home/ubuntu/pepe.tif', >>>> my_input_vector_layer, options) >>>> warp = None >>>> complains about including 3 and not 2 arguments. Same if I try >>>> including >>>> all arguments at warp. >>>> Thanks. >>>> _______________________________________________ >>>> gdal-dev mailing list >>>> gdal-dev at lists.osgeo.org >>>> https://lists.osgeo.org/mailman/listinfo/gdal-dev >> _______________________________________________ >> gdal-dev mailing list >> gdal-dev at lists.osgeo.org >> https://lists.osgeo.org/mailman/listinfo/gdal-dev From lists at home.gofferje.net Sun Oct 5 00:19:44 2025 From: lists at home.gofferje.net (Stefan Gofferje) Date: Sun, 5 Oct 2025 07:19:44 +0000 Subject: [gdal-dev] AVIF write support Message-ID: <01070199b33da15b-4c5e1157-39df-437a-91fd-ea7cac4d6443-000000@eu-central-1.amazonses.com> Morning all, I would have a use case for AVIF tiles in a pipeline for creating xyz tiles from satellite imagery but it seems, as of now, gdal doesn't support AVIF writing. Are there any plans to implement that? -Stefan -- (o_ Stefan Gofferje | SCLT, MCP, CCSA //\ Reg'd Linux User #247167 | VCP #2263 V_/_ https://www.gofferje.net | https://www.saakeskus.fi From andrew at aitchison.me.uk Sun Oct 5 02:09:51 2025 From: andrew at aitchison.me.uk (Andrew C Aitchison) Date: Sun, 5 Oct 2025 10:09:51 +0100 (BST) Subject: [gdal-dev] AVIF write support In-Reply-To: <01070199b33da15b-4c5e1157-39df-437a-91fd-ea7cac4d6443-000000@eu-central-1.amazonses.com> References: <01070199b33da15b-4c5e1157-39df-437a-91fd-ea7cac4d6443-000000@eu-central-1.amazonses.com> Message-ID: <61674911-7835-65f0-bd51-572d2da0cbba@aitchison.me.uk> On Sun, 5 Oct 2025, Stefan Gofferje via gdal-dev wrote: > I would have a use case for AVIF tiles in a pipeline for creating xyz tiles > from satellite imagery but it seems, as of now, gdal doesn't support AVIF > writing. Are there any plans to implement that? ?? # gdalinfo --version GDAL 3.10.2, released 2025/02/11 # gdalinfo --formats | egrep -i "avif|support" Supported Formats: (ro:read-only, rw:read-write, +:update, v:virtual-I/O s:subdatasets) AVIF -raster- (rwvs): AV1 Image File Format (*.avif) which suggests to me that GDAL *does* support writing AVIF files. -- Andrew C. Aitchison Kendal, UK andrew at aitchison.me.uk From lnicola at dend.ro Sun Oct 5 02:50:51 2025 From: lnicola at dend.ro (=?UTF-8?Q?Lauren=C8=9Biu_Nicola?=) Date: Sun, 05 Oct 2025 12:50:51 +0300 Subject: [gdal-dev] AVIF write support In-Reply-To: <61674911-7835-65f0-bd51-572d2da0cbba@aitchison.me.uk> References: <01070199b33da15b-4c5e1157-39df-437a-91fd-ea7cac4d6443-000000@eu-central-1.amazonses.com> <61674911-7835-65f0-bd51-572d2da0cbba@aitchison.me.uk> Message-ID: Even easier: https://gdal.org/en/stable/drivers/raster/avif.html. To be fair, the Debian and Docker builds have AVIF, while Fedora doesn't. Laurentiu On Sun, Oct 5, 2025, at 12:09, Andrew C Aitchison via gdal-dev wrote: > On Sun, 5 Oct 2025, Stefan Gofferje via gdal-dev wrote: > >> I would have a use case for AVIF tiles in a pipeline for creating xyz tiles >> from satellite imagery but it seems, as of now, gdal doesn't support AVIF >> writing. Are there any plans to implement that? > > ?? > > # gdalinfo --version > GDAL 3.10.2, released 2025/02/11 > # gdalinfo --formats | egrep -i "avif|support" > Supported Formats: (ro:read-only, rw:read-write, +:update, v:virtual-I/O > s:subdatasets) > AVIF -raster- (rwvs): AV1 Image File Format (*.avif) > > which suggests to me that GDAL *does* support writing AVIF files. > > -- > Andrew C. Aitchison Kendal, UK > andrew at aitchison.me.uk > _______________________________________________ > gdal-dev mailing list > gdal-dev at lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev From andrey_vi at list.ru Mon Oct 6 00:59:27 2025 From: andrey_vi at list.ru (=?UTF-8?B?QW5kcmV5IFZJ?=) Date: Mon, 06 Oct 2025 10:59:27 +0300 Subject: [gdal-dev] =?utf-8?q?gdal=5Frasterize_ERROR_1=3A_Size_and_resolu?= =?utf-8?q?tion_are_missing?= Message-ID: <1759737567.389566129@f542.i.mail.ru> Hello. ? A few years ago a command like this: gdal_rasterize -burn 0 shape.shp hillshade.tif worked. But with the?latest GDAL it?s not: gdal_rasterize -burn 0 shape.shp hillshade.tif ERROR 1: Size and resolution are missing ? gdal_rasterize examples section ?of?GDAL documentation?includes 2 examples without size and resolution options. This means that to use the gdal_rasterize , you first need to know the raster size or resolution. This is inconvenient. ? Is this a bug or an?intentional change? -- Andrey -------------- next part -------------- An HTML attachment was scrubbed... URL: From jukka.rahkonen at maanmittauslaitos.fi Mon Oct 6 01:32:21 2025 From: jukka.rahkonen at maanmittauslaitos.fi (Rahkonen Jukka) Date: Mon, 6 Oct 2025 08:32:21 +0000 Subject: [gdal-dev] gdal_rasterize ERROR 1: Size and resolution are missing In-Reply-To: <1759737567.389566129@f542.i.mail.ru> References: <1759737567.389566129@f542.i.mail.ru> Message-ID: Hi, Are you sure that it worked like gdal_rasterize -burn 0 shape.shp hillshade.tif if hillshade.tif file did not exist? It the file exists then the command should still work. Examples 1 and 2 are burning geometries into an existing file. The example 3 writes a new tif and it includes the target size parameter "-ts 1000 1000". The error "Size and resolution are missing" is slightly wrong, it should be "Size OR resolution are missing". You do not need both, and actually user would need to be careful for making them match because they depend on each other. In conclusion, you wrote "This means that to use the gdal_rasterize, you first need to know the raster size or resolution." - If the target image exists all you need to know is the name of the file - If target image does not exists, you really need to know what resolution or image size in pixels you want. GDAL does not guess what resolution might please you. -Jukka Rahkonen- ________________________________________ L?hett?j?: gdal-dev k?ytt?j?n Andrey VI via gdal-dev puolesta L?hetetty: Maanantai 6. lokakuuta 2025 10.59 Vastaanottaja: gdal-dev at lists.osgeo.org Aihe: [gdal-dev] gdal_rasterize ERROR 1: Size and resolution are missing Hello.?A few years ago a command like this:gdal_rasterize -burn 0 shape.shp hillshade.tifworked.But with the?latest GDAL it?s not:gdal_rasterize -burn 0 shape.shp hillshade.tifERROR 1: Size and resolution are missing?gdal_rasterize examples section?of?GDAL documentation?includes 2 examples without size and resolution options.This means that to use the gdal_rasterize, you first need to know the raster size or resolution. This is inconvenient.?Is this a bug or an?intentional change?--Andrey From andreaerdna at libero.it Mon Oct 6 01:37:36 2025 From: andreaerdna at libero.it (Andrea Giudiceandrea) Date: Mon, 6 Oct 2025 10:37:36 +0200 Subject: [gdal-dev] gdal_rasterize ERROR 1: Size and resolution are missing Message-ID: <9b327da8-0f38-4afe-a1d8-8102ebbef3fa@libero.it> Il 06/10/2025 09:59, Andrey VI via gdal-dev ha scritto: > Hello. > A few years ago a command like this: > gdal_rasterize -burn 0 shape.shp hillshade.tif > worked. Hi Andrey VI, probably such command worked because the hillshade.tif files already existed. If the output raster file doesn't exist (i.e. you want to create a new raster file), then "the resolution or size must be specified using the -tr or -ts option", as explained in the gdal_rasterize docs. Andrea From andrey_vi at list.ru Mon Oct 6 02:18:11 2025 From: andrey_vi at list.ru (=?UTF-8?B?QW5kcmV5IFZJ?=) Date: Mon, 06 Oct 2025 12:18:11 +0300 Subject: [gdal-dev] =?utf-8?q?gdal=5Frasterize_ERROR_1=3A_Size_and_resolu?= =?utf-8?q?tion_are_missing?= In-Reply-To: References: <1759737567.389566129@f542.i.mail.ru> Message-ID: <1759742291.306320812@f513.i.mail.ru> Of course in my example?the file exists. Otherwise this command would not make sense without -ts or -ts?options. I figured out what the problem was. ?Existed raster was created with COG layout. And in this case gdal_rasterize? error ( Size and resolution are missing ) is very confusing. ? >Monday, October 6, 2025 11:32 AM +03:00 from Rahkonen Jukka < jukka.rahkonen at maanmittauslaitos.fi >: >? >Hi, > >Are you sure that it worked like >gdal_rasterize -burn 0 shape.shp hillshade.tif >if hillshade.tif file did not exist? It the file exists then the command should still work. Examples 1 and 2 are burning geometries into an existing file. The example 3 writes a new tif and it includes the target size parameter "-ts 1000 1000". >The error "Size and resolution are missing" is slightly wrong, it should be "Size OR resolution are missing". You do not need both, and actually user would need to be careful for making them match because they depend on each other. > >In conclusion, you wrote "This means that to use the gdal_rasterize, you first need to know the raster size or resolution." >- If the target image exists all you need to know is the name of the file >- If target image does not exists, you really need to know what resolution or image size in pixels you want. GDAL does not guess what resolution might please you. > >-Jukka Rahkonen- > >________________________________________ >L?hett?j?: gdal-dev < gdal-dev-bounces at lists.osgeo.org > k?ytt?j?n Andrey VI via gdal-dev < gdal-dev at lists.osgeo.org > puolesta >L?hetetty: Maanantai 6. lokakuuta 2025 10.59 >Vastaanottaja: gdal-dev at lists.osgeo.org < gdal-dev at lists.osgeo.org > >Aihe: [gdal-dev] gdal_rasterize ERROR 1: Size and resolution are missing > >Hello.?A few years ago a command like this:gdal_rasterize -burn 0 shape.shp hillshade.tifworked.But with the?latest GDAL it?s not:gdal_rasterize -burn 0 shape.shp hillshade.tifERROR 1: Size and resolution are missing?gdal_rasterize examples section?of?GDAL documentation?includes 2 examples without size and resolution options.This means that to use the gdal_rasterize, you first need to know the raster size or resolution. This is inconvenient.?Is this a bug or an?intentional change?--Andrey -- Andrey VI Sent from Mail -------------- next part -------------- An HTML attachment was scrubbed... URL: From jukka.rahkonen at maanmittauslaitos.fi Mon Oct 6 03:01:50 2025 From: jukka.rahkonen at maanmittauslaitos.fi (Rahkonen Jukka) Date: Mon, 6 Oct 2025 10:01:50 +0000 Subject: [gdal-dev] gdal_rasterize ERROR 1: Size and resolution are missing In-Reply-To: <1759742291.306320812@f513.i.mail.ru> References: <1759737567.389566129@f542.i.mail.ru> <1759742291.306320812@f513.i.mail.ru> Message-ID: Hi, I have seen people using GDAL commands which do not make sense, that happens for everybody, but obviously not this time. I can see two issues: - The error message does not tell what is the real issue - The real issue is documented in https://gdal.org/en/stable/drivers/raster/cog.html#update, but I am not sure how to set the IGNORE_COG_LAYOUT_BREAK open option with gdal_rasterize. -Jukka Rahkonen- ________________________________________ L?hett?j?: Andrey VI L?hetetty: Maanantai 6. lokakuuta 2025 12.18 Vastaanottaja: Rahkonen Jukka Kopio: gdal-dev at lists.osgeo.org Aihe: Re: [gdal-dev] gdal_rasterize ERROR 1: Size and resolution are missing Of course in my example?the file exists. Otherwise this command would not make sense without -ts or -ts?options.I figured out what the problem was.?Existed raster was created with COG layout. And in this case gdal_rasterize?error (Size and resolution are missing) is very confusing.?Monday, October 6, 2025 11:32 AM +03:00 from Rahkonen Jukka :?Hi,Are you sure that it worked likegdal_rasterize -burn 0 shape.shp hillshade.tifif hillshade.tif file did not exist? It the file exists then the command should still work. Examples 1 and 2 are burning geometries into an existing file. The example 3 writes a new tif and it includes the target size parameter "-ts 1000 1000".The error "Size and resolution are missing" is slightly wrong, it should be "Size OR resolution are missing". You do not need both, and actually user would need to be careful for making them match because they depend on each other.In conclusion, you wrote "This means that to use the gdal_rasterize, you first need to know the raster size or resolution."- If the target image exists all you need to know is the name of the file- If target image does not exists, you really need to know what resolution or image size in pixels you want. GDAL does not guess what resolution might please you.-Jukka Rahkonen-________________________________________L?hett?j?: gdal-dev <[/compose?To=gdal%2ddev%2dbounces at lists.osgeo.org]gdal-dev-bounces at lists.osgeo.org> k?ytt?j?n Andrey VI via gdal-dev <[/compose?To=gdal%2ddev at lists.osgeo.org]gdal-dev at lists.osgeo.org> puolestaL?hetetty: Maanantai 6. lokakuuta 2025 10.59Vastaanottaja: [/compose?To=gdal%2ddev at lists.osgeo.org]gdal-dev at lists.osgeo.org <[/compose?To=gdal%2ddev at lists.osgeo.org]gdal-dev at lists.osgeo.org>Aihe: [gdal-dev] gdal_rasterize ERROR 1: Size and resolution are missingHello.?A few years ago a command like this:gdal_rasterize -burn 0 shape.shp hillshade.tifworked.But with the?latest GDAL it?s not:gdal_rasterize -burn 0 shape.shp hillshade.tifERROR 1: Size and resolution are missing?gdal_rasterize examples section?of?GDAL documentation?includes 2 examples without size and resolution options.This means that to use the gdal_rasterize, you first need to know the raster size or resolution. This is inconvenient.?Is this a bug or an?intentional change?--Andrey--Andrey VISent from Mail From jukka.rahkonen at maanmittauslaitos.fi Mon Oct 6 03:20:37 2025 From: jukka.rahkonen at maanmittauslaitos.fi (Rahkonen Jukka) Date: Mon, 6 Oct 2025 10:20:37 +0000 Subject: [gdal-dev] gdal_rasterize ERROR 1: Size and resolution are missing In-Reply-To: References: <1759737567.389566129@f542.i.mail.ru> <1759742291.306320812@f513.i.mail.ru> Message-ID: Hi, By the way, the same error message "ERROR 1: Size and resolution are missing" can be triggered by setting an existing non-COG GeoTIFF file to be read-only in the file system. -Jukka Rahkonen- ________________________________________ L?hett?j?: gdal-dev k?ytt?j?n Rahkonen Jukka via gdal-dev puolesta L?hetetty: Maanantai 6. lokakuuta 2025 13.01 Vastaanottaja: Andrey VI Kopio: gdal-dev at lists.osgeo.org Aihe: Re: [gdal-dev] gdal_rasterize ERROR 1: Size and resolution are missing Hi, I have seen people using GDAL commands which do not make sense, that happens for everybody, but obviously not this time. I can see two issues: - The error message does not tell what is the real issue - The real issue is documented in https://gdal.org/en/stable/drivers/raster/cog.html#update, but I am not sure how to set the IGNORE_COG_LAYOUT_BREAK open option with gdal_rasterize. -Jukka Rahkonen- ________________________________________ L?hett?j?: Andrey VI L?hetetty: Maanantai 6. lokakuuta 2025 12.18 Vastaanottaja: Rahkonen Jukka Kopio: gdal-dev at lists.osgeo.org Aihe: Re: [gdal-dev] gdal_rasterize ERROR 1: Size and resolution are missing Of course in my example the file exists. Otherwise this command would not make sense without -ts or -ts options.I figured out what the problem was. Existed raster was created with COG layout. And in this case gdal_rasterize error (Size and resolution are missing) is very confusing. Monday, October 6, 2025 11:32 AM +03:00 from Rahkonen Jukka : Hi,Are you sure that it worked likegdal_rasterize -burn 0 shape.shp hillshade.tifif hillshade.tif file did not exist? It the file exists then the command should still work. Examples 1 and 2 are burning geometries into an existing file. The example 3 writes a new tif and it includes the target size parameter "-ts 1000 1000".The error "Size and resolution are missing" is slightly wrong, it should be "Size OR resolution are missing". You do not need both, and actually user would need to be careful for making them match because they depend on each other.In conclusion, you wrote "This means that to use the gdal_rasterize, you first need to know the raster size or resolution."- If the target image exists all you need to know is the name of the file- If target image does not exists, you really need to know what resolution or image size in pixels you want. GDAL does not guess what resolution might please you.-Jukka Rahkonen-________________________________________L?hett?j?: gdal-dev <[/compose?To=gdal%2ddev%2dbounces at lists.osgeo.org]gdal-dev-bounces at lists.osgeo.org> k?ytt?j?n Andrey VI via gdal-dev <[/compose?To=gdal%2ddev at lists.osgeo.org]gdal-dev at lists.osgeo.org> puolestaL?hetetty: Maanantai 6. lokakuuta 2025 10.59Vastaanottaja: [/compose?To=gdal%2ddev at lists.osgeo.org]gdal-dev at lists.osgeo.org <[/compose?To=gdal%2ddev at lists.osgeo.org]gdal-dev at lists.osgeo.org>Aihe: [gdal-dev] gdal_rasterize ERROR 1: Size and resolution are missingHello. A few years ago a command like this:gdal_rasterize -burn 0 shape.shp hillshade.tifworked.But with the latest GDAL it?s not:gdal_rasterize -burn 0 shape.shp hillshade.tifERROR 1: Size and resolution are missing gdal_rasterize examples section of GDAL documentation includes 2 examples without size and resolution options.This means that to use the gdal_rasterize, you first need to know the raster size or resolution. This is inconvenient. Is this a bug or an intentional change?--Andrey--Andrey VISent from Mail _______________________________________________ gdal-dev mailing list gdal-dev at lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev From andrew at aitchison.me.uk Tue Oct 7 04:00:35 2025 From: andrew at aitchison.me.uk (Andrew C Aitchison) Date: Tue, 7 Oct 2025 12:00:35 +0100 (BST) Subject: [gdal-dev] AVIF write support In-Reply-To: <01070199bdcdd9ed-bc837cb3-8760-4be5-9350-a6c0b2c1ce7e-000000@eu-central-1.amazonses.com> References: <01070199b33da15b-4c5e1157-39df-437a-91fd-ea7cac4d6443-000000@eu-central-1.amazonses.com> <61674911-7835-65f0-bd51-572d2da0cbba@aitchison.me.uk> <01070199bdcdd9ed-bc837cb3-8760-4be5-9350-a6c0b2c1ce7e-000000@eu-central-1.amazonses.com> Message-ID: <4b1e2a14-14cf-5edb-106a-ca4e42390740@aitchison.me.uk> On Sun, 5 Oct 2025, Stefan Gofferje via gdal-dev wrote: >>> I would have a use case for AVIF tiles in a pipeline for creating xyz >>> tiles from satellite imagery but it seems, as of now, gdal doesn't support >>> AVIF writing. Are there any plans to implement that? On 10/5/25 12:09, Andrew C Aitchison via gdal-dev wrote:>> >> ?? >> >> # gdalinfo --version >> GDAL 3.10.2, released 2025/02/11 >> # gdalinfo --formats | egrep -i "avif|support" >> Supported Formats: (ro:read-only, rw:read-write, +:update, v:virtual-I/O >> s:subdatasets) >> ? AVIF -raster- (rwvs): AV1 Image File Format (*.avif) >> >> which suggests to me that GDAL *does* support writing AVIF files. On Tue, 7 Oct 2025, Stefan Gofferje wrote privately: > ??[sgofferj at enterprise][~] > ???? gdalinfo --formats | egrep -i "avif|support" > Supported Formats: (ro:read-only, rw:read-write, +:write from scratch, > u:update, v:virtual-I/O s:subdatasets) > AVIF_HEIF -raster- (rov): AV1 Image File Format (using libheif) (*.avif) > ??[sgofferj at enterprise][~] > ???? gdalinfo --version > GDAL 3.11.4 "Eganville", released 2025/09/04 > > Ubuntu 24.04, installed via gdal install script. Hmm. I am on Ubuntu 25.10. The gdal installed from the Ubuntu .deb supports writing AVIF with the AVIF driver. ... but I have built 3.10.2 with GCC and with Clang and only the Clang version has the AVIF driver. Something odd is going on. I see the following in frmts/heif/heifdataset.cpp // If the AVIF dedicated driver is not available, register an AVIF driver, // called AVIF_HEIF, based on libheif, if it has AV1 decoding capabilities. if (heif_have_decoder_for_format(heif_compression_AV1)) { poDriver->SetMetadataItem("SUPPORTS_AVIF", "YES", "HEIF"); poDriver->SetMetadataItem("SUPPORTS_AV1", "YES", "HEIF"); } if (heif_have_encoder_for_format(heif_compression_AV1)) { poDriver->SetMetadataItem("SUPPORTS_AV1_WRITE", "YES", "HEIF"); } so you have been doubly unlucky; firstly the AVIF dedicated driver is not available and secondly your HEIF library has readonly AVIF suport. -- Andrew C. Aitchison Kendal, UK andrew at aitchison.me.uk From dklaus at carlsonsw.com Tue Oct 7 08:00:58 2025 From: dklaus at carlsonsw.com (David Klaus) Date: Tue, 7 Oct 2025 11:00:58 -0400 Subject: [gdal-dev] Properly reinitializing PROJ_LIB search paths after supplementary file download Message-ID: Hello GDAL community, The product that I work on uses the CPP GDAL library for a number of routines. Occasionally, this requires supplementary files -- Currently, we supply proj-data 1.20 -- which we download when we detect that the dataset is needed. Here's the problem, often GDALRegisterAll() is called before the proj-data 1.20 files are downloaded. From my testing, it appears that if the proj-data 1.20 files are not available when GDALRegisterAll() is called, GDAL will not use them regardless of whether or not these files are available later. Now, what I think might be a good solution is appropriately updating GDAL's state after proj-data is downloaded s.t. it is able to detect the proj-data files. But, I'm not sure how to do this. Calling the following seems to produce correct results: GDALDestroy() GDALAllRegister() However, the documentation says the following: ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////FROM DOCUMENTATION/////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// Finalize GDAL/OGR library. This function calls GDALDestroyDriverManager() and OGRCleanupAll() and finalize Thread Local Storage variables. Prior to GDAL 2.4.0, this function should normally be explicitly called by application code if GDAL is dynamically linked (but that does not hurt), since it was automatically called through the unregistration mechanisms of dynamic library loading. Since GDAL 2.4.0, this function may be called by application code, since it is no longer called automatically, on non-MSVC builds, due to ordering problems with respect to automatic destruction of global C++ objects. *Note: no GDAL/OGR code should be called after this call!* ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////FROM DOCUMENTATION/////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// So, this doesn't seem like the correct way to address this problem. How should I go about updating GDAL's state s.t. it will detect and read the downloaded supplementary proj-data files? Or, is there something more I should be doing before GDALAllRegister() is called? GDAL Version: GDAL 3.10.3, released 2025/04/01 Proj Version: 9.6.0 P.S. To get out in front of what I expect to be the first question, yes I am setting the PROJ_LIB environmental variable at runtime to the correct folder. Further, I have tried creating an empty proj-data folder available to GDAL when GDALRegisterAll() is called. This did not change GDAL's behavior. Here is the code I use for setting PROJ_LIB: SetEnvironmentVariable("PROJ_LIB", "c:/path/to/proj-data/folder"); _putenv_s("PROJ_LIB", "c:/path/to/proj-data/folder"); // (See: GETENV_NOTE) P.P.S Here is the GETENV_NOTE: // GETENV_NOTE: // _putenv_s() is used to ensure compatibility with getenv(). getenv() operates on environment variables loaded into the _environ global variable (loaded at "process startup.") // SetEnvironmentVariable() affects the environment variables set for this process but does not change _environ. _putenv_s() updates _environ and ensures compatibility w/ // genenv(). P.P.P.S I do know that there is a more current proj-data dataset available. We may update to this dataset in the future. But unless that will fix the current issue, I'd rather not do that now. -- David Klaus Carlson Software Disclaimer The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful. -------------- next part -------------- An HTML attachment was scrubbed... URL: From public at postholer.com Tue Oct 7 09:30:27 2025 From: public at postholer.com (Scott) Date: Tue, 7 Oct 2025 09:30:27 -0700 Subject: [gdal-dev] gdal raster calc - a modulo operator (%)? Message-ID: Greetings! I've been unable to find a modulo operator in CLI calc. It's not in the muparser functions or appear to be in the forth-coming 3.12 C pixel functions. This has been working with muparser, but it's messy every time you need a simple modulo. pixel is real or int, div is int: rint(pixel) - (rint(rint(pixel) / div) * div) Am I missing something? If not, consider this a feature request! :) Thanks! Scott -- www.postholer.com From lnicola at dend.ro Tue Oct 7 09:34:25 2025 From: lnicola at dend.ro (=?UTF-8?Q?Lauren=C8=9Biu_Nicola?=) Date: Tue, 07 Oct 2025 19:34:25 +0300 Subject: [gdal-dev] gdal raster calc - a modulo operator (%)? In-Reply-To: References: Message-ID: <93e4d104-8cae-445d-8bc7-9e46bf758427@betaapp.fastmail.com> Hi, GDAL also supports https://github.com/ArashPartow/exprtk (which is available in the -full Docker image), which has a % operator, but I've never used it. Laurentiu On Tue, Oct 7, 2025, at 19:30, Scott via gdal-dev wrote: > Greetings! > > I've been unable to find a modulo operator in CLI calc. It's not in the > muparser functions or appear to be in the forth-coming 3.12 C pixel > functions. > > This has been working with muparser, but it's messy every time you need > a simple modulo. pixel is real or int, div is int: > > rint(pixel) - (rint(rint(pixel) / div) * div) > > Am I missing something? If not, consider this a feature request! :) > > Thanks! > Scott > > -- > www.postholer.com > > _______________________________________________ > gdal-dev mailing list > gdal-dev at lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev From dbaston at gmail.com Tue Oct 7 09:52:55 2025 From: dbaston at gmail.com (Daniel Baston) Date: Tue, 7 Oct 2025 12:52:55 -0400 Subject: [gdal-dev] gdal raster calc - a modulo operator (%)? In-Reply-To: References: Message-ID: It's fairly straightforward to define functions in C++ and register them with muparser. You can see how we do this with "isnan" and "isnodata" here: https://github.com/OSGeo/gdal/blob/ca7a2d38893a8d6cdc03653b47712f4442d99d65/frmts/vrt/vrtexpression_muparser.cpp#L27 I don't think we want too many custom functions, but adding "mod" seems like a good idea. Dan On Tue, Oct 7, 2025 at 12:30?PM Scott via gdal-dev wrote: > Greetings! > > I've been unable to find a modulo operator in CLI calc. It's not in the > muparser functions or appear to be in the forth-coming 3.12 C pixel > functions. > > This has been working with muparser, but it's messy every time you need > a simple modulo. pixel is real or int, div is int: > > rint(pixel) - (rint(rint(pixel) / div) * div) > > Am I missing something? If not, consider this a feature request! :) > > Thanks! > Scott > > -- > www.postholer.com > > _______________________________________________ > gdal-dev mailing list > gdal-dev at lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jmckenna at gatewaygeomatics.com Tue Oct 7 10:07:36 2025 From: jmckenna at gatewaygeomatics.com (Jeff McKenna) Date: Tue, 7 Oct 2025 14:07:36 -0300 Subject: [gdal-dev] Properly reinitializing PROJ_LIB search paths after supplementary file download In-Reply-To: References: Message-ID: <22204a5c-409d-4f9b-8f04-e1728fd78c87@gatewaygeomatics.com> Hi David, be careful that the former PROJ_LIB environment variable was changed to PROJ_DATA since the PROJ 9.1.0 release (the former may work for you now, but, you should change your packaging process to the correct name, depending on what PROJ version you have). Also, you can follow the PROJ_DATA releases at https://github.com/OSGeo/PROJ-data/releases Thanks, -jeff -- Jeff McKenna GatewayGeo: Developers of MS4W, & offering MapServer Consulting/Dev co-founder of FOSS4G http://gatewaygeo.com/ On 2025-10-07 12:00 p.m., David Klaus via gdal-dev wrote: > Hello GDAL community, > > The product that I work on uses the CPP GDAL library for a number of > routines. Occasionally, this requires supplementary files -- Currently, > we supply proj-data 1.20 tag/1.20.0> -- which we download when we detect that the dataset is > needed. Here's the problem, often GDALRegisterAll() is called before the > proj-data 1.20 files are downloaded. From my testing, it appears that if > the proj-data 1.20 files are not available when GDALRegisterAll() is > called, GDAL will not use them regardless of whether or not these files > are available later. > > Now, what I think might be a good solution is appropriately updating > GDAL's state after proj-data is downloaded s.t. it is able to detect the > proj-data files. But, I'm not sure how to do this. Calling the following > seems to produce correct results: > > GDALDestroy() > GDALAllRegister() > > However, the documentation says the following: > > ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////FROM DOCUMENTATION/////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// > Finalize GDAL/OGR library. > > This function calls GDALDestroyDriverManager() and OGRCleanupAll() and > finalize Thread Local Storage variables. > > Prior to GDAL 2.4.0, this function should normally be explicitly called > by application code if GDAL is dynamically linked (but that does not > hurt), since it was automatically called through the unregistration > mechanisms of dynamic library loading. > > Since GDAL 2.4.0, this function may be called by application code, since > it is no longer called automatically, on non-MSVC builds, due to > ordering problems with respect to automatic destruction of global C++ > objects. > > *Note: no GDAL/OGR code should be called after this call!* > ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////FROM DOCUMENTATION/////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// > > So, this doesn't seem like the correct way to address this problem. How > should I go about updating GDAL's state s.t. it will detect and read the > downloaded supplementary proj-data files? Or, is there something more I > should be doing before GDALAllRegister() is called? > > GDAL Version: GDAL 3.10.3, released 2025/04/01 > Proj Version: 9.6.0 > > P.S. To get out in front of what I expect to be the first question, yes > I am setting the?PROJ_LIB environmental variable at runtime to the > correct folder. Further, I have tried creating an empty proj-data folder > available to GDAL when GDALRegisterAll() is called. This did not change > GDAL's behavior. Here is the code I use for setting PROJ_LIB: > > SetEnvironmentVariable("PROJ_LIB", "c:/path/to/proj-data/folder"); > _putenv_s("PROJ_LIB", "c:/path/to/proj-data/folder"); // (See: GETENV_NOTE) > > P.P.S Here is the GETENV_NOTE: > // GETENV_NOTE: > // _putenv_s() is used to ensure compatibility with getenv(). getenv() > operates on environment variables loaded into? the _environ global > variable (loaded at "process startup.") > // SetEnvironmentVariable() affects the environment variables set for > this process but does not change _environ. _putenv_s() updates _environ > and ensures compatibility w/ > //?genenv(). > > P.P.P.S I do know that there is a more current proj-data dataset > available. We may update to this dataset in the future. But unless that > will fix the current issue, I'd rather not do that now. > > > -- > David Klaus > Carlson Software > > > *Disclaimer* > > The information contained in this communication from the sender is > confidential. It is intended solely for use by the recipient and others > authorized to receive it. If you are not the recipient, you are hereby > notified that any disclosure, copying, distribution or taking action in > relation of the contents of this information is strictly prohibited and > may be unlawful. > > > _______________________________________________ > gdal-dev mailing list > gdal-dev at lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev From dklaus at carlsonsw.com Tue Oct 7 10:10:28 2025 From: dklaus at carlsonsw.com (David Klaus) Date: Tue, 7 Oct 2025 13:10:28 -0400 Subject: [gdal-dev] Properly reinitializing PROJ_LIB search paths after supplementary file download In-Reply-To: <22204a5c-409d-4f9b-8f04-e1728fd78c87@gatewaygeomatics.com> References: <22204a5c-409d-4f9b-8f04-e1728fd78c87@gatewaygeomatics.com> Message-ID: Jeff, Thank you for that information. I wasn't aware that had changed. I will update our code, On Tue, Oct 7, 2025 at 1:07?PM Jeff McKenna via gdal-dev < gdal-dev at lists.osgeo.org> wrote: > Hi David, be careful that the former PROJ_LIB environment variable was > changed to PROJ_DATA since the PROJ 9.1.0 release (the former may work > for you now, but, you should change your packaging process to the > correct name, depending on what PROJ version you have). > > Also, you can follow the PROJ_DATA releases at > > https://github.com/OSGeo/PROJ-data/releases > > Thanks, > > -jeff > > > > > -- > Jeff McKenna > GatewayGeo: Developers of MS4W, & offering MapServer Consulting/Dev > co-founder of FOSS4G > > http://gatewaygeo.com > > > > > On 2025-10-07 12:00 p.m., David Klaus via gdal-dev wrote: > > Hello GDAL community, > > > > The product that I work on uses the CPP GDAL library for a number of > > routines. Occasionally, this requires supplementary files -- Currently, > > we supply proj-data 1.20 < > https://github.com/OSGeo/PROJ-data/releases/ > > tag/1.20.0> -- which we download when we detect that the dataset is > > needed. Here's the problem, often GDALRegisterAll() is called before the > > proj-data 1.20 files are downloaded. From my testing, it appears that if > > the proj-data 1.20 files are not available when GDALRegisterAll() is > > called, GDAL will not use them regardless of whether or not these files > > are available later. > > > > Now, what I think might be a good solution is appropriately updating > > GDAL's state after proj-data is downloaded s.t. it is able to detect the > > proj-data files. But, I'm not sure how to do this. Calling the following > > seems to produce correct results: > > > > GDALDestroy() > > GDALAllRegister() > > > > However, the documentation says the following: > > > > > ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////FROM > DOCUMENTATION/////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// > > Finalize GDAL/OGR library. > > > > This function calls GDALDestroyDriverManager() and OGRCleanupAll() and > > finalize Thread Local Storage variables. > > > > Prior to GDAL 2.4.0, this function should normally be explicitly called > > by application code if GDAL is dynamically linked (but that does not > > hurt), since it was automatically called through the unregistration > > mechanisms of dynamic library loading. > > > > Since GDAL 2.4.0, this function may be called by application code, since > > it is no longer called automatically, on non-MSVC builds, due to > > ordering problems with respect to automatic destruction of global C++ > > objects. > > > > *Note: no GDAL/OGR code should be called after this call!* > > > ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////FROM > DOCUMENTATION/////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// > > > > So, this doesn't seem like the correct way to address this problem. How > > should I go about updating GDAL's state s.t. it will detect and read the > > downloaded supplementary proj-data files? Or, is there something more I > > should be doing before GDALAllRegister() is called? > > > > GDAL Version: GDAL 3.10.3, released 2025/04/01 > > Proj Version: 9.6.0 > > > > P.S. To get out in front of what I expect to be the first question, yes > > I am setting the PROJ_LIB environmental variable at runtime to the > > correct folder. Further, I have tried creating an empty proj-data folder > > available to GDAL when GDALRegisterAll() is called. This did not change > > GDAL's behavior. Here is the code I use for setting PROJ_LIB: > > > > SetEnvironmentVariable("PROJ_LIB", "c:/path/to/proj-data/folder"); > > _putenv_s("PROJ_LIB", "c:/path/to/proj-data/folder"); // (See: > GETENV_NOTE) > > > > P.P.S Here is the GETENV_NOTE: > > // GETENV_NOTE: > > // _putenv_s() is used to ensure compatibility with getenv(). getenv() > > operates on environment variables loaded into the _environ global > > variable (loaded at "process startup.") > > // SetEnvironmentVariable() affects the environment variables set for > > this process but does not change _environ. _putenv_s() updates _environ > > and ensures compatibility w/ > > // genenv(). > > > > P.P.P.S I do know that there is a more current proj-data dataset > > available. We may update to this dataset in the future. But unless that > > will fix the current issue, I'd rather not do that now. > > > > > > -- > > David Klaus > > Carlson Software > > > > > > *Disclaimer* > > > > The information contained in this communication from the sender is > > confidential. It is intended solely for use by the recipient and others > > authorized to receive it. If you are not the recipient, you are hereby > > notified that any disclosure, copying, distribution or taking action in > > relation of the contents of this information is strictly prohibited and > > may be unlawful. > > > > > > _______________________________________________ > > gdal-dev mailing list > > gdal-dev at lists.osgeo.org > > > https://lists.osgeo.org/mailman/listinfo/gdal-dev > > > _______________________________________________ > gdal-dev mailing list > gdal-dev at lists.osgeo.org > > https://lists.osgeo.org/mailman/listinfo/gdal-dev > > -- David Klaus Carlson Software Disclaimer The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful. -------------- next part -------------- An HTML attachment was scrubbed... URL: From j1 at jimenezshaw.com Tue Oct 7 10:14:57 2025 From: j1 at jimenezshaw.com (Javier Jimenez Shaw) Date: Tue, 7 Oct 2025 19:14:57 +0200 Subject: [gdal-dev] gdal raster calc - a modulo operator (%)? In-Reply-To: References: Message-ID: Just for the record, "mod" of a floating point number should be properly documented. On Tue, 7 Oct 2025, 18:53 Daniel Baston via gdal-dev, < gdal-dev at lists.osgeo.org> wrote: > It's fairly straightforward to define functions in C++ and register them > with muparser. You can see how we do this with "isnan" and "isnodata" here: > > > https://github.com/OSGeo/gdal/blob/ca7a2d38893a8d6cdc03653b47712f4442d99d65/frmts/vrt/vrtexpression_muparser.cpp#L27 > > I don't think we want too many custom functions, but adding "mod" seems > like a good idea. > > Dan > > On Tue, Oct 7, 2025 at 12:30?PM Scott via gdal-dev < > gdal-dev at lists.osgeo.org> wrote: > >> Greetings! >> >> I've been unable to find a modulo operator in CLI calc. It's not in the >> muparser functions or appear to be in the forth-coming 3.12 C pixel >> functions. >> >> This has been working with muparser, but it's messy every time you need >> a simple modulo. pixel is real or int, div is int: >> >> rint(pixel) - (rint(rint(pixel) / div) * div) >> >> Am I missing something? If not, consider this a feature request! :) >> >> Thanks! >> Scott >> >> -- >> www.postholer.com >> >> _______________________________________________ >> gdal-dev mailing list >> gdal-dev at lists.osgeo.org >> https://lists.osgeo.org/mailman/listinfo/gdal-dev >> > _______________________________________________ > gdal-dev mailing list > gdal-dev at lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From public at postholer.com Tue Oct 7 14:24:26 2025 From: public at postholer.com (Scott) Date: Tue, 7 Oct 2025 14:24:26 -0700 Subject: [gdal-dev] gdal raster calc - a modulo operator (%)? In-Reply-To: References: Message-ID: <9d29bfaf-6f3e-46c0-8e95-3897179e3ccf@postholer.com> Thanks for the idea Dan. Building on that, I added it upstream to the MuParser source. Now I have it in libmuparser as well: ./muparser-2.3.5/include/muParserTemplateMagic.h static T Mod(T pixel, T div) { return lround(pixel) - ((lround(pixel) / lround(div)) * lround(div)); } ./muparser-2.3.5/src/muParser.cpp DefineFun(_T("mod"), MathImpl::Mod); After rebuild, leads to: gdal raster calc \ --input "p=input.tif" \ --calc="mod(p[1], 50) == 0 ? p[1] : 255" \ --ot=Byte \ --output="output.tif" --co COMPRESS=DEFLATE --overwrite --progress Ideally, though, having a mod function in GDAL itself would be awesome. Scott On 10/7/25 09:52, Daniel Baston wrote: > It's fairly straightforward to define functions in C++ and register them > with muparser. You can see how we do this with "isnan" and "isnodata" here: > > https://github.com/OSGeo/gdal/blob/ > ca7a2d38893a8d6cdc03653b47712f4442d99d65/frmts/vrt/ > vrtexpression_muparser.cpp#L27 ca7a2d38893a8d6cdc03653b47712f4442d99d65/frmts/vrt/ > vrtexpression_muparser.cpp#L27> > > I don't think we want too many custom functions, but adding "mod" seems > like a good idea. > > Dan > > On Tue, Oct 7, 2025 at 12:30?PM Scott via gdal-dev dev at lists.osgeo.org > wrote: > > Greetings! > > I've been unable to find a modulo operator in CLI calc. It's not in the > muparser functions or appear to be in the forth-coming 3.12 C pixel > functions. > > This has been working with muparser, but it's messy every time you need > a simple modulo. pixel is real or int, div is int: > > rint(pixel) - (rint(rint(pixel) / div) * div) > > Am I missing something? If not, consider this a feature request! :) > > Thanks! > Scott > > -- > www.postholer.com > > _______________________________________________ > gdal-dev mailing list > gdal-dev at lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev lists.osgeo.org/mailman/listinfo/gdal-dev> > From dklaus at carlsonsw.com Tue Oct 7 14:26:33 2025 From: dklaus at carlsonsw.com (David Klaus) Date: Tue, 7 Oct 2025 17:26:33 -0400 Subject: [gdal-dev] Properly reinitializing PROJ_LIB search paths after supplementary file download In-Reply-To: References: <22204a5c-409d-4f9b-8f04-e1728fd78c87@gatewaygeomatics.com> Message-ID: All, I just wanted to give a quick update. I have switched to using PROJ_DATA. Though I did not expect this to fix my original issue, I can confirm that it has not. Is there perhaps a cache that I need to clear or something like that? Or, is there some component of GDAL/Proj that I need to reset? On Tue, Oct 7, 2025 at 1:10?PM David Klaus wrote: > Jeff, > > Thank you for that information. I wasn't aware that had changed. I will > update our code, > > On Tue, Oct 7, 2025 at 1:07?PM Jeff McKenna via gdal-dev < > gdal-dev at lists.osgeo.org> wrote: > >> Hi David, be careful that the former PROJ_LIB environment variable was >> changed to PROJ_DATA since the PROJ 9.1.0 release (the former may work >> for you now, but, you should change your packaging process to the >> correct name, depending on what PROJ version you have). >> >> Also, you can follow the PROJ_DATA releases at >> >> https://github.com/OSGeo/PROJ-data/releases >> >> Thanks, >> >> -jeff >> >> >> >> >> -- >> Jeff McKenna >> GatewayGeo: Developers of MS4W, & offering MapServer Consulting/Dev >> co-founder of FOSS4G >> >> http://gatewaygeo.com >> >> >> >> >> On 2025-10-07 12:00 p.m., David Klaus via gdal-dev wrote: >> > Hello GDAL community, >> > >> > The product that I work on uses the CPP GDAL library for a number of >> > routines. Occasionally, this requires supplementary files -- Currently, >> > we supply proj-data 1.20 < >> https://github.com/OSGeo/PROJ-data/releases/ >> > tag/1.20.0> -- which we download when we detect that the dataset is >> > needed. Here's the problem, often GDALRegisterAll() is called before >> the >> > proj-data 1.20 files are downloaded. From my testing, it appears that >> if >> > the proj-data 1.20 files are not available when GDALRegisterAll() is >> > called, GDAL will not use them regardless of whether or not these files >> > are available later. >> > >> > Now, what I think might be a good solution is appropriately updating >> > GDAL's state after proj-data is downloaded s.t. it is able to detect >> the >> > proj-data files. But, I'm not sure how to do this. Calling the >> following >> > seems to produce correct results: >> > >> > GDALDestroy() >> > GDALAllRegister() >> > >> > However, the documentation says the following: >> > >> > >> ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////FROM >> DOCUMENTATION/////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// >> > Finalize GDAL/OGR library. >> > >> > This function calls GDALDestroyDriverManager() and OGRCleanupAll() and >> > finalize Thread Local Storage variables. >> > >> > Prior to GDAL 2.4.0, this function should normally be explicitly called >> > by application code if GDAL is dynamically linked (but that does not >> > hurt), since it was automatically called through the unregistration >> > mechanisms of dynamic library loading. >> > >> > Since GDAL 2.4.0, this function may be called by application code, >> since >> > it is no longer called automatically, on non-MSVC builds, due to >> > ordering problems with respect to automatic destruction of global C++ >> > objects. >> > >> > *Note: no GDAL/OGR code should be called after this call!* >> > >> ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////FROM >> DOCUMENTATION/////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// >> > >> > So, this doesn't seem like the correct way to address this problem. How >> > should I go about updating GDAL's state s.t. it will detect and read >> the >> > downloaded supplementary proj-data files? Or, is there something more I >> > should be doing before GDALAllRegister() is called? >> > >> > GDAL Version: GDAL 3.10.3, released 2025/04/01 >> > Proj Version: 9.6.0 >> > >> > P.S. To get out in front of what I expect to be the first question, yes >> > I am setting the PROJ_LIB environmental variable at runtime to the >> > correct folder. Further, I have tried creating an empty proj-data >> folder >> > available to GDAL when GDALRegisterAll() is called. This did not change >> > GDAL's behavior. Here is the code I use for setting PROJ_LIB: >> > >> > SetEnvironmentVariable("PROJ_LIB", "c:/path/to/proj-data/folder"); >> > _putenv_s("PROJ_LIB", "c:/path/to/proj-data/folder"); // (See: >> GETENV_NOTE) >> > >> > P.P.S Here is the GETENV_NOTE: >> > // GETENV_NOTE: >> > // _putenv_s() is used to ensure compatibility with getenv(). getenv() >> > operates on environment variables loaded into the _environ global >> > variable (loaded at "process startup.") >> > // SetEnvironmentVariable() affects the environment variables set for >> > this process but does not change _environ. _putenv_s() updates _environ >> > and ensures compatibility w/ >> > // genenv(). >> > >> > P.P.P.S I do know that there is a more current proj-data dataset >> > available. We may update to this dataset in the future. But unless that >> > will fix the current issue, I'd rather not do that now. >> > >> > >> > -- >> > David Klaus >> > Carlson Software >> > >> > >> > *Disclaimer* >> > >> > The information contained in this communication from the sender is >> > confidential. It is intended solely for use by the recipient and others >> > authorized to receive it. If you are not the recipient, you are hereby >> > notified that any disclosure, copying, distribution or taking action in >> > relation of the contents of this information is strictly prohibited and >> > may be unlawful. >> > >> > >> > _______________________________________________ >> > gdal-dev mailing list >> > gdal-dev at lists.osgeo.org >> > >> https://lists.osgeo.org/mailman/listinfo/gdal-dev >> >> >> _______________________________________________ >> gdal-dev mailing list >> gdal-dev at lists.osgeo.org >> >> https://lists.osgeo.org/mailman/listinfo/gdal-dev >> >> > > -- > David Klaus > Carlson Software > -- David Klaus Carlson Software Disclaimer The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bdezonia at gmail.com Tue Oct 7 14:30:31 2025 From: bdezonia at gmail.com (Barry DeZonia) Date: Tue, 7 Oct 2025 16:30:31 -0500 Subject: [gdal-dev] gdal raster calc - a modulo operator (%)? In-Reply-To: <9d29bfaf-6f3e-46c0-8e95-3897179e3ccf@postholer.com> References: <9d29bfaf-6f3e-46c0-8e95-3897179e3ccf@postholer.com> Message-ID: This might be ill behaved for floating point. Rounding errors and correct behavior for Infs and NaNs etc can spring surprises on you. Java has a floating point modulus operator. I am betting the Double or Float class java api javadoc might discuss implementation notes. On Tue, Oct 7, 2025 at 4:24?PM Scott via gdal-dev wrote: > > Thanks for the idea Dan. Building on that, I added it upstream to the > MuParser source. Now I have it in libmuparser as well: > > ./muparser-2.3.5/include/muParserTemplateMagic.h > static T Mod(T pixel, T div) { return lround(pixel) - > ((lround(pixel) / lround(div)) * lround(div)); } > > ./muparser-2.3.5/src/muParser.cpp > DefineFun(_T("mod"), MathImpl::Mod); > > After rebuild, leads to: > > gdal raster calc \ > --input "p=input.tif" \ > --calc="mod(p[1], 50) == 0 ? p[1] : 255" \ > --ot=Byte \ > --output="output.tif" --co COMPRESS=DEFLATE --overwrite --progress > > Ideally, though, having a mod function in GDAL itself would be awesome. > > Scott > > > On 10/7/25 09:52, Daniel Baston wrote: > > It's fairly straightforward to define functions in C++ and register them > > with muparser. You can see how we do this with "isnan" and "isnodata" here: > > > > https://github.com/OSGeo/gdal/blob/ > > ca7a2d38893a8d6cdc03653b47712f4442d99d65/frmts/vrt/ > > vrtexpression_muparser.cpp#L27 > ca7a2d38893a8d6cdc03653b47712f4442d99d65/frmts/vrt/ > > vrtexpression_muparser.cpp#L27> > > > > I don't think we want too many custom functions, but adding "mod" seems > > like a good idea. > > > > Dan > > > > On Tue, Oct 7, 2025 at 12:30?PM Scott via gdal-dev > dev at lists.osgeo.org > wrote: > > > > Greetings! > > > > I've been unable to find a modulo operator in CLI calc. It's not in the > > muparser functions or appear to be in the forth-coming 3.12 C pixel > > functions. > > > > This has been working with muparser, but it's messy every time you need > > a simple modulo. pixel is real or int, div is int: > > > > rint(pixel) - (rint(rint(pixel) / div) * div) > > > > Am I missing something? If not, consider this a feature request! :) > > > > Thanks! > > Scott > > > > -- > > www.postholer.com > > > > _______________________________________________ > > gdal-dev mailing list > > gdal-dev at lists.osgeo.org > > https://lists.osgeo.org/mailman/listinfo/gdal-dev > lists.osgeo.org/mailman/listinfo/gdal-dev> > > > > _______________________________________________ > gdal-dev mailing list > gdal-dev at lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev From even.rouault at spatialys.com Tue Oct 7 14:43:09 2025 From: even.rouault at spatialys.com (Even Rouault) Date: Tue, 7 Oct 2025 23:43:09 +0200 Subject: [gdal-dev] Properly reinitializing PROJ_LIB search paths after supplementary file download In-Reply-To: References: Message-ID: <1e99883e-09a2-4420-acc5-ee021acc20b1@spatialys.com> David, does calling OSRCleanup() help ? Even Le 07/10/2025 ? 17:00, David Klaus via gdal-dev a ?crit?: > Hello GDAL community, > > The product that I work on uses the CPP GDAL library for a number of > routines. Occasionally, this requires supplementary files -- > Currently, we supply proj-data 1.20 > -- which we > download when we detect that the dataset is needed. Here's the > problem, often GDALRegisterAll() is called before the proj-data 1.20 > files are downloaded. From my testing, it appears that if the > proj-data 1.20 files are not available when GDALRegisterAll() is > called, GDAL will not use them regardless of whether or not these > files are available later. > > Now, what I think might be a good solution is appropriately updating > GDAL's state after proj-data is downloaded s.t. it is able to detect > the proj-data files. But, I'm not sure how to do this. Calling the > following seems to produce correct results: > > GDALDestroy() > GDALAllRegister() > > However, the documentation says the following: > > ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////FROM > DOCUMENTATION/////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// > Finalize GDAL/OGR library. > > This function calls GDALDestroyDriverManager() and OGRCleanupAll() and > finalize Thread Local Storage variables. > > Prior to GDAL 2.4.0, this function should normally be explicitly > called by application code if GDAL is dynamically linked (but that > does not hurt), since it was automatically called through the > unregistration mechanisms of dynamic library loading. > > Since GDAL 2.4.0, this function may be called by application code, > since it is no longer called automatically, on non-MSVC builds, due to > ordering problems with respect to automatic destruction of global C++ > objects. > > *Note: no GDAL/OGR code should be called after this call!* > ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////FROM > DOCUMENTATION/////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// > > So, this doesn't seem like the correct way to address this problem. > How should I go about updating GDAL's state s.t. it will detect and > read the downloaded supplementary proj-data files? Or, is there > something more I should be doing before GDALAllRegister() is called? > > GDAL Version: GDAL 3.10.3, released 2025/04/01 > Proj Version: 9.6.0 > > P.S. To get out in front of what I expect to be the first question, > yes I am setting the?PROJ_LIB environmental variable at runtime to the > correct folder. Further, I have tried creating an empty proj-data > folder available to GDAL when GDALRegisterAll() is called. This did > not change GDAL's behavior. Here is the code I use for setting PROJ_LIB: > > SetEnvironmentVariable("PROJ_LIB", "c:/path/to/proj-data/folder"); > _putenv_s("PROJ_LIB", "c:/path/to/proj-data/folder"); // (See: > GETENV_NOTE) > > P.P.S Here is the GETENV_NOTE: > // GETENV_NOTE: > // _putenv_s() is used to ensure compatibility with getenv(). getenv() > operates on environment variables loaded into? the _environ global > variable (loaded at "process startup.") > // SetEnvironmentVariable() affects the environment variables set for > this process but does not change _environ. _putenv_s() updates > _environ and ensures compatibility w/ > //?genenv(). > > P.P.P.S I do know that there is a more current proj-data dataset > available. We may update to this dataset in the future. But unless > that will fix the current issue, I'd rather not do that now. > > > -- > David Klaus > Carlson Software > > > *Disclaimer* > > The information contained in this communication from the sender is > confidential. It is intended solely for use by the recipient and > others authorized to receive it. If you are not the recipient, you are > hereby notified that any disclosure, copying, distribution or taking > action in relation of the contents of this information is strictly > prohibited and may be unlawful. > > > _______________________________________________ > gdal-dev mailing list > gdal-dev at lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev -- http://www.spatialys.com My software is free, but my time generally not. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dklaus at carlsonsw.com Tue Oct 7 14:47:06 2025 From: dklaus at carlsonsw.com (David Klaus) Date: Tue, 7 Oct 2025 17:47:06 -0400 Subject: [gdal-dev] Properly reinitializing PROJ_LIB search paths after supplementary file download In-Reply-To: <1e99883e-09a2-4420-acc5-ee021acc20b1@spatialys.com> References: <1e99883e-09a2-4420-acc5-ee021acc20b1@spatialys.com> Message-ID: Even, I will check on this in a moment. If there are OGR/Proj/GDAL entities on the heap/stack -- such as OGRCoordinateTransformation -- will calling OSRCleanup cause issues? On Tue, Oct 7, 2025 at 5:43?PM Even Rouault wrote: > David, > > does calling OSRCleanup() help ? > > Even > Le 07/10/2025 ? 17:00, David Klaus via gdal-dev a ?crit : > > Hello GDAL community, > > The product that I work on uses the CPP GDAL library for a number of > routines. Occasionally, this requires supplementary files -- Currently, we > supply proj-data 1.20 > > -- which we download when we detect that the dataset is needed. Here's the > problem, often GDALRegisterAll() is called before the proj-data 1.20 files > are downloaded. From my testing, it appears that if the proj-data 1.20 > files are not available when GDALRegisterAll() is called, GDAL will not use > them regardless of whether or not these files are available later. > > Now, what I think might be a good solution is appropriately updating > GDAL's state after proj-data is downloaded s.t. it is able to detect the > proj-data files. But, I'm not sure how to do this. Calling the following > seems to produce correct results: > > GDALDestroy() > GDALAllRegister() > > However, the documentation says the following: > > ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////FROM > DOCUMENTATION/////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// > Finalize GDAL/OGR library. > > This function calls GDALDestroyDriverManager() and OGRCleanupAll() and > finalize Thread Local Storage variables. > > Prior to GDAL 2.4.0, this function should normally be explicitly called by > application code if GDAL is dynamically linked (but that does not hurt), > since it was automatically called through the unregistration mechanisms of > dynamic library loading. > > Since GDAL 2.4.0, this function may be called by application code, since > it is no longer called automatically, on non-MSVC builds, due to ordering > problems with respect to automatic destruction of global C++ objects. > > *Note: no GDAL/OGR code should be called after this call!* > ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////FROM > DOCUMENTATION/////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// > > So, this doesn't seem like the correct way to address this problem. How > should I go about updating GDAL's state s.t. it will detect and read the > downloaded supplementary proj-data files? Or, is there something more I > should be doing before GDALAllRegister() is called? > > GDAL Version: GDAL 3.10.3, released 2025/04/01 > Proj Version: 9.6.0 > > P.S. To get out in front of what I expect to be the first question, yes I > am setting the PROJ_LIB environmental variable at runtime to the correct > folder. Further, I have tried creating an empty proj-data folder available > to GDAL when GDALRegisterAll() is called. This did not change GDAL's > behavior. Here is the code I use for setting PROJ_LIB: > > SetEnvironmentVariable("PROJ_LIB", "c:/path/to/proj-data/folder"); > _putenv_s("PROJ_LIB", "c:/path/to/proj-data/folder"); // (See: GETENV_NOTE) > > P.P.S Here is the GETENV_NOTE: > // GETENV_NOTE: > // _putenv_s() is used to ensure compatibility with getenv(). getenv() > operates on environment variables loaded into the _environ global variable > (loaded at "process startup.") > // SetEnvironmentVariable() affects the environment variables set for this > process but does not change _environ. _putenv_s() updates _environ and > ensures compatibility w/ > // genenv(). > > P.P.P.S I do know that there is a more current proj-data dataset > available. We may update to this dataset in the future. But unless that > will fix the current issue, I'd rather not do that now. > > > -- > David Klaus > Carlson Software > > > *Disclaimer* > > The information contained in this communication from the sender is > confidential. It is intended solely for use by the recipient and others > authorized to receive it. If you are not the recipient, you are hereby > notified that any disclosure, copying, distribution or taking action in > relation of the contents of this information is strictly prohibited and may > be unlawful. > > _______________________________________________ > gdal-dev mailing listgdal-dev at lists.osgeo.orghttps://lists.osgeo.org/mailman/listinfo/gdal-dev > > -- http://www.spatialys.com > My software is free, but my time generally not. > > -- David Klaus Carlson Software Disclaimer The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful. -------------- next part -------------- An HTML attachment was scrubbed... URL: From public at postholer.com Tue Oct 7 14:47:38 2025 From: public at postholer.com (Scott) Date: Tue, 7 Oct 2025 14:47:38 -0700 Subject: [gdal-dev] gdal raster calc - a modulo operator (%)? In-Reply-To: References: <9d29bfaf-6f3e-46c0-8e95-3897179e3ccf@postholer.com> Message-ID: <6e9ad71f-c48f-42e9-9140-949153837c0a@postholer.com> Hey Barry, I can confirm there are a whole slew of issues I haven't begun to think through! ;) In context of a raster pixel, integers shouldn't be an issue. Can a raster pixel even have a NaN value? A nodata IS a value. I don't even know. Is this a chunk of code I'd encourage everyone to use? Definitely not. Scott On 10/7/25 14:30, Barry DeZonia wrote: > This might be ill behaved for floating point. Rounding errors and > correct behavior for Infs and NaNs etc can spring surprises on you. > Java has a floating point modulus operator. I am betting the Double or > Float class java api javadoc might discuss implementation notes. > > On Tue, Oct 7, 2025 at 4:24?PM Scott via gdal-dev > wrote: >> >> Thanks for the idea Dan. Building on that, I added it upstream to the >> MuParser source. Now I have it in libmuparser as well: >> >> ./muparser-2.3.5/include/muParserTemplateMagic.h >> static T Mod(T pixel, T div) { return lround(pixel) - >> ((lround(pixel) / lround(div)) * lround(div)); } >> >> ./muparser-2.3.5/src/muParser.cpp >> DefineFun(_T("mod"), MathImpl::Mod); >> >> After rebuild, leads to: >> >> gdal raster calc \ >> --input "p=input.tif" \ >> --calc="mod(p[1], 50) == 0 ? p[1] : 255" \ >> --ot=Byte \ >> --output="output.tif" --co COMPRESS=DEFLATE --overwrite --progress >> >> Ideally, though, having a mod function in GDAL itself would be awesome. >> >> Scott >> >> >> On 10/7/25 09:52, Daniel Baston wrote: >>> It's fairly straightforward to define functions in C++ and register them >>> with muparser. You can see how we do this with "isnan" and "isnodata" here: >>> >>> https://github.com/OSGeo/gdal/blob/ >>> ca7a2d38893a8d6cdc03653b47712f4442d99d65/frmts/vrt/ >>> vrtexpression_muparser.cpp#L27 >> ca7a2d38893a8d6cdc03653b47712f4442d99d65/frmts/vrt/ >>> vrtexpression_muparser.cpp#L27> >>> >>> I don't think we want too many custom functions, but adding "mod" seems >>> like a good idea. >>> >>> Dan >>> >>> On Tue, Oct 7, 2025 at 12:30?PM Scott via gdal-dev >> dev at lists.osgeo.org > wrote: >>> >>> Greetings! >>> >>> I've been unable to find a modulo operator in CLI calc. It's not in the >>> muparser functions or appear to be in the forth-coming 3.12 C pixel >>> functions. >>> >>> This has been working with muparser, but it's messy every time you need >>> a simple modulo. pixel is real or int, div is int: >>> >>> rint(pixel) - (rint(rint(pixel) / div) * div) >>> >>> Am I missing something? If not, consider this a feature request! :) >>> >>> Thanks! >>> Scott >>> >>> -- >>> www.postholer.com >>> >>> _______________________________________________ >>> gdal-dev mailing list >>> gdal-dev at lists.osgeo.org >>> https://lists.osgeo.org/mailman/listinfo/gdal-dev >> lists.osgeo.org/mailman/listinfo/gdal-dev> >>> >> >> _______________________________________________ >> gdal-dev mailing list >> gdal-dev at lists.osgeo.org >> https://lists.osgeo.org/mailman/listinfo/gdal-dev From even.rouault at spatialys.com Tue Oct 7 14:50:14 2025 From: even.rouault at spatialys.com (Even Rouault) Date: Tue, 7 Oct 2025 23:50:14 +0200 Subject: [gdal-dev] Properly reinitializing PROJ_LIB search paths after supplementary file download In-Reply-To: References: <1e99883e-09a2-4420-acc5-ee021acc20b1@spatialys.com> Message-ID: <0f18b0e7-ee52-4aa3-a0bd-0cb68938cf0e@spatialys.com> Le 07/10/2025 ? 23:47, David Klaus a ?crit?: > Even, > > I will check on this in a moment. If there are OGR/Proj/GDAL entities > on the heap/stack -- such as?OGRCoordinateTransformation -- will > calling OSRCleanup cause issues? That should be fine. You should however avoid any concurrent GDAL call in another thread > > On Tue, Oct 7, 2025 at 5:43?PM Even Rouault > wrote: > > David, > > does calling OSRCleanup() help ? > > Even > > Le 07/10/2025 ? 17:00, David Klaus via gdal-dev a ?crit?: >> Hello GDAL community, >> >> The product that I work on uses the CPP GDAL library for a number >> of routines. Occasionally, this requires supplementary files -- >> Currently, we supply proj-data 1.20 >> -- which >> we download when we detect that the dataset is needed. Here's the >> problem, often GDALRegisterAll() is called before the proj-data >> 1.20 files are downloaded. From my testing, it appears that if >> the proj-data 1.20 files are not available when GDALRegisterAll() >> is called, GDAL will not use them regardless of whether or not >> these files are available later. >> >> Now, what I think might be a good solution is appropriately >> updating GDAL's state after proj-data is downloaded s.t. it is >> able to detect the proj-data files. But, I'm not sure how to do >> this. Calling the following seems to produce correct results: >> >> GDALDestroy() >> GDALAllRegister() >> >> However, the documentation says the following: >> >> ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////FROM >> DOCUMENTATION/////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// >> Finalize GDAL/OGR library. >> >> This function calls GDALDestroyDriverManager() and >> OGRCleanupAll() and finalize Thread Local Storage variables. >> >> Prior to GDAL 2.4.0, this function should normally be explicitly >> called by application code if GDAL is dynamically linked (but >> that does not hurt), since it was automatically called through >> the unregistration mechanisms of dynamic library loading. >> >> Since GDAL 2.4.0, this function may be called by application >> code, since it is no longer called automatically, on non-MSVC >> builds, due to ordering problems with respect to automatic >> destruction of global C++ objects. >> >> *Note: no GDAL/OGR code should be called after this call!* >> ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////FROM >> DOCUMENTATION/////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// >> >> So, this doesn't seem like the correct way to address this >> problem. How should I go about updating GDAL's state s.t. it will >> detect and read the downloaded supplementary proj-data files? Or, >> is there something more I should be doing before >> GDALAllRegister() is called? >> >> GDAL Version: GDAL 3.10.3, released 2025/04/01 >> Proj Version: 9.6.0 >> >> P.S. To get out in front of what I expect to be the first >> question, yes I am setting the?PROJ_LIB environmental variable at >> runtime to the correct folder. Further, I have tried creating an >> empty proj-data folder available to GDAL when GDALRegisterAll() >> is called. This did not change GDAL's behavior. Here is the code >> I use for setting PROJ_LIB: >> >> SetEnvironmentVariable("PROJ_LIB", "c:/path/to/proj-data/folder"); >> _putenv_s("PROJ_LIB", "c:/path/to/proj-data/folder"); // (See: >> GETENV_NOTE) >> >> P.P.S Here is the GETENV_NOTE: >> // GETENV_NOTE: >> // _putenv_s() is used to ensure compatibility with getenv(). >> getenv() operates on environment variables loaded into? the >> _environ global variable (loaded at "process startup.") >> // SetEnvironmentVariable() affects the environment variables set >> for this process but does not change _environ. _putenv_s() >> updates _environ and ensures compatibility w/ >> //?genenv(). >> >> P.P.P.S I do know that there is a more current proj-data dataset >> available. We may update to this dataset in the future. But >> unless that will fix the current issue, I'd rather not do that now. >> >> >> -- >> David Klaus >> Carlson Software >> >> >> *Disclaimer* >> >> The information contained in this communication from the sender >> is confidential. It is intended solely for use by the recipient >> and others authorized to receive it. If you are not the >> recipient, you are hereby notified that any disclosure, copying, >> distribution or taking action in relation of the contents of this >> information is strictly prohibited and may be unlawful. >> >> >> _______________________________________________ >> gdal-dev mailing list >> gdal-dev at lists.osgeo.org >> https://lists.osgeo.org/mailman/listinfo/gdal-dev > > -- > http://www.spatialys.com > My software is free, but my time generally not. > > > > -- > David Klaus > Carlson Software > > > *Disclaimer* > > The information contained in this communication from the sender is > confidential. It is intended solely for use by the recipient and > others authorized to receive it. If you are not the recipient, you are > hereby notified that any disclosure, copying, distribution or taking > action in relation of the contents of this information is strictly > prohibited and may be unlawful. > -- http://www.spatialys.com My software is free, but my time generally not. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bdezonia at gmail.com Tue Oct 7 14:55:12 2025 From: bdezonia at gmail.com (Barry DeZonia) Date: Tue, 7 Oct 2025 16:55:12 -0500 Subject: [gdal-dev] gdal raster calc - a modulo operator (%)? In-Reply-To: <6e9ad71f-c48f-42e9-9140-949153837c0a@postholer.com> References: <9d29bfaf-6f3e-46c0-8e95-3897179e3ccf@postholer.com> <6e9ad71f-c48f-42e9-9140-949153837c0a@postholer.com> Message-ID: I'm not sure what the operational definition of raster is. In my world you can have a floating point raster (a grid of floating point values instead of a grid of integer values). I might be wrong about that in your context. Interestingly the Java language specs are helpful: See Section 15.17.3 Remainder Operator % in this document: https://docs.oracle.com/javase/specs/jls/se25/jls25.pdf On Tue, Oct 7, 2025 at 4:47?PM Scott wrote: > > Hey Barry, > > I can confirm there are a whole slew of issues I haven't begun to think > through! ;) > > In context of a raster pixel, integers shouldn't be an issue. Can a > raster pixel even have a NaN value? A nodata IS a value. I don't even > know. Is this a chunk of code I'd encourage everyone to use? Definitely not. > > Scott > > On 10/7/25 14:30, Barry DeZonia wrote: > > This might be ill behaved for floating point. Rounding errors and > > correct behavior for Infs and NaNs etc can spring surprises on you. > > Java has a floating point modulus operator. I am betting the Double or > > Float class java api javadoc might discuss implementation notes. > > > > On Tue, Oct 7, 2025 at 4:24?PM Scott via gdal-dev > > wrote: > >> > >> Thanks for the idea Dan. Building on that, I added it upstream to the > >> MuParser source. Now I have it in libmuparser as well: > >> > >> ./muparser-2.3.5/include/muParserTemplateMagic.h > >> static T Mod(T pixel, T div) { return lround(pixel) - > >> ((lround(pixel) / lround(div)) * lround(div)); } > >> > >> ./muparser-2.3.5/src/muParser.cpp > >> DefineFun(_T("mod"), MathImpl::Mod); > >> > >> After rebuild, leads to: > >> > >> gdal raster calc \ > >> --input "p=input.tif" \ > >> --calc="mod(p[1], 50) == 0 ? p[1] : 255" \ > >> --ot=Byte \ > >> --output="output.tif" --co COMPRESS=DEFLATE --overwrite --progress > >> > >> Ideally, though, having a mod function in GDAL itself would be awesome. > >> > >> Scott > >> > >> > >> On 10/7/25 09:52, Daniel Baston wrote: > >>> It's fairly straightforward to define functions in C++ and register them > >>> with muparser. You can see how we do this with "isnan" and "isnodata" here: > >>> > >>> https://github.com/OSGeo/gdal/blob/ > >>> ca7a2d38893a8d6cdc03653b47712f4442d99d65/frmts/vrt/ > >>> vrtexpression_muparser.cpp#L27 >>> ca7a2d38893a8d6cdc03653b47712f4442d99d65/frmts/vrt/ > >>> vrtexpression_muparser.cpp#L27> > >>> > >>> I don't think we want too many custom functions, but adding "mod" seems > >>> like a good idea. > >>> > >>> Dan > >>> > >>> On Tue, Oct 7, 2025 at 12:30?PM Scott via gdal-dev >>> dev at lists.osgeo.org > wrote: > >>> > >>> Greetings! > >>> > >>> I've been unable to find a modulo operator in CLI calc. It's not in the > >>> muparser functions or appear to be in the forth-coming 3.12 C pixel > >>> functions. > >>> > >>> This has been working with muparser, but it's messy every time you need > >>> a simple modulo. pixel is real or int, div is int: > >>> > >>> rint(pixel) - (rint(rint(pixel) / div) * div) > >>> > >>> Am I missing something? If not, consider this a feature request! :) > >>> > >>> Thanks! > >>> Scott > >>> > >>> -- > >>> www.postholer.com > >>> > >>> _______________________________________________ > >>> gdal-dev mailing list > >>> gdal-dev at lists.osgeo.org > >>> https://lists.osgeo.org/mailman/listinfo/gdal-dev >>> lists.osgeo.org/mailman/listinfo/gdal-dev> > >>> > >> > >> _______________________________________________ > >> gdal-dev mailing list > >> gdal-dev at lists.osgeo.org > >> https://lists.osgeo.org/mailman/listinfo/gdal-dev > From bdezonia at gmail.com Tue Oct 7 15:11:25 2025 From: bdezonia at gmail.com (Barry DeZonia) Date: Tue, 7 Oct 2025 17:11:25 -0500 Subject: [gdal-dev] gdal raster calc - a modulo operator (%)? In-Reply-To: References: <9d29bfaf-6f3e-46c0-8e95-3897179e3ccf@postholer.com> <6e9ad71f-c48f-42e9-9140-949153837c0a@postholer.com> Message-ID: Yes, and I think you need to choose carefully between the 3 sources (java and the two C++/C ones) based upon what you want edge case behavior to be and whether to copysign() or not or sometimes etc. On Tue, Oct 7, 2025 at 5:04?PM Javier Jimenez Shaw wrote: > > In c++ there is > https://en.cppreference.com/w/cpp/numeric/math/fmod > And > https://en.cppreference.com/w/cpp/numeric/math/remainder.html > > On Tue, 7 Oct 2025, 23:55 Barry DeZonia via gdal-dev, wrote: >> >> I'm not sure what the operational definition of raster is. In my world >> you can have a floating point raster (a grid of floating point values >> instead of a grid of integer values). I might be wrong about that in >> your context. >> >> Interestingly the Java language specs are helpful: >> >> See Section 15.17.3 Remainder Operator % >> >> in this document: >> >> https://docs.oracle.com/javase/specs/jls/se25/jls25.pdf >> >> >> On Tue, Oct 7, 2025 at 4:47?PM Scott wrote: >> > >> > Hey Barry, >> > >> > I can confirm there are a whole slew of issues I haven't begun to think >> > through! ;) >> > >> > In context of a raster pixel, integers shouldn't be an issue. Can a >> > raster pixel even have a NaN value? A nodata IS a value. I don't even >> > know. Is this a chunk of code I'd encourage everyone to use? Definitely not. >> > >> > Scott >> > >> > On 10/7/25 14:30, Barry DeZonia wrote: >> > > This might be ill behaved for floating point. Rounding errors and >> > > correct behavior for Infs and NaNs etc can spring surprises on you. >> > > Java has a floating point modulus operator. I am betting the Double or >> > > Float class java api javadoc might discuss implementation notes. >> > > >> > > On Tue, Oct 7, 2025 at 4:24?PM Scott via gdal-dev >> > > wrote: >> > >> >> > >> Thanks for the idea Dan. Building on that, I added it upstream to the >> > >> MuParser source. Now I have it in libmuparser as well: >> > >> >> > >> ./muparser-2.3.5/include/muParserTemplateMagic.h >> > >> static T Mod(T pixel, T div) { return lround(pixel) - >> > >> ((lround(pixel) / lround(div)) * lround(div)); } >> > >> >> > >> ./muparser-2.3.5/src/muParser.cpp >> > >> DefineFun(_T("mod"), MathImpl::Mod); >> > >> >> > >> After rebuild, leads to: >> > >> >> > >> gdal raster calc \ >> > >> --input "p=input.tif" \ >> > >> --calc="mod(p[1], 50) == 0 ? p[1] : 255" \ >> > >> --ot=Byte \ >> > >> --output="output.tif" --co COMPRESS=DEFLATE --overwrite --progress >> > >> >> > >> Ideally, though, having a mod function in GDAL itself would be awesome. >> > >> >> > >> Scott >> > >> >> > >> >> > >> On 10/7/25 09:52, Daniel Baston wrote: >> > >>> It's fairly straightforward to define functions in C++ and register them >> > >>> with muparser. You can see how we do this with "isnan" and "isnodata" here: >> > >>> >> > >>> https://github.com/OSGeo/gdal/blob/ >> > >>> ca7a2d38893a8d6cdc03653b47712f4442d99d65/frmts/vrt/ >> > >>> vrtexpression_muparser.cpp#L27 > > >>> ca7a2d38893a8d6cdc03653b47712f4442d99d65/frmts/vrt/ >> > >>> vrtexpression_muparser.cpp#L27> >> > >>> >> > >>> I don't think we want too many custom functions, but adding "mod" seems >> > >>> like a good idea. >> > >>> >> > >>> Dan >> > >>> >> > >>> On Tue, Oct 7, 2025 at 12:30?PM Scott via gdal-dev > > >>> dev at lists.osgeo.org > wrote: >> > >>> >> > >>> Greetings! >> > >>> >> > >>> I've been unable to find a modulo operator in CLI calc. It's not in the >> > >>> muparser functions or appear to be in the forth-coming 3.12 C pixel >> > >>> functions. >> > >>> >> > >>> This has been working with muparser, but it's messy every time you need >> > >>> a simple modulo. pixel is real or int, div is int: >> > >>> >> > >>> rint(pixel) - (rint(rint(pixel) / div) * div) >> > >>> >> > >>> Am I missing something? If not, consider this a feature request! :) >> > >>> >> > >>> Thanks! >> > >>> Scott >> > >>> >> > >>> -- >> > >>> www.postholer.com >> > >>> >> > >>> _______________________________________________ >> > >>> gdal-dev mailing list >> > >>> gdal-dev at lists.osgeo.org >> > >>> https://lists.osgeo.org/mailman/listinfo/gdal-dev > > >>> lists.osgeo.org/mailman/listinfo/gdal-dev> >> > >>> >> > >> >> > >> _______________________________________________ >> > >> gdal-dev mailing list >> > >> gdal-dev at lists.osgeo.org >> > >> https://lists.osgeo.org/mailman/listinfo/gdal-dev >> > >> _______________________________________________ >> gdal-dev mailing list >> gdal-dev at lists.osgeo.org >> https://lists.osgeo.org/mailman/listinfo/gdal-dev From j1 at jimenezshaw.com Tue Oct 7 15:04:31 2025 From: j1 at jimenezshaw.com (Javier Jimenez Shaw) Date: Wed, 8 Oct 2025 00:04:31 +0200 Subject: [gdal-dev] gdal raster calc - a modulo operator (%)? In-Reply-To: References: <9d29bfaf-6f3e-46c0-8e95-3897179e3ccf@postholer.com> <6e9ad71f-c48f-42e9-9140-949153837c0a@postholer.com> Message-ID: In c++ there is https://en.cppreference.com/w/cpp/numeric/math/fmod And https://en.cppreference.com/w/cpp/numeric/math/remainder.html On Tue, 7 Oct 2025, 23:55 Barry DeZonia via gdal-dev, < gdal-dev at lists.osgeo.org> wrote: > I'm not sure what the operational definition of raster is. In my world > you can have a floating point raster (a grid of floating point values > instead of a grid of integer values). I might be wrong about that in > your context. > > Interestingly the Java language specs are helpful: > > See Section 15.17.3 Remainder Operator % > > in this document: > > https://docs.oracle.com/javase/specs/jls/se25/jls25.pdf > > > On Tue, Oct 7, 2025 at 4:47?PM Scott wrote: > > > > Hey Barry, > > > > I can confirm there are a whole slew of issues I haven't begun to think > > through! ;) > > > > In context of a raster pixel, integers shouldn't be an issue. Can a > > raster pixel even have a NaN value? A nodata IS a value. I don't even > > know. Is this a chunk of code I'd encourage everyone to use? Definitely > not. > > > > Scott > > > > On 10/7/25 14:30, Barry DeZonia wrote: > > > This might be ill behaved for floating point. Rounding errors and > > > correct behavior for Infs and NaNs etc can spring surprises on you. > > > Java has a floating point modulus operator. I am betting the Double or > > > Float class java api javadoc might discuss implementation notes. > > > > > > On Tue, Oct 7, 2025 at 4:24?PM Scott via gdal-dev > > > wrote: > > >> > > >> Thanks for the idea Dan. Building on that, I added it upstream to the > > >> MuParser source. Now I have it in libmuparser as well: > > >> > > >> ./muparser-2.3.5/include/muParserTemplateMagic.h > > >> static T Mod(T pixel, T div) { return lround(pixel) - > > >> ((lround(pixel) / lround(div)) * lround(div)); } > > >> > > >> ./muparser-2.3.5/src/muParser.cpp > > >> DefineFun(_T("mod"), MathImpl::Mod); > > >> > > >> After rebuild, leads to: > > >> > > >> gdal raster calc \ > > >> --input "p=input.tif" \ > > >> --calc="mod(p[1], 50) == 0 ? p[1] : 255" \ > > >> --ot=Byte \ > > >> --output="output.tif" --co COMPRESS=DEFLATE --overwrite > --progress > > >> > > >> Ideally, though, having a mod function in GDAL itself would be > awesome. > > >> > > >> Scott > > >> > > >> > > >> On 10/7/25 09:52, Daniel Baston wrote: > > >>> It's fairly straightforward to define functions in C++ and register > them > > >>> with muparser. You can see how we do this with "isnan" and > "isnodata" here: > > >>> > > >>> https://github.com/OSGeo/gdal/blob/ > > >>> ca7a2d38893a8d6cdc03653b47712f4442d99d65/frmts/vrt/ > > >>> vrtexpression_muparser.cpp#L27 > >>> ca7a2d38893a8d6cdc03653b47712f4442d99d65/frmts/vrt/ > > >>> vrtexpression_muparser.cpp#L27> > > >>> > > >>> I don't think we want too many custom functions, but adding "mod" > seems > > >>> like a good idea. > > >>> > > >>> Dan > > >>> > > >>> On Tue, Oct 7, 2025 at 12:30?PM Scott via gdal-dev > >>> dev at lists.osgeo.org > wrote: > > >>> > > >>> Greetings! > > >>> > > >>> I've been unable to find a modulo operator in CLI calc. It's > not in the > > >>> muparser functions or appear to be in the forth-coming 3.12 C > pixel > > >>> functions. > > >>> > > >>> This has been working with muparser, but it's messy every time > you need > > >>> a simple modulo. pixel is real or int, div is int: > > >>> > > >>> rint(pixel) - (rint(rint(pixel) / div) * div) > > >>> > > >>> Am I missing something? If not, consider this a feature > request! :) > > >>> > > >>> Thanks! > > >>> Scott > > >>> > > >>> -- > > >>> www.postholer.com > > >>> > > >>> _______________________________________________ > > >>> gdal-dev mailing list > > >>> gdal-dev at lists.osgeo.org > > >>> https://lists.osgeo.org/mailman/listinfo/gdal-dev > >>> lists.osgeo.org/mailman/listinfo/gdal-dev> > > >>> > > >> > > >> _______________________________________________ > > >> gdal-dev mailing list > > >> gdal-dev at lists.osgeo.org > > >> https://lists.osgeo.org/mailman/listinfo/gdal-dev > > > _______________________________________________ > gdal-dev mailing list > gdal-dev at lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dklaus at carlsonsw.com Tue Oct 7 15:16:29 2025 From: dklaus at carlsonsw.com (David Klaus) Date: Tue, 7 Oct 2025 18:16:29 -0400 Subject: [gdal-dev] Properly reinitializing PROJ_LIB search paths after supplementary file download In-Reply-To: <0f18b0e7-ee52-4aa3-a0bd-0cb68938cf0e@spatialys.com> References: <1e99883e-09a2-4420-acc5-ee021acc20b1@spatialys.com> <0f18b0e7-ee52-4aa3-a0bd-0cb68938cf0e@spatialys.com> Message-ID: Even, Thank you for the suggestion. Calling OSRCleanup() corrected the issue I was seeing. So, I guess the previously used projections were cached and because of this the proj-data files weren't being used after download, On Tue, Oct 7, 2025 at 5:50?PM Even Rouault wrote: > Le 07/10/2025 ? 23:47, David Klaus a ?crit : > > Even, > > I will check on this in a moment. If there are OGR/Proj/GDAL entities on > the heap/stack -- such as OGRCoordinateTransformation -- will calling > OSRCleanup cause issues? > > That should be fine. You should however avoid any concurrent GDAL call in > another thread > > > On Tue, Oct 7, 2025 at 5:43?PM Even Rouault > wrote: > >> David, >> >> does calling OSRCleanup() help ? >> >> Even >> Le 07/10/2025 ? 17:00, David Klaus via gdal-dev a ?crit : >> >> Hello GDAL community, >> >> The product that I work on uses the CPP GDAL library for a number of >> routines. Occasionally, this requires supplementary files -- Currently, we >> supply proj-data 1.20 >> >> -- which we download when we detect that the dataset is needed. Here's the >> problem, often GDALRegisterAll() is called before the proj-data 1.20 files >> are downloaded. From my testing, it appears that if the proj-data 1.20 >> files are not available when GDALRegisterAll() is called, GDAL will not use >> them regardless of whether or not these files are available later. >> >> Now, what I think might be a good solution is appropriately updating >> GDAL's state after proj-data is downloaded s.t. it is able to detect the >> proj-data files. But, I'm not sure how to do this. Calling the following >> seems to produce correct results: >> >> GDALDestroy() >> GDALAllRegister() >> >> However, the documentation says the following: >> >> ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////FROM >> DOCUMENTATION/////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// >> Finalize GDAL/OGR library. >> >> This function calls GDALDestroyDriverManager() and OGRCleanupAll() and >> finalize Thread Local Storage variables. >> >> Prior to GDAL 2.4.0, this function should normally be explicitly called >> by application code if GDAL is dynamically linked (but that does not hurt), >> since it was automatically called through the unregistration mechanisms of >> dynamic library loading. >> >> Since GDAL 2.4.0, this function may be called by application code, since >> it is no longer called automatically, on non-MSVC builds, due to ordering >> problems with respect to automatic destruction of global C++ objects. >> >> *Note: no GDAL/OGR code should be called after this call!* >> ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////FROM >> DOCUMENTATION/////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// >> >> So, this doesn't seem like the correct way to address this problem. How >> should I go about updating GDAL's state s.t. it will detect and read the >> downloaded supplementary proj-data files? Or, is there something more I >> should be doing before GDALAllRegister() is called? >> >> GDAL Version: GDAL 3.10.3, released 2025/04/01 >> Proj Version: 9.6.0 >> >> P.S. To get out in front of what I expect to be the first question, yes I >> am setting the PROJ_LIB environmental variable at runtime to the correct >> folder. Further, I have tried creating an empty proj-data folder available >> to GDAL when GDALRegisterAll() is called. This did not change GDAL's >> behavior. Here is the code I use for setting PROJ_LIB: >> >> SetEnvironmentVariable("PROJ_LIB", "c:/path/to/proj-data/folder"); >> _putenv_s("PROJ_LIB", "c:/path/to/proj-data/folder"); // (See: >> GETENV_NOTE) >> >> P.P.S Here is the GETENV_NOTE: >> // GETENV_NOTE: >> // _putenv_s() is used to ensure compatibility with getenv(). getenv() >> operates on environment variables loaded into the _environ global variable >> (loaded at "process startup.") >> // SetEnvironmentVariable() affects the environment variables set for >> this process but does not change _environ. _putenv_s() updates _environ and >> ensures compatibility w/ >> // genenv(). >> >> P.P.P.S I do know that there is a more current proj-data dataset >> available. We may update to this dataset in the future. But unless that >> will fix the current issue, I'd rather not do that now. >> >> >> -- >> David Klaus >> Carlson Software >> >> >> *Disclaimer* >> >> The information contained in this communication from the sender is >> confidential. It is intended solely for use by the recipient and others >> authorized to receive it. If you are not the recipient, you are hereby >> notified that any disclosure, copying, distribution or taking action in >> relation of the contents of this information is strictly prohibited and may >> be unlawful. >> >> _______________________________________________ >> gdal-dev mailing listgdal-dev at lists.osgeo.orghttps://lists.osgeo.org/mailman/listinfo/gdal-dev >> >> -- http://www.spatialys.com >> My software is free, but my time generally not. >> >> > > -- > David Klaus > Carlson Software > > > *Disclaimer* > > The information contained in this communication from the sender is > confidential. It is intended solely for use by the recipient and others > authorized to receive it. If you are not the recipient, you are hereby > notified that any disclosure, copying, distribution or taking action in > relation of the contents of this information is strictly prohibited and may > be unlawful. > > -- http://www.spatialys.com > My software is free, but my time generally not. > > -- David Klaus Carlson Software Disclaimer The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at home.gofferje.net Wed Oct 8 22:36:57 2025 From: lists at home.gofferje.net (Stefan Gofferje) Date: Thu, 9 Oct 2025 05:36:57 +0000 Subject: [gdal-dev] AVIF write support In-Reply-To: <81e24739-e1b2-fdea-0d6a-b0eb71acdc62@aitchison.me.uk> References: <01070199b33da15b-4c5e1157-39df-437a-91fd-ea7cac4d6443-000000@eu-central-1.amazonses.com> <61674911-7835-65f0-bd51-572d2da0cbba@aitchison.me.uk> <01070199bdcdd9ed-bc837cb3-8760-4be5-9350-a6c0b2c1ce7e-000000@eu-central-1.amazonses.com> <81e24739-e1b2-fdea-0d6a-b0eb71acdc62@aitchison.me.uk> Message-ID: <01070199c778f518-e9106d54-a8cf-4bf5-964a-a99d2a18510e-000000@eu-central-1.amazonses.com> ??[sgofferj at enterprise][~] ???? apt list *avif* Listing... Done libavif-bin/noble 1.0.4-1ubuntu3 amd64 libavif-bin/noble 1.0.4-1ubuntu3 i386 libavif-dev/noble,now 1.0.4-1ubuntu3 amd64 [installed,automatic] libavif-dev/noble 1.0.4-1ubuntu3 i386 libavif-gdk-pixbuf/noble 1.0.4-1ubuntu3 amd64 libavif-gdk-pixbuf/noble 1.0.4-1ubuntu3 i386 libavif16/noble,now 1.0.4-1ubuntu3 amd64 [installed,automatic] libavif16/noble 1.0.4-1ubuntu3 i386 libavifile-0.7-bin/noble 1:0.7.48~20090503.ds-26build1 amd64 libavifile-0.7-common/noble,noble 1:0.7.48~20090503.ds-26build1 all libavifile-0.7-dev/noble 1:0.7.48~20090503.ds-26build1 amd64 libavifile-0.7c2/noble 1:0.7.48~20090503.ds-26build1 amd64 qt5-avif-image-plugin/noble 0.7.1+dfsg-1build2 amd64 Seems AVIF is installed. I also installed gdal through curl -sL "https://url.geocarpentry.org/gdal-ubuntu" | bash i.e. not through the package manager because the Ubuntu 24.04 stock version is a little dated. -- (o_ Stefan Gofferje | SCLT, MCP, CCSA //\ Reg'd Linux User #247167 | VCP #2263 V_/_ https://www.gofferje.net | https://www.saakeskus.fi On 10/7/25 14:34, Andrew C Aitchison wrote: > On Tue, 7 Oct 2025, Stefan Gofferje wrote: > >> ??[sgofferj at enterprise][~] >> ???? gdalinfo --formats | egrep -i "avif|support" >> Supported Formats: (ro:read-only, rw:read-write, +:write from scratch, >> u:update, v:virtual-I/O s:subdatasets) >> ?AVIF_HEIF -raster- (rov): AV1 Image File Format (using libheif) (*.avif) >> ??[sgofferj at enterprise][~] >> ???? gdalinfo --version >> GDAL 3.11.4 "Eganville", released 2025/09/04 >> >> Ubuntu 24.04, installed via gdal install script. > > Do you have any libavif or libheif package installed ? > > apt list libavif* libheif* avif* heif* | grep installed > > WARNING: apt does not have a stable CLI interface. Use with caution in > scripts. > > heif-gdk-pixbuf/plucky,now 1.19.7-1 amd64 [installed] > heif-thumbnailer/plucky,now 1.19.7-1 amd64 [installed] > libavif-dev/plucky,now 1.2.1-1ubuntu1 amd64 [installed,automatic] > libavif16/plucky,now 1.2.1-1ubuntu1 amd64 [installed,automatic] > libheif-dev/plucky,now 1.19.7-1 amd64 [installed,automatic] > libheif-examples/plucky,now 1.19.7-1 amd64 [installed] > libheif-plugin-aomdec/plucky,now 1.19.7-1 i386 [installed,automatic] > libheif-plugin-aomenc/plucky,now 1.19.7-1 amd64 [installed,automatic] > libheif-plugin-aomenc/plucky,now 1.19.7-1 i386 [installed,automatic] > libheif-plugin-dav1d/plucky,now 1.19.7-1 amd64 [installed,automatic] > libheif-plugin-libde265/plucky,now 1.19.7-1 amd64 [installed,automatic] > libheif-plugin-libde265/plucky,now 1.19.7-1 i386 [installed,automatic] > libheif-plugin-x265/plucky,now 1.19.7-1 amd64 [installed,automatic] > libheif1/plucky,now 1.19.7-1 amd64 [installed] > libheif1/plucky,now 1.19.7-1 i386 [installed,automatic] > > > From lnicola at dend.ro Wed Oct 8 22:52:35 2025 From: lnicola at dend.ro (=?UTF-8?Q?Lauren=C8=9Biu_Nicola?=) Date: Thu, 09 Oct 2025 08:52:35 +0300 Subject: [gdal-dev] AVIF write support In-Reply-To: <01070199c778f518-e9106d54-a8cf-4bf5-964a-a99d2a18510e-000000@eu-central-1.amazonses.com> References: <01070199b33da15b-4c5e1157-39df-437a-91fd-ea7cac4d6443-000000@eu-central-1.amazonses.com> <61674911-7835-65f0-bd51-572d2da0cbba@aitchison.me.uk> <01070199bdcdd9ed-bc837cb3-8760-4be5-9350-a6c0b2c1ce7e-000000@eu-central-1.amazonses.com> <81e24739-e1b2-fdea-0d6a-b0eb71acdc62@aitchison.me.uk> <01070199c778f518-e9106d54-a8cf-4bf5-964a-a99d2a18510e-000000@eu-central-1.amazonses.com> Message-ID: <29f11d31-cbc4-4610-95dd-2ced6e01fdc3@betaapp.fastmail.com> It's the first time I see that script, but it looks like it would just pick up the libraries that are installed. Did you install libavif-dev after GDAL? https://gist.githubusercontent.com/samapriya/412babdfd3530c2766acb9d603ed1bb9/raw/e70025540623c40bc141f7db440fcb04959d5793/gdal-ubuntu-latest.sh Laurentiu On Thu, Oct 9, 2025, at 08:36, Stefan Gofferje via gdal-dev wrote: > ??[sgofferj at enterprise][~] > ???? apt list *avif* > Listing... Done > libavif-bin/noble 1.0.4-1ubuntu3 amd64 > libavif-bin/noble 1.0.4-1ubuntu3 i386 > libavif-dev/noble,now 1.0.4-1ubuntu3 amd64 [installed,automatic] > libavif-dev/noble 1.0.4-1ubuntu3 i386 > libavif-gdk-pixbuf/noble 1.0.4-1ubuntu3 amd64 > libavif-gdk-pixbuf/noble 1.0.4-1ubuntu3 i386 > libavif16/noble,now 1.0.4-1ubuntu3 amd64 [installed,automatic] > libavif16/noble 1.0.4-1ubuntu3 i386 > libavifile-0.7-bin/noble 1:0.7.48~20090503.ds-26build1 amd64 > libavifile-0.7-common/noble,noble 1:0.7.48~20090503.ds-26build1 all > libavifile-0.7-dev/noble 1:0.7.48~20090503.ds-26build1 amd64 > libavifile-0.7c2/noble 1:0.7.48~20090503.ds-26build1 amd64 > qt5-avif-image-plugin/noble 0.7.1+dfsg-1build2 amd64 > > Seems AVIF is installed. I also installed gdal through > curl -sL "https://url.geocarpentry.org/gdal-ubuntu" | bash > i.e. not through the package manager because the Ubuntu 24.04 stock > version is a little dated. > > -- > (o_ Stefan Gofferje | SCLT, MCP, CCSA > //\ Reg'd Linux User #247167 | VCP #2263 > V_/_ https://www.gofferje.net | https://www.saakeskus.fi > > On 10/7/25 14:34, Andrew C Aitchison wrote: >> On Tue, 7 Oct 2025, Stefan Gofferje wrote: >> >>> ??[sgofferj at enterprise][~] >>> ???? gdalinfo --formats | egrep -i "avif|support" >>> Supported Formats: (ro:read-only, rw:read-write, +:write from scratch, >>> u:update, v:virtual-I/O s:subdatasets) >>> ?AVIF_HEIF -raster- (rov): AV1 Image File Format (using libheif) (*.avif) >>> ??[sgofferj at enterprise][~] >>> ???? gdalinfo --version >>> GDAL 3.11.4 "Eganville", released 2025/09/04 >>> >>> Ubuntu 24.04, installed via gdal install script. >> >> Do you have any libavif or libheif package installed ? >> >> apt list libavif* libheif* avif* heif* | grep installed >> >> WARNING: apt does not have a stable CLI interface. Use with caution in >> scripts. >> >> heif-gdk-pixbuf/plucky,now 1.19.7-1 amd64 [installed] >> heif-thumbnailer/plucky,now 1.19.7-1 amd64 [installed] >> libavif-dev/plucky,now 1.2.1-1ubuntu1 amd64 [installed,automatic] >> libavif16/plucky,now 1.2.1-1ubuntu1 amd64 [installed,automatic] >> libheif-dev/plucky,now 1.19.7-1 amd64 [installed,automatic] >> libheif-examples/plucky,now 1.19.7-1 amd64 [installed] >> libheif-plugin-aomdec/plucky,now 1.19.7-1 i386 [installed,automatic] >> libheif-plugin-aomenc/plucky,now 1.19.7-1 amd64 [installed,automatic] >> libheif-plugin-aomenc/plucky,now 1.19.7-1 i386 [installed,automatic] >> libheif-plugin-dav1d/plucky,now 1.19.7-1 amd64 [installed,automatic] >> libheif-plugin-libde265/plucky,now 1.19.7-1 amd64 [installed,automatic] >> libheif-plugin-libde265/plucky,now 1.19.7-1 i386 [installed,automatic] >> libheif-plugin-x265/plucky,now 1.19.7-1 amd64 [installed,automatic] >> libheif1/plucky,now 1.19.7-1 amd64 [installed] >> libheif1/plucky,now 1.19.7-1 i386 [installed,automatic] >> >> >> > > > > _______________________________________________ > gdal-dev mailing list > gdal-dev at lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev From lists at home.gofferje.net Thu Oct 9 00:39:16 2025 From: lists at home.gofferje.net (Stefan Gofferje) Date: Thu, 9 Oct 2025 07:39:16 +0000 Subject: [gdal-dev] AVIF write support In-Reply-To: <29f11d31-cbc4-4610-95dd-2ced6e01fdc3@betaapp.fastmail.com> References: <01070199b33da15b-4c5e1157-39df-437a-91fd-ea7cac4d6443-000000@eu-central-1.amazonses.com> <61674911-7835-65f0-bd51-572d2da0cbba@aitchison.me.uk> <01070199bdcdd9ed-bc837cb3-8760-4be5-9350-a6c0b2c1ce7e-000000@eu-central-1.amazonses.com> <81e24739-e1b2-fdea-0d6a-b0eb71acdc62@aitchison.me.uk> <01070199c778f518-e9106d54-a8cf-4bf5-964a-a99d2a18510e-000000@eu-central-1.amazonses.com> <29f11d31-cbc4-4610-95dd-2ced6e01fdc3@betaapp.fastmail.com> Message-ID: <01070199c7e8f40a-000137b6-aa72-4dfb-bddd-4517d8e513bb-000000@eu-central-1.amazonses.com> THAT is a good question :D. I don't really remember installing libavif manually at all. But as it seems to work fine in the docker version I'll just change my pipeline to use docker. -- (o_ Stefan Gofferje | SCLT, MCP, CCSA //\ Reg'd Linux User #247167 | VCP #2263 V_/_ https://www.gofferje.net | https://www.saakeskus.fi On 10/9/25 08:52, Lauren?iu Nicola via gdal-dev wrote: > It's the first time I see that script, but it looks like it would just pick up the libraries that are installed. Did you install libavif-dev after GDAL? > > https://gist.githubusercontent.com/samapriya/412babdfd3530c2766acb9d603ed1bb9/raw/e70025540623c40bc141f7db440fcb04959d5793/gdal-ubuntu-latest.sh > > Laurentiu > > On Thu, Oct 9, 2025, at 08:36, Stefan Gofferje via gdal-dev wrote: >> ??[sgofferj at enterprise][~] >> ???? apt list *avif* >> Listing... Done >> libavif-bin/noble 1.0.4-1ubuntu3 amd64 >> libavif-bin/noble 1.0.4-1ubuntu3 i386 >> libavif-dev/noble,now 1.0.4-1ubuntu3 amd64 [installed,automatic] >> libavif-dev/noble 1.0.4-1ubuntu3 i386 >> libavif-gdk-pixbuf/noble 1.0.4-1ubuntu3 amd64 >> libavif-gdk-pixbuf/noble 1.0.4-1ubuntu3 i386 >> libavif16/noble,now 1.0.4-1ubuntu3 amd64 [installed,automatic] >> libavif16/noble 1.0.4-1ubuntu3 i386 >> libavifile-0.7-bin/noble 1:0.7.48~20090503.ds-26build1 amd64 >> libavifile-0.7-common/noble,noble 1:0.7.48~20090503.ds-26build1 all >> libavifile-0.7-dev/noble 1:0.7.48~20090503.ds-26build1 amd64 >> libavifile-0.7c2/noble 1:0.7.48~20090503.ds-26build1 amd64 >> qt5-avif-image-plugin/noble 0.7.1+dfsg-1build2 amd64 >> >> Seems AVIF is installed. I also installed gdal through >> curl -sL "https://url.geocarpentry.org/gdal-ubuntu" | bash >> i.e. not through the package manager because the Ubuntu 24.04 stock >> version is a little dated. >> >> -- >> (o_ Stefan Gofferje | SCLT, MCP, CCSA >> //\ Reg'd Linux User #247167 | VCP #2263 >> V_/_ https://www.gofferje.net | https://www.saakeskus.fi >> >> On 10/7/25 14:34, Andrew C Aitchison wrote: >>> On Tue, 7 Oct 2025, Stefan Gofferje wrote: >>> >>>> ??[sgofferj at enterprise][~] >>>> ???? gdalinfo --formats | egrep -i "avif|support" >>>> Supported Formats: (ro:read-only, rw:read-write, +:write from scratch, >>>> u:update, v:virtual-I/O s:subdatasets) >>>> ?AVIF_HEIF -raster- (rov): AV1 Image File Format (using libheif) (*.avif) >>>> ??[sgofferj at enterprise][~] >>>> ???? gdalinfo --version >>>> GDAL 3.11.4 "Eganville", released 2025/09/04 >>>> >>>> Ubuntu 24.04, installed via gdal install script. >>> >>> Do you have any libavif or libheif package installed ? >>> >>> apt list libavif* libheif* avif* heif* | grep installed >>> >>> WARNING: apt does not have a stable CLI interface. Use with caution in >>> scripts. >>> >>> heif-gdk-pixbuf/plucky,now 1.19.7-1 amd64 [installed] >>> heif-thumbnailer/plucky,now 1.19.7-1 amd64 [installed] >>> libavif-dev/plucky,now 1.2.1-1ubuntu1 amd64 [installed,automatic] >>> libavif16/plucky,now 1.2.1-1ubuntu1 amd64 [installed,automatic] >>> libheif-dev/plucky,now 1.19.7-1 amd64 [installed,automatic] >>> libheif-examples/plucky,now 1.19.7-1 amd64 [installed] >>> libheif-plugin-aomdec/plucky,now 1.19.7-1 i386 [installed,automatic] >>> libheif-plugin-aomenc/plucky,now 1.19.7-1 amd64 [installed,automatic] >>> libheif-plugin-aomenc/plucky,now 1.19.7-1 i386 [installed,automatic] >>> libheif-plugin-dav1d/plucky,now 1.19.7-1 amd64 [installed,automatic] >>> libheif-plugin-libde265/plucky,now 1.19.7-1 amd64 [installed,automatic] >>> libheif-plugin-libde265/plucky,now 1.19.7-1 i386 [installed,automatic] >>> libheif-plugin-x265/plucky,now 1.19.7-1 amd64 [installed,automatic] >>> libheif1/plucky,now 1.19.7-1 amd64 [installed] >>> libheif1/plucky,now 1.19.7-1 i386 [installed,automatic] >>> >>> >>> >> >> >> >> _______________________________________________ >> gdal-dev mailing list >> gdal-dev at lists.osgeo.org >> https://lists.osgeo.org/mailman/listinfo/gdal-dev > _______________________________________________ > gdal-dev mailing list > gdal-dev at lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev From daniel.fred.evans at gmail.com Thu Oct 9 07:26:45 2025 From: daniel.fred.evans at gmail.com (Daniel Evans) Date: Thu, 9 Oct 2025 15:26:45 +0100 Subject: [gdal-dev] GRIB file being scanned despite .idx being present Message-ID: Hi all, I am attempting to read a single band from a NOAA GRIB2 file on S3, with an associated .idx file. Reading the GRIB2 driver documentation, it is stated that the existence of such an idx file allows a file to be opened without reading all bands. However, looking at the CPL_CURL_VERBOSE=True logs, it appears that GDAL is still paging through the file from the start until reaching the requested band. GDAL identifies the existence of the .idx file: DEBUG:CPLE_None in GRIB: Reading inventories from sidecar file /vsis3/noaa-gfs-bdp-pds/gfs.20250918/00/atmos/gfs.t00z.pgrb2.0p25.f012.idx DEBUG:CPLE_None in S3: Downloading 0-41215 ( https://s3.amazonaws.com/noaa-gfs-bdp-pds/gfs.20250918/00/atmos/gfs.t00z.pgrb2.0p25.f012.idx). .. But it then appears to scan the file from the start until it has passed the requested band: DEBUG:CPLE_None in S3: Downloading 16384-999423 ( https://s3.amazonaws.com/noaa-gfs-bdp-pds/gfs.20250918/00/atmos/gfs.t00z.pgrb2.0p25.f012). .. DEBUG:CPLE_None in S3: Downloading 999424-2965503 ( https://s3.amazonaws.com/noaa-gfs-bdp-pds/gfs.20250918/00/atmos/gfs.t00z.pgrb2.0p25.f012). .. DEBUG:CPLE_None in S3: Downloading 2965504-6897663 ( https://s3.amazonaws.com/noaa-gfs-bdp-pds/gfs.20250918/00/atmos/gfs.t00z.pgrb2.0p25.f012). .. [...] DEBUG:S3: Downloading 449626112-450461695 ( https://s3.amazonaws.com/noaa-gfs-bdp-pds/gfs.20250918/00/atmos/gfs.t00z.pgrb2.0p25.f012). .. Band 636 is listed in the .idx with offset 443333308, Band 637 having offset 444174665. The total filesize is 545533166. Do I need to do something extra to trigger GDAL to read only the requested band based on the .idx? Are some GRIB/.idx files not able to be loaded in this way? I am running via rasterio v1.4.3 which is using GDAL v3.9.3. My code is below, the file is in a public NOAA-hosted bucket. Cheers, Daniel ### import logging import rasterio logging.basicConfig(format="%(levelname)s:%(message)s", level=logging.DEBUG) with rasterio.Env(USE_IDX=True, AWS_VIRTUAL_HOSTING=False, CPL_DEBUG=True, CPL_CURL_VERBOSE=True): with rasterio.open("s3://noaa-gfs-bdp-pds/gfs.20250918/00/atmos/gfs.t00z.pgrb2.0p25.f012") as ds: band = ds.read(636) ### -------------- next part -------------- An HTML attachment was scrubbed... URL: From dbaston at gmail.com Thu Oct 9 08:36:17 2025 From: dbaston at gmail.com (Daniel Baston) Date: Thu, 9 Oct 2025 11:36:17 -0400 Subject: [gdal-dev] GRIB file being scanned despite .idx being present In-Reply-To: References: Message-ID: FWIW, the following snippet is working with gdal master: from osgeo import gdal with gdal.config_options({"AWS_NO_SIGN_REQUEST":"True", "CPL_DEBUG":"True", "CPL_CURL_VERBOSE":"True"}): ds = gdal.Open("/vsis3/noaa-gfs-bdp-pds/gfs.20250918/00/atmos/gfs.t00z.pgrb2.0p25.f012") band = ds.GetRasterBand(636) x = band.ReadAsArray() print(x.mean()) Dan On Thu, Oct 9, 2025 at 10:27?AM Daniel Evans via gdal-dev < gdal-dev at lists.osgeo.org> wrote: > Hi all, > > I am attempting to read a single band from a NOAA GRIB2 file on S3, with > an associated .idx file. Reading the GRIB2 driver documentation, it is > stated that the existence of such an idx file allows a file to be opened > without reading all bands. > > However, looking at the CPL_CURL_VERBOSE=True logs, it appears that GDAL > is still paging through the file from the start until reaching the > requested band. > > GDAL identifies the existence of the .idx file: > > DEBUG:CPLE_None in GRIB: Reading inventories from sidecar file > /vsis3/noaa-gfs-bdp-pds/gfs.20250918/00/atmos/gfs.t00z.pgrb2.0p25.f012.idx > DEBUG:CPLE_None in S3: Downloading 0-41215 ( > https://s3.amazonaws.com/noaa-gfs-bdp-pds/gfs.20250918/00/atmos/gfs.t00z.pgrb2.0p25.f012.idx). > .. > > But it then appears to scan the file from the start until it has passed > the requested band: > > DEBUG:CPLE_None in S3: Downloading 16384-999423 ( > https://s3.amazonaws.com/noaa-gfs-bdp-pds/gfs.20250918/00/atmos/gfs.t00z.pgrb2.0p25.f012). > .. > DEBUG:CPLE_None in S3: Downloading 999424-2965503 ( > https://s3.amazonaws.com/noaa-gfs-bdp-pds/gfs.20250918/00/atmos/gfs.t00z.pgrb2.0p25.f012). > .. > DEBUG:CPLE_None in S3: Downloading 2965504-6897663 ( > https://s3.amazonaws.com/noaa-gfs-bdp-pds/gfs.20250918/00/atmos/gfs.t00z.pgrb2.0p25.f012). > .. > [...] > DEBUG:S3: Downloading 449626112-450461695 ( > https://s3.amazonaws.com/noaa-gfs-bdp-pds/gfs.20250918/00/atmos/gfs.t00z.pgrb2.0p25.f012). > .. > > Band 636 is listed in the .idx with offset 443333308, Band 637 having > offset 444174665. The total filesize is 545533166. > > > Do I need to do something extra to trigger GDAL to read only the requested > band based on the .idx? Are some GRIB/.idx files not able to be loaded in > this way? > > I am running via rasterio v1.4.3 which is using GDAL v3.9.3. My code is > below, the file is in a public NOAA-hosted bucket. > > Cheers, > Daniel > > ### > > import logging > import rasterio > > logging.basicConfig(format="%(levelname)s:%(message)s", > level=logging.DEBUG) > > with rasterio.Env(USE_IDX=True, AWS_VIRTUAL_HOSTING=False, CPL_DEBUG=True, > CPL_CURL_VERBOSE=True): > with > rasterio.open("s3://noaa-gfs-bdp-pds/gfs.20250918/00/atmos/gfs.t00z.pgrb2.0p25.f012") > as ds: > band = ds.read(636) > > ### > > _______________________________________________ > gdal-dev mailing list > gdal-dev at lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdsumner at gmail.com Thu Oct 9 16:58:07 2025 From: mdsumner at gmail.com (Michael Sumner) Date: Fri, 10 Oct 2025 10:58:07 +1100 Subject: [gdal-dev] mdim: BLOCKSIZE creation syntax is not array-specific? Message-ID: Hi, emailing in case I'm missing something about how BLOCKSIZE is applied to an output ZARR. This source ZARR has 3 arrays: lat[20], lon[40], var[20,40]. There are 2 messages when it iterates the "lon" and "lat" arrays because the BLOCKSIZE syntax has the wrong number of elements for a 1D array: gdal mdim convert gdrivers/data/zarr/array_dimensions.zarr -o /vsimem/out.zarr --overwrite --creation-option ARRAY:BLOCKSIZE=20,40 ERROR 1: Invalid number of values in BLOCKSIZE ERROR 1: Invalid number of values in BLOCKSIZE Here there are 3 messages because the number of elements is wrong for all arrays: gdal mdim convert gdrivers/data/zarr/array_dimensions.zarr -o /vsimem/out.zarr --overwrite --creation-option ARRAY:BLOCKSIZE=20,40,5 ERROR 1: Invalid number of values in BLOCKSIZE ERROR 1: Invalid number of values in BLOCKSIZE ERROR 1: Invalid number of values in BLOCKSIZE And here, only one because it's wrong for "var" alone: gdal mdim convert gdrivers/data/zarr/array_dimensions.zarr -o /vsimem/out.zarr --overwrite --creation-option ARRAY:BLOCKSIZE=20 0...10...20...30ERROR 1: Invalid number of values in BLOCKSIZE Blocksize creation option should be bound to a dimension specifically? Cheers, Mike -- Michael Sumner Ordinary Member, Streets People Love Hobart Association Research Software Engineer Australian Antarctic Division Hobart, Australia 0438489030 e-mail: mdsumner at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From even.rouault at spatialys.com Thu Oct 9 17:29:53 2025 From: even.rouault at spatialys.com (Even Rouault) Date: Fri, 10 Oct 2025 02:29:53 +0200 Subject: [gdal-dev] mdim: BLOCKSIZE creation syntax is not array-specific? In-Reply-To: References: Message-ID: <82a90246-38aa-4f85-ad63-e36a5d7e76ce@spatialys.com> Hi Michael, > Blocksize creation option should be bound to a dimension specifically? yes, you can do that with something like: ARRAY:IF(DIM=2):BLOCKSIZE=256,256 See https://gdal.org/en/stable/api/gdalgroup_cpp.html#_CPPv4N9GDALGroup8CopyFromERKNSt10shared_ptrI9GDALGroupEEP11GDALDatasetRKNSt10shared_ptrI9GDALGroupEEbR7GUInt64K7GUInt6416GDALProgressFuncPv12CSLConstList Even > > Cheers, Mike > > > > -- > Michael Sumner > Ordinary Member,? Streets People Love Hobart Association > Research Software Engineer > Australian Antarctic Division > Hobart, Australia > 0438489030 > e-mail: mdsumner at gmail.com > > _______________________________________________ > gdal-dev mailing list > gdal-dev at lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev -- http://www.spatialys.com My software is free, but my time generally not. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdsumner at gmail.com Fri Oct 10 01:45:51 2025 From: mdsumner at gmail.com (Michael Sumner) Date: Fri, 10 Oct 2025 19:45:51 +1100 Subject: [gdal-dev] mdim: BLOCKSIZE creation syntax is not array-specific? In-Reply-To: <82a90246-38aa-4f85-ad63-e36a5d7e76ce@spatialys.com> References: <82a90246-38aa-4f85-ad63-e36a5d7e76ce@spatialys.com> Message-ID: Ah, very good, exactly what I need. Thanks Even. On Fri, Oct 10, 2025 at 11:29?AM Even Rouault wrote: > Hi Michael, > > Blocksize creation option should be bound to a dimension specifically? > > yes, you can do that with something like: ARRAY:IF(DIM=2):BLOCKSIZE=256,256 > > See > https://gdal.org/en/stable/api/gdalgroup_cpp.html#_CPPv4N9GDALGroup8CopyFromERKNSt10shared_ptrI9GDALGroupEEP11GDALDatasetRKNSt10shared_ptrI9GDALGroupEEbR7GUInt64K7GUInt6416GDALProgressFuncPv12CSLConstList > > Even > > > Cheers, Mike > > > > -- > Michael Sumner > Ordinary Member, Streets People Love Hobart Association > Research Software Engineer > Australian Antarctic Division > Hobart, Australia > 0438489030 > e-mail: mdsumner at gmail.com > > _______________________________________________ > gdal-dev mailing listgdal-dev at lists.osgeo.orghttps://lists.osgeo.org/mailman/listinfo/gdal-dev > > -- http://www.spatialys.com > My software is free, but my time generally not. > > -- Michael Sumner Ordinary Member, Streets People Love Hobart Association Research Software Engineer Australian Antarctic Division Hobart, Australia 0438489030 e-mail: mdsumner at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.fred.evans at gmail.com Fri Oct 10 03:47:02 2025 From: daniel.fred.evans at gmail.com (Daniel Evans) Date: Fri, 10 Oct 2025 11:47:02 +0100 Subject: [gdal-dev] GRIB file being scanned despite .idx being present In-Reply-To: References: Message-ID: Hmm, yes - I see it jumping straight to the relevant band when run via a locally compiled GDAL 3.11.4 using your code, but when using rasterio built on top of that same GDAL 3.11.4, it's paging through the whole file. Seems like there's something with how my Python environment/code configures things when using rasterio, or something that rasterio configures, that is modifying the behaviour. Thoughts on where to look welcome, but it doesn't appear to be a GDAL-level problem. Cheers, Daniel On Thu, 9 Oct 2025 at 16:36, Daniel Baston wrote: > FWIW, the following snippet is working with gdal master: > > from osgeo import gdal > > with gdal.config_options({"AWS_NO_SIGN_REQUEST":"True", > "CPL_DEBUG":"True", "CPL_CURL_VERBOSE":"True"}): > ds = > gdal.Open("/vsis3/noaa-gfs-bdp-pds/gfs.20250918/00/atmos/gfs.t00z.pgrb2.0p25.f012") > band = ds.GetRasterBand(636) > x = band.ReadAsArray() > print(x.mean()) > > Dan > > On Thu, Oct 9, 2025 at 10:27?AM Daniel Evans via gdal-dev < > gdal-dev at lists.osgeo.org> wrote: > >> Hi all, >> >> I am attempting to read a single band from a NOAA GRIB2 file on S3, with >> an associated .idx file. Reading the GRIB2 driver documentation, it is >> stated that the existence of such an idx file allows a file to be opened >> without reading all bands. >> >> However, looking at the CPL_CURL_VERBOSE=True logs, it appears that GDAL >> is still paging through the file from the start until reaching the >> requested band. >> >> GDAL identifies the existence of the .idx file: >> >> DEBUG:CPLE_None in GRIB: Reading inventories from sidecar file >> /vsis3/noaa-gfs-bdp-pds/gfs.20250918/00/atmos/gfs.t00z.pgrb2.0p25.f012.idx >> DEBUG:CPLE_None in S3: Downloading 0-41215 ( >> https://s3.amazonaws.com/noaa-gfs-bdp-pds/gfs.20250918/00/atmos/gfs.t00z.pgrb2.0p25.f012.idx). >> .. >> >> But it then appears to scan the file from the start until it has passed >> the requested band: >> >> DEBUG:CPLE_None in S3: Downloading 16384-999423 ( >> https://s3.amazonaws.com/noaa-gfs-bdp-pds/gfs.20250918/00/atmos/gfs.t00z.pgrb2.0p25.f012). >> .. >> DEBUG:CPLE_None in S3: Downloading 999424-2965503 ( >> https://s3.amazonaws.com/noaa-gfs-bdp-pds/gfs.20250918/00/atmos/gfs.t00z.pgrb2.0p25.f012). >> .. >> DEBUG:CPLE_None in S3: Downloading 2965504-6897663 ( >> https://s3.amazonaws.com/noaa-gfs-bdp-pds/gfs.20250918/00/atmos/gfs.t00z.pgrb2.0p25.f012). >> .. >> [...] >> DEBUG:S3: Downloading 449626112-450461695 ( >> https://s3.amazonaws.com/noaa-gfs-bdp-pds/gfs.20250918/00/atmos/gfs.t00z.pgrb2.0p25.f012). >> .. >> >> Band 636 is listed in the .idx with offset 443333308, Band 637 having >> offset 444174665. The total filesize is 545533166. >> >> >> Do I need to do something extra to trigger GDAL to read only the >> requested band based on the .idx? Are some GRIB/.idx files not able to be >> loaded in this way? >> >> I am running via rasterio v1.4.3 which is using GDAL v3.9.3. My code is >> below, the file is in a public NOAA-hosted bucket. >> >> Cheers, >> Daniel >> >> ### >> >> import logging >> import rasterio >> >> logging.basicConfig(format="%(levelname)s:%(message)s", >> level=logging.DEBUG) >> >> with rasterio.Env(USE_IDX=True, AWS_VIRTUAL_HOSTING=False, >> CPL_DEBUG=True, CPL_CURL_VERBOSE=True): >> with >> rasterio.open("s3://noaa-gfs-bdp-pds/gfs.20250918/00/atmos/gfs.t00z.pgrb2.0p25.f012") >> as ds: >> band = ds.read(636) >> >> ### >> >> _______________________________________________ >> gdal-dev mailing list >> gdal-dev at lists.osgeo.org >> https://lists.osgeo.org/mailman/listinfo/gdal-dev >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.fred.evans at gmail.com Fri Oct 10 04:06:07 2025 From: daniel.fred.evans at gmail.com (Daniel Evans) Date: Fri, 10 Oct 2025 12:06:07 +0100 Subject: [gdal-dev] GRIB file being scanned despite .idx being present In-Reply-To: References: Message-ID: To close this out on the GDAL side - from the rasterio logging and the source code emitting the log lines, it appears this is occurring due to rasterio eagerly determining the `nodata` for all bands even if only one is requested, which is what is then forcing GDAL to have to read through most of the file. I'll continue chasing on the rasterio side separately. Cheers, Daniel On Fri, 10 Oct 2025 at 11:47, Daniel Evans wrote: > Hmm, yes - I see it jumping straight to the relevant band when run via a > locally compiled GDAL 3.11.4 using your code, but when using rasterio built > on top of that same GDAL 3.11.4, it's paging through the whole file. Seems > like there's something with how my Python environment/code configures > things when using rasterio, or something that rasterio configures, that is > modifying the behaviour. > > Thoughts on where to look welcome, but it doesn't appear to be a > GDAL-level problem. > > Cheers, > Daniel > > On Thu, 9 Oct 2025 at 16:36, Daniel Baston wrote: > >> FWIW, the following snippet is working with gdal master: >> >> from osgeo import gdal >> >> with gdal.config_options({"AWS_NO_SIGN_REQUEST":"True", >> "CPL_DEBUG":"True", "CPL_CURL_VERBOSE":"True"}): >> ds = >> gdal.Open("/vsis3/noaa-gfs-bdp-pds/gfs.20250918/00/atmos/gfs.t00z.pgrb2.0p25.f012") >> band = ds.GetRasterBand(636) >> x = band.ReadAsArray() >> print(x.mean()) >> >> Dan >> >> On Thu, Oct 9, 2025 at 10:27?AM Daniel Evans via gdal-dev < >> gdal-dev at lists.osgeo.org> wrote: >> >>> Hi all, >>> >>> I am attempting to read a single band from a NOAA GRIB2 file on S3, with >>> an associated .idx file. Reading the GRIB2 driver documentation, it is >>> stated that the existence of such an idx file allows a file to be opened >>> without reading all bands. >>> >>> However, looking at the CPL_CURL_VERBOSE=True logs, it appears that GDAL >>> is still paging through the file from the start until reaching the >>> requested band. >>> >>> GDAL identifies the existence of the .idx file: >>> >>> DEBUG:CPLE_None in GRIB: Reading inventories from sidecar file >>> /vsis3/noaa-gfs-bdp-pds/gfs.20250918/00/atmos/gfs.t00z.pgrb2.0p25.f012.idx >>> DEBUG:CPLE_None in S3: Downloading 0-41215 ( >>> https://s3.amazonaws.com/noaa-gfs-bdp-pds/gfs.20250918/00/atmos/gfs.t00z.pgrb2.0p25.f012.idx). >>> .. >>> >>> But it then appears to scan the file from the start until it has passed >>> the requested band: >>> >>> DEBUG:CPLE_None in S3: Downloading 16384-999423 ( >>> https://s3.amazonaws.com/noaa-gfs-bdp-pds/gfs.20250918/00/atmos/gfs.t00z.pgrb2.0p25.f012). >>> .. >>> DEBUG:CPLE_None in S3: Downloading 999424-2965503 ( >>> https://s3.amazonaws.com/noaa-gfs-bdp-pds/gfs.20250918/00/atmos/gfs.t00z.pgrb2.0p25.f012). >>> .. >>> DEBUG:CPLE_None in S3: Downloading 2965504-6897663 ( >>> https://s3.amazonaws.com/noaa-gfs-bdp-pds/gfs.20250918/00/atmos/gfs.t00z.pgrb2.0p25.f012). >>> .. >>> [...] >>> DEBUG:S3: Downloading 449626112-450461695 ( >>> https://s3.amazonaws.com/noaa-gfs-bdp-pds/gfs.20250918/00/atmos/gfs.t00z.pgrb2.0p25.f012). >>> .. >>> >>> Band 636 is listed in the .idx with offset 443333308, Band 637 having >>> offset 444174665. The total filesize is 545533166. >>> >>> >>> Do I need to do something extra to trigger GDAL to read only the >>> requested band based on the .idx? Are some GRIB/.idx files not able to be >>> loaded in this way? >>> >>> I am running via rasterio v1.4.3 which is using GDAL v3.9.3. My code is >>> below, the file is in a public NOAA-hosted bucket. >>> >>> Cheers, >>> Daniel >>> >>> ### >>> >>> import logging >>> import rasterio >>> >>> logging.basicConfig(format="%(levelname)s:%(message)s", >>> level=logging.DEBUG) >>> >>> with rasterio.Env(USE_IDX=True, AWS_VIRTUAL_HOSTING=False, >>> CPL_DEBUG=True, CPL_CURL_VERBOSE=True): >>> with >>> rasterio.open("s3://noaa-gfs-bdp-pds/gfs.20250918/00/atmos/gfs.t00z.pgrb2.0p25.f012") >>> as ds: >>> band = ds.read(636) >>> >>> ### >>> >>> _______________________________________________ >>> gdal-dev mailing list >>> gdal-dev at lists.osgeo.org >>> https://lists.osgeo.org/mailman/listinfo/gdal-dev >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: