[gdal-dev] Motion: adopt RFC 87: Signed int8 data type for raster

Thu Nov 17 03:59:42 PST 2022

I concur that signed byte data is fairly common and we have to handle it in our Python application. The recent addition of Int64 types actually broke the dictionary-based way we were converting between numpy and GDAL datatypes and we had to implement a more robust solution.

Is it the case that gdal.GetDataTypeName and gdal.GetDataTypeByName currently return the wrong values – they suggest np.int8 is equivalent to gdal.GDT_Byte when it should be np.uint8. Hopefully adding a signed 8bit type should clear this up. Having to also check the PIXELTYPE metadata isn’t always convenient so it will be better to have a more consistent conversion.

Jon

e: Jon.Morris at jbarisk.com
d:+44 (0)1756 587229
t: +44 (0)1756 799919
www.jbarisk.com
All JBA Risk Management's email messages contain confidential information and are intended only for the individual(s) named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail.
Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. 
JBA Risk Management Limited is registered in England, company number 07732946, 1 Broughton Park, Old Lane North, Broughton, Skipton, North Yorkshire, BD23 3FD, England.
From: gdal-dev <gdal-dev-bounces at lists.osgeo.org> On Behalf Of Idan Miara
Sent: 17 November 2022 08:47
To: Even Rouault <even.rouault at spatialys.com>
Cc: gdal-dev at lists.osgeo.org
Subject: Re: [gdal-dev] Motion: adopt RFC 87: Signed int8 data type for raster

CAUTION: This email originated from outside of JBA and contains one or more links. DO NOT click links unless you recognise the sender's email address and are absolutely certain that the content is safe.
See the Phishing page on IMS on SP for more information about how to spot and report suspicious messages.
WARNING: This email contains one or more attachments with links within the attachment(s). Please take extra care when handling the attachments.
Just adding my two cents...

I highly support adding the signed int8 datatype - this would probably lead to downstream projects that support different raster datatypes to eventually fully support int8 rasters as a first class citizen.
Yes - downstream projects would need to invest some time adapting but the result would be a PROPER support for int8 rasters.
 (I would imagine that searching the code for some key search words would yield most places that need fixing),
Nobody is forced to upgrade a major version and sometimes a downstream effort is required to support the evolution of the upstream project.

I was a maintainer of an app that used signed int8 data vastly and wanted to incorporate save/load/calc rasters using GDAL.
After spending countless hours of trying to use the signed byte flag properly in every case I ended up changing the logic of the application to use only unsigned byte rasters since there were too many cases where the signed byte flag was not honored or caused weird outputs (for instance, I think that QGIS didn't fully support int8 rasters properly).

Even, Thanks for investing the time to make this right,
Idan

On Wed, 16 Nov 2022 at 21:39, Even Rouault <even.rouault at spatialys.com<mailto:even.rouault at spatialys.com>> wrote:

I think long term compatibility is a very desirable feature, and several applications just use GDAL as part of their code base without even being aware of what is happening in the GDAL development front. The same is true for several other libraries, given the fact the use of package managers (VCPKG, conan etc) are becoming more common. For some code bases, GDAL is just a small cog, and I believe it is very easy for the developers to simply download the next version from the package manager thinking that they are getting mostly bug fixes and not being aware of new silent incompatibilities.

We try to avoid breakage or silent breakage when reasonable, but sometimes it is the best thing to do. People/software who depends on the past functionality will see the tests related to their code testing GetMetadataItem("PIXELTYPE", "IMAGE_STRUCTURE") == "SIGNEDBYTE" break and will probably figure out they should read GDAL release notes to understand what happens. It is not the first time nor the last one we will do that.

Sorry, but I think it is funny that you mention removing GDT_Byte would have breaking implications, since I am expressing my reservations against a change that will have breaking implications. :)
My perception is that > 90% of current uses of GDT_Byte are for the unsigned usage. Of course their might be some domains where this is not the case.

I don't understand what limitations you are referring to. Perhaps the support was broken to some types of raster files?
This is described in the RFC: " the absence of a proper data type means that it is easy to forget to test the PIXELTYPE=SIGNEDBYTE metadata item in all places where the value of pixels is taken into account. There were special cases for statistics computations, but most of the other code, such as the overview or warping computation code had no provision for it, and consequently mis-interpreted negative values in the range [-128,-1] as positive values in the range [128,255]." . So current support of signed 8-byte within GDAL was broken in a number of places, and the new GDT_Int8 data type enables to fix it. One could argue that the past behavior of reporting a metadata item that altered the semantics of GDT_Byte was a silent breakage of the API promise of GDT_Byte being unsigned.

In my field, signed 8-bit images representing categorical images are very common, given the fact they can be used fully decompressed and represent a null value/nodata in very convenient way (as a negative value).

You should have a better experience with GDAL 3.7 then, pending perhaps a few changes in your code.
Even

--

http://www.spatialys.com

My software is free, but my time generally not.
_______________________________________________
gdal-dev mailing list
gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org>
https://lists.osgeo.org/mailman/listinfo/gdal-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20221117/6d1cf802/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image553391.png
Type: image/png
Size: 9321 bytes
Desc: image553391.png
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20221117/6d1cf802/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image038714.png
Type: image/png
Size: 651 bytes
Desc: image038714.png
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20221117/6d1cf802/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image994012.png
Type: image/png
Size: 715 bytes
Desc: image994012.png
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20221117/6d1cf802/attachment-0007.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image270771.png
Type: image/png
Size: 966 bytes
Desc: image270771.png
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20221117/6d1cf802/attachment-0008.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image039784.png
Type: image/png
Size: 784 bytes
Desc: image039784.png
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20221117/6d1cf802/attachment-0009.png>