[gdal-dev] Troubleshooting palette management. Potential bug in palette for overviews?

Fri Nov 25 02:06:08 PST 2016

For future references (...mainly for me), I'm updating the previous links
using a tagged version.
Otherwise, the line numbers referring a trunk repository will point to some
comment or unrelated things :)

https://github.com/OSGeo/gdal/blob/tags/2.1.2/gdal/frmts/gtiff/geotiff.cpp#L4565
https://github.com/OSGeo/gdal/blob/tags/2.1.2/gdal/frmts/gtiff/geotiff.cpp#L8014
https://github.com/OSGeo/gdal/blob/tags/2.1.2/gdal/frmts/gtiff/geotiff.cpp#L8367

Daniele

On Thu, Nov 24, 2016 at 5:22 PM, Daniele Romagnoli <
daniele.romagnoli at geo-solutions.it> wrote:

>
>
> On Thu, Nov 24, 2016 at 5:01 PM, Even Rouault <even.rouault at spatialys.com>
> wrote:
>
>> On jeudi 24 novembre 2016 16:51:59 CET Daniele Romagnoli wrote:
>>
>> > Doh!
>>
>> > Thanks for the feedback, Even.
>>
>
>
>> (Indeed it seemed strange to me that nobody had ever noticed before)
>>
> Sorry, what I meant with that was that I didn't see that ticket before
> sending my email so it seemed strange to me that nobody noticed the palette
> math weirdness :)
>
>
>>
>>
>> Not that much regarding software using GDAL since if you don't explicitly
>> request the overview bands/datasets but rather use the subsampling
>> RasterIO() calls, you'll never work directly with the color table of the
>> overviews but still use the one of the main dataset (or even if you request
>> the overviews directly I guess in most cases applications don't bother
>> checking if the color table of the overview).
>>
>
> As far as I can see, the TIFFImageReader parses the metadata information
> based on imageIndex (so overview number). I will double check if things
> were working differently in the colormap topic before our
> changes/optimizations to this class.
>
>
>> In GDAL, overviews conceptually share the same palette than the main
>> image. Otherwise subsampling RasterIO() will return values of the overview
>> palette that will be interpreted with the palette of the main image. You
>> can imagine the fun visual effects if the palette are not the same... So
>> this is assumed, but not checked.
>>
>>
>>
>> Could be worth fixing that if that cause issues (apparently in your use
>> case at least), but I'd be interested by information of what is the
>> standard/recommanded/best-practice-in-the-industry/etc... formulas to
>> use for the mapping between the [0,255] and [0,65535] ranges.
>>
>>
>>
>
> Cheers,
> Daniele
>
>> >
>>
>> > Best Regards,
>>
>> > Daniele
>>
>> >
>>
>> > On Thu, Nov 24, 2016 at 1:35 PM, Even Rouault <
>> even.rouault at spatialys.com>
>>
>> >
>>
>> > wrote:
>>
>> > > Daniele,
>>
>> > >
>>
>> > >
>>
>> > >
>>
>> > > This is indeed an inconsistency I spotted a long time ago :
>>
>> > >
>>
>> > > https://trac.osgeo.org/gdal/ticket/2213
>>
>> > >
>>
>> > >
>>
>> > >
>>
>> > > Nobody was apparently sufficiently annoyed by this to fix it.
>> Actually I
>>
>> > > see I proposed a patch. Was probably waiting for feedback. I'm not
>> sure if
>>
>> > > there's a standardized way of converting between the [0,255] and
>> [0,65535]
>>
>> > > ranges.
>>
>> > >
>>
>> > >
>>
>> > >
>>
>> > > Even
>>
>> > >
>>
>> > > > Hi List,
>>
>> > > >
>>
>> > > > I'm investigating on a issue I have with a paletted image with
>>
>> > > > overviews,
>>
>> > > >
>>
>> > > > using java ImageIO tiff reader (wait... I know on this list we
>> discuss
>>
>> > > >
>>
>> > > > about GDAL topics :) ).
>>
>> > > >
>>
>> > > > I'm trying to better understand palette interpretation and palette
>>
>> > >
>>
>> > > storing
>>
>> > >
>>
>> > > > on tags and how they are handled by GDAL.
>>
>> > > >
>>
>> > > >
>>
>> > > >
>>
>> > > > From TIFF specifications, colormap entries are stored as short
>> values.
>>
>> > > >
>>
>> > > > I have a sample file with this color tab (as reported by gdalinfo).
>>
>> > > >
>>
>> > > >
>>
>> > > >
>>
>> > > > Color Table (RGB with 256 entries)
>>
>> > > >
>>
>> > > > 0: *164*,164,164,255
>>
>> > > >
>>
>> > > > 1: *255*,255,255,255
>>
>> > > >
>>
>> > > > 2: 0,0,0,255
>>
>> > > >
>>
>> > > > 3: 0,0,0,255
>>
>> > > >
>>
>> > > > .... repeat 0,0,0,255 to the end.
>>
>> > > >
>>
>> > > >
>>
>> > > >
>>
>> > > > Using an Hex editor or AsTiffTagViewer, I can see the byte values
>> of the
>>
>> > > >
>>
>> > > > colormap as stored on the file.
>>
>> > > >
>>
>> > > > For the first 4 mappings of the colortable the Red components are:
>>
>> > > >
>>
>> > > > *A4 A4* *FF FF* 00 00 00 00
>>
>> > > >
>>
>> > > > (8 bytes = 4 shorts)
>>
>> > > >
>>
>> > > >
>>
>> > > >
>>
>> > > > As far as I have understood, this is related to that part of the
>>
>> > > >
>>
>> > > > specification (quoting it, section 5):
>>
>> > > >
>>
>> > > >
>>
>> > > >
>>
>> > > >
>>
>> > > >
>>
>> > > > *In a TIFF ColorMap, all the Red values come first, followed by the
>>
>> > > > Green
>>
>> > > >
>>
>> > > > values, then the Blue values. In the ColorMap, black is represented
>> by
>>
>> > > >
>>
>> > > > 0,0,0 and white is represented by 65535, 65535, 65535*
>>
>> > > >
>>
>> > > >
>>
>> > > >
>>
>> > > > So the second entry of the map, being white (255, 255, 255) has its
>> red
>>
>> > > >
>>
>> > > > value stored as 65535 = *FF FF *whilst the first color
>> (164,164,164) has
>>
>> > > >
>>
>> > > > its red value stored as 42148 = *A4 A4*
>>
>> > > >
>>
>> > > >
>>
>> > > >
>>
>> > > > I see this behavior is confirmed by the way the geotiff.cpp create
>> the
>>
>> > > >
>>
>> > > > colortable:
>>
>> > > >
>>
>> > > > https://github.com/OSGeo/gdal/blob/trunk/gdal/frmts/gtiff/
>>
>> > >
>>
>> > > geotiff.cpp#L5043
>>
>> > >
>>
>> > > > panTRed[iColor] = static_cast<unsigned short>(257 * sRGB.c1);
>>
>> > > >
>>
>> > > > panTGreen[iColor] = static_cast<unsigned short>(257 * sRGB.c2);
>>
>> > > >
>>
>> > > > panTBlue[iColor] = static_cast<unsigned short>(257 * sRGB.c3)
>>
>> > > >
>>
>> > > >
>>
>> > > >
>>
>> > > > Applying a color*257 multiplication allows to have its 8 bits color
>>
>> > > >
>>
>> > > > representation stored into a short where intensity goes from 0 to
>> 65535.
>>
>> > > >
>>
>> > > >
>>
>> > > >
>>
>> > > > The issue I have is that when using gdaladdo, the overviews are
>>
>> > > > generated
>>
>> > > >
>>
>> > > > with a different color table.
>>
>> > > >
>>
>> > > > So that the first 8 bytes of the red component stored in the
>> colormap,as
>>
>> > > >
>>
>> > > > reported by AsTiffTagViewer are:
>>
>> > > >
>>
>> > > > *A4* 00 *FF* 00 00 00 00 00
>>
>> > > >
>>
>> > > >
>>
>> > > >
>>
>> > > > When converting it back to a 0-255 color component with the formula
>>
>> > > >
>>
>> > > > R*255/65535 it become *163* instead of 164.
>>
>> > > >
>>
>> > > >
>>
>> > > >
>>
>> > > > I see in the GDAL code that both the IBuildOverviews and
>>
>> > > >
>>
>> > > > CreateOverviewsFromSrcOverviews call the CreateTIFFColorTable
>> function
>>
>> > > >
>>
>> > > > which multiplies the colorMapEntry by 256 instead of 257.
>>
>> > > >
>>
>> > > >
>>
>> > > >
>>
>> > > > https://github.com/OSGeo/gdal/blob/trunk/gdal/frmts/gtiff/
>>
>> > >
>>
>> > > geotiff.cpp#L9137
>>
>> > >
>>
>> > > > This may explain the *A4* 00 *FF* 00 entries in the overview with
>>
>> > > > respect
>>
>> > > >
>>
>> > > > to the *A4 A4 FF FF *entries in the main level.
>>
>> > > >
>>
>> > > >
>>
>> > > >
>>
>> > > > Do you have any feedback on this?
>>
>> > > >
>>
>> > > > Please, let me know what do you think about it.
>>
>> > > >
>>
>> > > >
>>
>> > > >
>>
>> > > > Best Regards,
>>
>> > > >
>>
>> > > > Daniele
>>
>> > >
>>
>> > > --
>>
>> > >
>>
>> > > Spatialys - Geospatial professional services
>>
>> > >
>>
>> > > http://www.spatialys.com
>>
>> > >
>>
>> > > _______________________________________________
>>
>> > > gdal-dev mailing list
>>
>> > > gdal-dev at lists.osgeo.org
>>
>> > > http://lists.osgeo.org/mailman/listinfo/gdal-dev
>>
>>
>>
>>
>>
>> --
>>
>> Spatialys - Geospatial professional services
>>
>> http://www.spatialys.com
>>
>> _______________________________________________
>> gdal-dev mailing list
>> gdal-dev at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>>
>
>
>
> --
> ==
> GeoServer Professional Services from the experts! Visit
> http://goo.gl/it488V for more information.
> ==
>
> Ing. Daniele Romagnoli
> Senior Software Engineer
>
> GeoSolutions S.A.S.
> Via di Montramito 3/A
> 55054  Massarosa (LU)
> Italy
> phone: +39 0584 962313
> fax:      +39 0584 1660272
>
> http://www.geo-solutions.it
> http://twitter.com/geosolutions_it
>
> -------------------------------------------------------
>
> *AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*
>
> Le informazioni contenute in questo messaggio di posta elettronica e/o
> nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
> loro utilizzo è consentito esclusivamente al destinatario del messaggio,
> per le finalità indicate nel messaggio stesso. Qualora riceviate questo
> messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
> darcene notizia via e-mail e di procedere alla distruzione del messaggio
> stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
> divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
> utilizzarlo per finalità diverse, costituisce comportamento contrario ai
> principi dettati dal D.Lgs. 196/2003.
>
>
>
> The information in this message and/or attachments, is intended solely for
> the attention and use of the named addressee(s) and may be confidential or
> proprietary in nature or covered by the provisions of privacy act
> (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
> Code).Any use not in accord with its purpose, any disclosure, reproduction,
> copying, distribution, or either dissemination, either whole or partial, is
> strictly forbidden except previous formal approval of the named
> addressee(s). If you are not the intended recipient, please contact
> immediately the sender by telephone, fax or e-mail and delete the
> information in this message that has been received in error. The sender
> does not give any warranty or accept liability as the content, accuracy or
> completeness of sent messages and accepts no responsibility  for changes
> made after they were sent or for other risks which arise as a result of
> e-mail transmission, viruses, etc.
>
>

-- 
==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.
==

Ing. Daniele Romagnoli
Senior Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax:      +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility  for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20161125/9deee5b7/attachment-0001.html>