[gdal-dev] libpng and autotest

Jack Howarth howarth at bromo.med.uc.edu
Fri Nov 8 12:46:12 PST 2013 

On Fri, Nov 08, 2013 at 03:31:05PM -0500, Jack Howarth wrote:
> On Fri, Nov 08, 2013 at 08:24:51PM +0100, Even Rouault wrote:
> > Le vendredi 08 novembre 2013 18:50:45, Jack Howarth a écrit :
> > > On Fri, Nov 08, 2013 at 09:12:05AM -0800, Kurt Schwehr wrote:
> > > > Jack,
> > > > 
> > > > Is there an explanation someplace with more detail as to what is wrong
> > > > those png's, why older versions of libpng are okay with it and why
> > > > libpng has changed its behavior?
> > > > 
> > > > Thanks,
> > > > -kurt
> > > 
> > > Kurt,
> > >    The following link...
> > > 
> > > http://comments.gmane.org/gmane.comp.graphics.png.devel/6017
> > > 
> > > appears to discuss the issue we are seeing with the test.png file...
> > > 
> > > >  Libpng-1.6.0 through 1.6.2 use the CMF bytes at the beginning of an zlib
> > > >  datastream to set the window size for decoding. Libpng-1.5.x and
> > > >  earlier ignored the CMF bytes and always used a 32kbyte window. 
> > > >  Libpng-1.6.x has uncovered the fact that there are hundreds of files in
> > > >  the wild (found in linux distributions such as gentoo) with a too-small
> > > >  CMF, which results in a "Too far back" error when decoding these PNGs. 
> > > >  I have not yet been able to discover what application is producing
> > > >  these.
> > > 
> > > If you read completely through the gmane.comp.graphics.png.devel/6017
> > > thread, you should be at least convinced that such corruption is a known
> > > issue.
> > 
> > This is strange. I can't reproduce any issue on Linux with libpng 1.6.2 or 
> > libpng 1.6.3 with gdalicon.png or autotest/gdrivers/data/test.png that is used 
> > by png_1 test. If it matters, the zlib is still my system zlib 1.2.3. 
> 
> Another thread discussing the same issue is at...
> 
> http://lilypond.1069038.n5.nabble.com/fixing-PNG-images-td146225.html
> 
> as well as here...
> 
> http://www.opendevs.org/qkup/libpng16-will-not-read-some-broken-png-images.pdf
> 
> I can see the problem with the pngfix from libpng 1.6.6 as follows...
> 
> # pngfix data/test.png
> IDAT TFB default 10 15 8575 160400 data/test.png
> 
> whereas when I repair the file with optipng, I see...
> 
> % optipng -nb -nc -np -fix test.png
> OptiPNG 0.6.3: Advanced PNG optimizer.
> Copyright (C) 2001-2009 Cosmin Truta.
> 
> ** Processing: test.png
> 400x400 pixels, 8 bits/pixel, 16 colors (16 transparent) in palette
> Input IDAT size = 8575 bytes
> Input file size = 8732 bytes
> 
> Trying:
>   zc = 9  zm = 8  zs = 0  f = 0		IDAT size = 7349
>                                
> Selecting parameters:
>   zc = 9  zm = 8  zs = 0  f = 0		IDAT size = 7349
> 
> Output IDAT size = 7349 bytes (1226 bytes decrease)
> Output file size = 7494 bytes (1238 bytes = 14.18% decrease)
> 
> # pngfix data/test.png
> IDAT OK  maximum 15 15 7349 160400 data/test.png
> 
> which is explained in the usage output from pngfix with no arguments as...
> 
>   The summary lines describe issues encountered with the zlib compressed
>   stream of a chunk.  They have the following format, which is SUBJECT TO
>   CHANGE in the future:
> 
>      chunk reason comp-level p1 p2 p3 p4 file
> 
>   p1 through p4 vary according to the 'reason'.  There are always 8 space
>   separated fields.  Reasons specific formats are:
> 
>      chunk ERR status code read-errno write-errno message file
>      chunk SKP comp-level file-bits zlib-rc compressed message file
>      chunk ??? comp-level file-bits ok-bits compressed uncompress file
> 
>   The various fields are
> 
> $1 chunk:      The chunk type of a chunk in the file or 'HEAD' if a problem
>                is reported by libpng at the start of the IDAT stream.
> $2 reason:     One of:
>           CHK: A zlib header checksum was detected and fixed.
>           TFB: The zlib too far back error was detected and fixed.
>           OK : No errors were detected in the zlib stream and optimization
>                was not requested, or was not possible.
>           OPT: The zlib stream window bits value could be improved (and was).
>           SKP: The chunk was skipped because of a zlib issue (zlib-rc) with
>                explanation 'message'
>           ERR: The read of the file was aborted.  The parameters explain why.
> $3 status:     For 'ERR' the accumulate status code from 'EXIT CODES' above.
>                This is printed as a 2 digit hexadecimal value
>    comp-level: The recorded compression level (FLEVEL) of a zlib stream
>                expressed as a string {supfast,stdfast,default,maximum}
> $4 code:       The file exit code; where stop was called, as a fairly terse
>                string {warning,libpng,zlib,invalid,read,write,unexpected}.
>    file-bits:  The zlib window bits recorded in the file.
> $5 read-errno: A system errno value from a read translated by strerror(3).
>    zlib-rc:    A zlib return code as a string (see zlib.h).
>    ok-bits:    The smallest zlib window bits value that works.
> $6 write-errno:A system errno value from a write translated by strerror(3).
>    compressed: The count of compressed bytes in the zlib stream, when the
>                reason is 'SKP'; this is a count of the bytes read from the
>                stream when the fatal error was encountered.
> $7 message:    An error message (spaces replaced by _, as in all parameters),
>    uncompress: The count of bytes from uncompressing the zlib stream; this
>                may not be the same as the number of bytes in the image.
> $8 file:       The name of the file (this may contain spaces).
> 
> Note that the current test.png that you are shipping is reported as TFB 
> for "The zlib too far back error was detected and fixed" by pngfix whereas
> the copy repaired by optipng reports OK for "No errors were detected in the zlib 
> stream and optimization was not requested, or was not possible.". So far
> I haven't been able to puzzle out how to get pngfix to repair the file but
> I notice most are using optipng for that.
>                      Jack 
> 

One last note. I reinstalled the older libpng16-1.6.3-1 packaging on fink and
it reports the same exact error for the bundled test.png file as the 1.6.6
release does...

% /sw/bin/pngfix test.png
IDAT TFB default 10 15 8575 160400 test.png.sav

So you should be able to find this problem with libpng 1.6.3 or later. Note
that the detection of this issue was added during the libpng 1.6.3 development
cycle so you need to be sure use its pngfix utility and not that from 1.6.2.

----------------------------------------------------------------------------
Changes since the last public release (1.6.2):
...

Added png-fix-itxt and png-fix-too-far-back to the built programs and
 removed warnings from the source code and timepng that are revealed as
 a result.
Detect wrong libpng versions linked to png-fix-too-far-back, which currently
only works with libpng versions that can be made to reliably fail when
the deflate data contains an out-of-window reference.  This means only
1.6 and later.
...
Attempt to detect configuration issues with png-fix-too-far-back, which
  requires both the correct libpng and the correct zlib to function
  correctly.
---------------------------------------------------------------------------

Note on darwin we typically use the system libz which appears to be
derivation of 1.2.5.

> > 
> > 
> > $ ldd apps/gdalinfo | grep png
> > 	libpng16.so.16 => /home/even/install-libpng-1.6.2/lib/libpng16.so.16 
> > (0x00007fda95e39000)
> > 
> > $ gdalinfo data/gdalicon.png -checksum
> > Driver: PNG/Portable Network Graphics
> > Files: data/gdalicon.png
> > Size is 32, 32
> > Coordinate System is `'
> > Image Structure Metadata:
> >   INTERLEAVE=PIXEL
> > Corner Coordinates:
> > Upper Left  (    0.0,    0.0)
> > Lower Left  (    0.0,   32.0)
> > Upper Right (   32.0,    0.0)
> > Lower Right (   32.0,   32.0)
> > Center      (   16.0,   16.0)
> > Band 1 Block=32x1 Type=Byte, ColorInterp=Red
> >   Checksum=7617
> >   Mask Flags: PER_DATASET ALPHA 
> > Band 2 Block=32x1 Type=Byte, ColorInterp=Green
> >   Checksum=7808
> >   Mask Flags: PER_DATASET ALPHA 
> > Band 3 Block=32x1 Type=Byte, ColorInterp=Blue
> >   Checksum=7377
> >   Mask Flags: PER_DATASET ALPHA 
> > Band 4 Block=32x1 Type=Byte, ColorInterp=Alpha
> >   Checksum=7926
> > 
> > $ gdalinfo ../autotest/gdrivers/data/test.png -checksum
> > Driver: PNG/Portable Network Graphics
> > Files: ../autotest/gdrivers/data/test.png
> >        ../autotest/gdrivers/data/test.wld
> > Size is 400, 400
> > Coordinate System is `'
> > GeoTransform =
> >   700000.3050000001, 0.38, 0.01
> >   4287500.695, -0.01, -0.38
> > Corner Coordinates:
> > Upper Left  (  700000.305, 4287500.695) 
> > Lower Left  (  700004.305, 4287348.695) 
> > Upper Right (  700152.305, 4287496.695) 
> > Lower Right (  700156.305, 4287344.695) 
> > Center      (  700078.305, 4287422.695) 
> > Band 1 Block=400x1 Type=Byte, ColorInterp=Palette
> >   Checksum=57921
> >   NoData Value=0
> >   Color Table (RGB with 16 entries)
> >     0: 255,255,255,0
> >     1: 255,255,208,255
> >     2: 255,255,204,255
> >     3: 153,204,255,255
> >     4: 0,153,255,255
> >     5: 102,102,102,255
> >     6: 153,153,153,255
> >     7: 150,150,150,255
> >     8: 0,0,0,255
> >     9: 100,160,210,255
> >    10: 255,255,255,255
> >    11: 226,226,226,255
> >    12: 51,51,51,255
> >    13: 0,0,0,255
> >    14: 0,0,0,255
> >    15: 0,0,0,255
> > 
> > 
> > 
> > >           Jack
> > > 
> > > > On Nov 8, 2013, at 8:02 AM, Jack Howarth <howarth at bromo.med.uc.edu> wrote:
> > > > > On Fri, Nov 08, 2013 at 10:31:08AM -0500, Jack Howarth wrote:
> > > > >> On Thu, Nov 07, 2013 at 04:34:07PM -0500, Jack Howarth wrote:
> > > > >>> The failures on x86_64-apple-darwin13 for the png.py test suite from
> > > > >>> auto test for gdal 0.10.1 are the same failures described here...
> > > > >>> 
> > > > >>> https://github.com/licq-im/licq/pull/32
> > > > >>> <https://github.com/licq-im/licq/pull/32>
> > > > >>> 
> > > > >>> The failures are completely suppressed by regenerating
> > > > >>> autotest/gdrivers/data/test.png with
> > > > >>> "optipng -nb -nc -np test.png"...
> > > > >> 
> > > > >> Just to clarify, I used optipng 0.6.3 for this. The newer optipng
> > > > >> 0.7.4 release seems more problematic as none of the obvious
> > > > >> combination of options can fix the test.png file without changing the
> > > > >> checksums and causing additional failures. So you will want to use
> > > > >> the optipng 0.6.x release series to fix the bad png files in gdal.
> > > > > 
> > > > > Also, my tests here using optipng 0.7.4 to detect damaged png files
> > > > > indicate that only data/gdalicon.png in gdal-0.10.1 needs to be fixed.
> > > > > 
> > > > > optipng -nb -nc -np -fix data/gdalicon.png
> > > > > ** Processing: data/gdalicon.png
> > > > > Warning:
> > > > > 32x32 pixels, 4x8 bits/pixel, RGB+alpha
> > > > > Recoverable errors found in input. Fixing...
> > > > > Input IDAT size = 1964 bytes
> > > > > Input file size = 2021 bytes
> > > > > 
> > > > > Trying:
> > > > >  zc = 9  zm = 8  zs = 0  f = 0		IDAT size = 240
> > > > > 
> > > > > Selecting parameters:
> > > > >  zc = 9  zm = 8  zs = 0  f = 0		IDAT size = 240
> > > > > 
> > > > > Output IDAT size = 240 bytes (1724 bytes decrease)
> > > > > Output file size = 313 bytes (1708 bytes = 84.51% decrease)
> > > > > 
> > > > > ** Status report
> > > > > 1 file(s) have been processed.
> > > > > 1 error(s) have been encountered.
> > > > > 1 erroneous file(s) have been fixed.
> > > > > 
> > > > >>> % ./png.py
> > > > >>> 
> > > > >>>  TEST: png_1 ... success
> > > > >>>  TEST: png_2 ... success
> > > > >>>  TEST: png_3 ... success
> > > > >>>  TEST: png_4 ... success
> > > > >>>  TEST: png_5 ... success
> > > > >>>  TEST: png_6 ... success
> > > > >>>  TEST: png_7 ... success
> > > > >>>  TEST: png_8 ... success
> > > > >>>  TEST: png_9 ... success
> > > > >>>  TEST: png_10 ... success
> > > > >>>  TEST: png_11 ... success
> > > > >>> 
> > > > >>> Test Script: png
> > > > >>> Succeeded: 11
> > > > >>> Failed:    0 (0 blew exceptions)
> > > > >>> Skipped:   0
> > > > >>> Expected fail:0
> > > > >>> Duration:  0.08s
> > > > >>> 
> > > > >>> _______________________________________________
> > > > >>> gdal-dev mailing list
> > > > >>> gdal-dev at lists.osgeo.org
> > > > >>> http://lists.osgeo.org/mailman/listinfo/gdal-dev
> > > > >> 
> > > > >> _______________________________________________
> > > > >> gdal-dev mailing list
> > > > >> gdal-dev at lists.osgeo.org
> > > > >> http://lists.osgeo.org/mailman/listinfo/gdal-dev
> > > > > 
> > > > > _______________________________________________
> > > > > gdal-dev mailing list
> > > > > gdal-dev at lists.osgeo.org
> > > > > http://lists.osgeo.org/mailman/listinfo/gdal-dev
> > > 
> > > _______________________________________________
> > > gdal-dev mailing list
> > > gdal-dev at lists.osgeo.org
> > > http://lists.osgeo.org/mailman/listinfo/gdal-dev
> > 
> > -- 
> > Geospatial professional services
> > http://even.rouault.free.fr/services.html
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev

More information about the gdal-dev mailing list