[gdal-dev] libpng and autotest

Jack Howarth howarth at bromo.med.uc.edu
Fri Nov 8 13:01:37 PST 2013 

On Fri, Nov 08, 2013 at 03:46:12PM -0500, Jack Howarth wrote:
> On Fri, Nov 08, 2013 at 03:31:05PM -0500, Jack Howarth wrote:
> > On Fri, Nov 08, 2013 at 08:24:51PM +0100, Even Rouault wrote:
> > > Le vendredi 08 novembre 2013 18:50:45, Jack Howarth a écrit :
> > > > On Fri, Nov 08, 2013 at 09:12:05AM -0800, Kurt Schwehr wrote:
> > > > > Jack,
> > > > > 
> > > > > Is there an explanation someplace with more detail as to what is wrong
> > > > > those png's, why older versions of libpng are okay with it and why
> > > > > libpng has changed its behavior?
> > > > > 
> > > > > Thanks,
> > > > > -kurt
> > > > 
> > > > Kurt,
> > > >    The following link...
> > > > 
> > > > http://comments.gmane.org/gmane.comp.graphics.png.devel/6017
> > > > 
> > > > appears to discuss the issue we are seeing with the test.png file...
> > > > 
> > > > >  Libpng-1.6.0 through 1.6.2 use the CMF bytes at the beginning of an zlib
> > > > >  datastream to set the window size for decoding. Libpng-1.5.x and
> > > > >  earlier ignored the CMF bytes and always used a 32kbyte window. 
> > > > >  Libpng-1.6.x has uncovered the fact that there are hundreds of files in
> > > > >  the wild (found in linux distributions such as gentoo) with a too-small
> > > > >  CMF, which results in a "Too far back" error when decoding these PNGs. 
> > > > >  I have not yet been able to discover what application is producing
> > > > >  these.
> > > > 
> > > > If you read completely through the gmane.comp.graphics.png.devel/6017
> > > > thread, you should be at least convinced that such corruption is a known
> > > > issue.
> > > 
> > > This is strange. I can't reproduce any issue on Linux with libpng 1.6.2 or 
> > > libpng 1.6.3 with gdalicon.png or autotest/gdrivers/data/test.png that is used 
> > > by png_1 test. If it matters, the zlib is still my system zlib 1.2.3. 
> > 
> > Another thread discussing the same issue is at...
> > 
> > http://lilypond.1069038.n5.nabble.com/fixing-PNG-images-td146225.html
> > 
> > as well as here...
> > 
> > http://www.opendevs.org/qkup/libpng16-will-not-read-some-broken-png-images.pdf
> > 
> > I can see the problem with the pngfix from libpng 1.6.6 as follows...
> > 
> > # pngfix data/test.png
> > IDAT TFB default 10 15 8575 160400 data/test.png
> > 
> > whereas when I repair the file with optipng, I see...
> > 
> > % optipng -nb -nc -np -fix test.png
> > OptiPNG 0.6.3: Advanced PNG optimizer.
> > Copyright (C) 2001-2009 Cosmin Truta.
> > 
> > ** Processing: test.png
> > 400x400 pixels, 8 bits/pixel, 16 colors (16 transparent) in palette
> > Input IDAT size = 8575 bytes
> > Input file size = 8732 bytes
> > 
> > Trying:
> >   zc = 9  zm = 8  zs = 0  f = 0		IDAT size = 7349
> >                                
> > Selecting parameters:
> >   zc = 9  zm = 8  zs = 0  f = 0		IDAT size = 7349
> > 
> > Output IDAT size = 7349 bytes (1226 bytes decrease)
> > Output file size = 7494 bytes (1238 bytes = 14.18% decrease)
> > 
> > # pngfix data/test.png
> > IDAT OK  maximum 15 15 7349 160400 data/test.png
> > 
> > which is explained in the usage output from pngfix with no arguments as...
> > 
> >   The summary lines describe issues encountered with the zlib compressed
> >   stream of a chunk.  They have the following format, which is SUBJECT TO
> >   CHANGE in the future:
> > 
> >      chunk reason comp-level p1 p2 p3 p4 file
> > 
> >   p1 through p4 vary according to the 'reason'.  There are always 8 space
> >   separated fields.  Reasons specific formats are:
> > 
> >      chunk ERR status code read-errno write-errno message file
> >      chunk SKP comp-level file-bits zlib-rc compressed message file
> >      chunk ??? comp-level file-bits ok-bits compressed uncompress file
> > 
> >   The various fields are
> > 
> > $1 chunk:      The chunk type of a chunk in the file or 'HEAD' if a problem
> >                is reported by libpng at the start of the IDAT stream.
> > $2 reason:     One of:
> >           CHK: A zlib header checksum was detected and fixed.
> >           TFB: The zlib too far back error was detected and fixed.
> >           OK : No errors were detected in the zlib stream and optimization
> >                was not requested, or was not possible.
> >           OPT: The zlib stream window bits value could be improved (and was).
> >           SKP: The chunk was skipped because of a zlib issue (zlib-rc) with
> >                explanation 'message'
> >           ERR: The read of the file was aborted.  The parameters explain why.
> > $3 status:     For 'ERR' the accumulate status code from 'EXIT CODES' above.
> >                This is printed as a 2 digit hexadecimal value
> >    comp-level: The recorded compression level (FLEVEL) of a zlib stream
> >                expressed as a string {supfast,stdfast,default,maximum}
> > $4 code:       The file exit code; where stop was called, as a fairly terse
> >                string {warning,libpng,zlib,invalid,read,write,unexpected}.
> >    file-bits:  The zlib window bits recorded in the file.
> > $5 read-errno: A system errno value from a read translated by strerror(3).
> >    zlib-rc:    A zlib return code as a string (see zlib.h).
> >    ok-bits:    The smallest zlib window bits value that works.
> > $6 write-errno:A system errno value from a write translated by strerror(3).
> >    compressed: The count of compressed bytes in the zlib stream, when the
> >                reason is 'SKP'; this is a count of the bytes read from the
> >                stream when the fatal error was encountered.
> > $7 message:    An error message (spaces replaced by _, as in all parameters),
> >    uncompress: The count of bytes from uncompressing the zlib stream; this
> >                may not be the same as the number of bytes in the image.
> > $8 file:       The name of the file (this may contain spaces).
> > 
> > Note that the current test.png that you are shipping is reported as TFB 
> > for "The zlib too far back error was detected and fixed" by pngfix whereas
> > the copy repaired by optipng reports OK for "No errors were detected in the zlib 
> > stream and optimization was not requested, or was not possible.". So far
> > I haven't been able to puzzle out how to get pngfix to repair the file but
> > I notice most are using optipng for that.
> >                      Jack 
> > 
> 
> One last note. I reinstalled the older libpng16-1.6.3-1 packaging on fink and
> it reports the same exact error for the bundled test.png file as the 1.6.6
> release does...
> 
> % /sw/bin/pngfix test.png
> IDAT TFB default 10 15 8575 160400 test.png.sav
> 
> So you should be able to find this problem with libpng 1.6.3 or later. Note
> that the detection of this issue was added during the libpng 1.6.3 development
> cycle so you need to be sure use its pngfix utility and not that from 1.6.2.
> 
> ----------------------------------------------------------------------------
> Changes since the last public release (1.6.2):
> ...
> 
> Added png-fix-itxt and png-fix-too-far-back to the built programs and
>  removed warnings from the source code and timepng that are revealed as
>  a result.
> Detect wrong libpng versions linked to png-fix-too-far-back, which currently
> only works with libpng versions that can be made to reliably fail when
> the deflate data contains an out-of-window reference.  This means only
> 1.6 and later.
> ...
> Attempt to detect configuration issues with png-fix-too-far-back, which
>   requires both the correct libpng and the correct zlib to function
>   correctly.
> ---------------------------------------------------------------------------
> 
> Note on darwin we typically use the system libz which appears to be
> derivation of 1.2.5.

I can reproduce this output from pngfix on a Fedora 15 box as follows...

% tar -zxvf libpng-1.6.6.tar.gz
% cd libpng-1.6.6
% ./configure
% make
% cp ~/autotest/gdrivers/data/test.png .
% ./pngfix test.png
IDAT TFB default 10 15 8575 160400 test.png


> 
> > > 
> > > 
> > > $ ldd apps/gdalinfo | grep png
> > > 	libpng16.so.16 => /home/even/install-libpng-1.6.2/lib/libpng16.so.16 
> > > (0x00007fda95e39000)
> > > 
> > > $ gdalinfo data/gdalicon.png -checksum
> > > Driver: PNG/Portable Network Graphics
> > > Files: data/gdalicon.png
> > > Size is 32, 32
> > > Coordinate System is `'
> > > Image Structure Metadata:
> > >   INTERLEAVE=PIXEL
> > > Corner Coordinates:
> > > Upper Left  (    0.0,    0.0)
> > > Lower Left  (    0.0,   32.0)
> > > Upper Right (   32.0,    0.0)
> > > Lower Right (   32.0,   32.0)
> > > Center      (   16.0,   16.0)
> > > Band 1 Block=32x1 Type=Byte, ColorInterp=Red
> > >   Checksum=7617
> > >   Mask Flags: PER_DATASET ALPHA 
> > > Band 2 Block=32x1 Type=Byte, ColorInterp=Green
> > >   Checksum=7808
> > >   Mask Flags: PER_DATASET ALPHA 
> > > Band 3 Block=32x1 Type=Byte, ColorInterp=Blue
> > >   Checksum=7377
> > >   Mask Flags: PER_DATASET ALPHA 
> > > Band 4 Block=32x1 Type=Byte, ColorInterp=Alpha
> > >   Checksum=7926
> > > 
> > > $ gdalinfo ../autotest/gdrivers/data/test.png -checksum
> > > Driver: PNG/Portable Network Graphics
> > > Files: ../autotest/gdrivers/data/test.png
> > >        ../autotest/gdrivers/data/test.wld
> > > Size is 400, 400
> > > Coordinate System is `'
> > > GeoTransform =
> > >   700000.3050000001, 0.38, 0.01
> > >   4287500.695, -0.01, -0.38
> > > Corner Coordinates:
> > > Upper Left  (  700000.305, 4287500.695) 
> > > Lower Left  (  700004.305, 4287348.695) 
> > > Upper Right (  700152.305, 4287496.695) 
> > > Lower Right (  700156.305, 4287344.695) 
> > > Center      (  700078.305, 4287422.695) 
> > > Band 1 Block=400x1 Type=Byte, ColorInterp=Palette
> > >   Checksum=57921
> > >   NoData Value=0
> > >   Color Table (RGB with 16 entries)
> > >     0: 255,255,255,0
> > >     1: 255,255,208,255
> > >     2: 255,255,204,255
> > >     3: 153,204,255,255
> > >     4: 0,153,255,255
> > >     5: 102,102,102,255
> > >     6: 153,153,153,255
> > >     7: 150,150,150,255
> > >     8: 0,0,0,255
> > >     9: 100,160,210,255
> > >    10: 255,255,255,255
> > >    11: 226,226,226,255
> > >    12: 51,51,51,255
> > >    13: 0,0,0,255
> > >    14: 0,0,0,255
> > >    15: 0,0,0,255
> > > 
> > > 
> > > 
> > > >           Jack
> > > > 
> > > > > On Nov 8, 2013, at 8:02 AM, Jack Howarth <howarth at bromo.med.uc.edu> wrote:
> > > > > > On Fri, Nov 08, 2013 at 10:31:08AM -0500, Jack Howarth wrote:
> > > > > >> On Thu, Nov 07, 2013 at 04:34:07PM -0500, Jack Howarth wrote:
> > > > > >>> The failures on x86_64-apple-darwin13 for the png.py test suite from
> > > > > >>> auto test for gdal 0.10.1 are the same failures described here...
> > > > > >>> 
> > > > > >>> https://github.com/licq-im/licq/pull/32
> > > > > >>> <https://github.com/licq-im/licq/pull/32>
> > > > > >>> 
> > > > > >>> The failures are completely suppressed by regenerating
> > > > > >>> autotest/gdrivers/data/test.png with
> > > > > >>> "optipng -nb -nc -np test.png"...
> > > > > >> 
> > > > > >> Just to clarify, I used optipng 0.6.3 for this. The newer optipng
> > > > > >> 0.7.4 release seems more problematic as none of the obvious
> > > > > >> combination of options can fix the test.png file without changing the
> > > > > >> checksums and causing additional failures. So you will want to use
> > > > > >> the optipng 0.6.x release series to fix the bad png files in gdal.
> > > > > > 
> > > > > > Also, my tests here using optipng 0.7.4 to detect damaged png files
> > > > > > indicate that only data/gdalicon.png in gdal-0.10.1 needs to be fixed.
> > > > > > 
> > > > > > optipng -nb -nc -np -fix data/gdalicon.png
> > > > > > ** Processing: data/gdalicon.png
> > > > > > Warning:
> > > > > > 32x32 pixels, 4x8 bits/pixel, RGB+alpha
> > > > > > Recoverable errors found in input. Fixing...
> > > > > > Input IDAT size = 1964 bytes
> > > > > > Input file size = 2021 bytes
> > > > > > 
> > > > > > Trying:
> > > > > >  zc = 9  zm = 8  zs = 0  f = 0		IDAT size = 240
> > > > > > 
> > > > > > Selecting parameters:
> > > > > >  zc = 9  zm = 8  zs = 0  f = 0		IDAT size = 240
> > > > > > 
> > > > > > Output IDAT size = 240 bytes (1724 bytes decrease)
> > > > > > Output file size = 313 bytes (1708 bytes = 84.51% decrease)
> > > > > > 
> > > > > > ** Status report
> > > > > > 1 file(s) have been processed.
> > > > > > 1 error(s) have been encountered.
> > > > > > 1 erroneous file(s) have been fixed.
> > > > > > 
> > > > > >>> % ./png.py
> > > > > >>> 
> > > > > >>>  TEST: png_1 ... success
> > > > > >>>  TEST: png_2 ... success
> > > > > >>>  TEST: png_3 ... success
> > > > > >>>  TEST: png_4 ... success
> > > > > >>>  TEST: png_5 ... success
> > > > > >>>  TEST: png_6 ... success
> > > > > >>>  TEST: png_7 ... success
> > > > > >>>  TEST: png_8 ... success
> > > > > >>>  TEST: png_9 ... success
> > > > > >>>  TEST: png_10 ... success
> > > > > >>>  TEST: png_11 ... success
> > > > > >>> 
> > > > > >>> Test Script: png
> > > > > >>> Succeeded: 11
> > > > > >>> Failed:    0 (0 blew exceptions)
> > > > > >>> Skipped:   0
> > > > > >>> Expected fail:0
> > > > > >>> Duration:  0.08s
> > > > > >>> 
> > > > > >>> _______________________________________________
> > > > > >>> gdal-dev mailing list
> > > > > >>> gdal-dev at lists.osgeo.org
> > > > > >>> http://lists.osgeo.org/mailman/listinfo/gdal-dev
> > > > > >> 
> > > > > >> _______________________________________________
> > > > > >> gdal-dev mailing list
> > > > > >> gdal-dev at lists.osgeo.org
> > > > > >> http://lists.osgeo.org/mailman/listinfo/gdal-dev
> > > > > > 
> > > > > > _______________________________________________
> > > > > > gdal-dev mailing list
> > > > > > gdal-dev at lists.osgeo.org
> > > > > > http://lists.osgeo.org/mailman/listinfo/gdal-dev
> > > > 
> > > > _______________________________________________
> > > > gdal-dev mailing list
> > > > gdal-dev at lists.osgeo.org
> > > > http://lists.osgeo.org/mailman/listinfo/gdal-dev
> > > 
> > > -- 
> > > Geospatial professional services
> > > http://even.rouault.free.fr/services.html
> > _______________________________________________
> > gdal-dev mailing list
> > gdal-dev at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/gdal-dev
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev

More information about the gdal-dev mailing list