[gdal-dev] libpng and autotest
Jack Howarth
howarth at bromo.med.uc.edu
Fri Nov 8 12:31:05 PST 2013
On Fri, Nov 08, 2013 at 08:24:51PM +0100, Even Rouault wrote:
> Le vendredi 08 novembre 2013 18:50:45, Jack Howarth a écrit :
> > On Fri, Nov 08, 2013 at 09:12:05AM -0800, Kurt Schwehr wrote:
> > > Jack,
> > >
> > > Is there an explanation someplace with more detail as to what is wrong
> > > those png's, why older versions of libpng are okay with it and why
> > > libpng has changed its behavior?
> > >
> > > Thanks,
> > > -kurt
> >
> > Kurt,
> > The following link...
> >
> > http://comments.gmane.org/gmane.comp.graphics.png.devel/6017
> >
> > appears to discuss the issue we are seeing with the test.png file...
> >
> > > Libpng-1.6.0 through 1.6.2 use the CMF bytes at the beginning of an zlib
> > > datastream to set the window size for decoding. Libpng-1.5.x and
> > > earlier ignored the CMF bytes and always used a 32kbyte window.
> > > Libpng-1.6.x has uncovered the fact that there are hundreds of files in
> > > the wild (found in linux distributions such as gentoo) with a too-small
> > > CMF, which results in a "Too far back" error when decoding these PNGs.
> > > I have not yet been able to discover what application is producing
> > > these.
> >
> > If you read completely through the gmane.comp.graphics.png.devel/6017
> > thread, you should be at least convinced that such corruption is a known
> > issue.
>
> This is strange. I can't reproduce any issue on Linux with libpng 1.6.2 or
> libpng 1.6.3 with gdalicon.png or autotest/gdrivers/data/test.png that is used
> by png_1 test. If it matters, the zlib is still my system zlib 1.2.3.
Another thread discussing the same issue is at...
http://lilypond.1069038.n5.nabble.com/fixing-PNG-images-td146225.html
as well as here...
http://www.opendevs.org/qkup/libpng16-will-not-read-some-broken-png-images.pdf
I can see the problem with the pngfix from libpng 1.6.6 as follows...
# pngfix data/test.png
IDAT TFB default 10 15 8575 160400 data/test.png
whereas when I repair the file with optipng, I see...
% optipng -nb -nc -np -fix test.png
OptiPNG 0.6.3: Advanced PNG optimizer.
Copyright (C) 2001-2009 Cosmin Truta.
** Processing: test.png
400x400 pixels, 8 bits/pixel, 16 colors (16 transparent) in palette
Input IDAT size = 8575 bytes
Input file size = 8732 bytes
Trying:
zc = 9 zm = 8 zs = 0 f = 0 IDAT size = 7349
Selecting parameters:
zc = 9 zm = 8 zs = 0 f = 0 IDAT size = 7349
Output IDAT size = 7349 bytes (1226 bytes decrease)
Output file size = 7494 bytes (1238 bytes = 14.18% decrease)
# pngfix data/test.png
IDAT OK maximum 15 15 7349 160400 data/test.png
which is explained in the usage output from pngfix with no arguments as...
The summary lines describe issues encountered with the zlib compressed
stream of a chunk. They have the following format, which is SUBJECT TO
CHANGE in the future:
chunk reason comp-level p1 p2 p3 p4 file
p1 through p4 vary according to the 'reason'. There are always 8 space
separated fields. Reasons specific formats are:
chunk ERR status code read-errno write-errno message file
chunk SKP comp-level file-bits zlib-rc compressed message file
chunk ??? comp-level file-bits ok-bits compressed uncompress file
The various fields are
$1 chunk: The chunk type of a chunk in the file or 'HEAD' if a problem
is reported by libpng at the start of the IDAT stream.
$2 reason: One of:
CHK: A zlib header checksum was detected and fixed.
TFB: The zlib too far back error was detected and fixed.
OK : No errors were detected in the zlib stream and optimization
was not requested, or was not possible.
OPT: The zlib stream window bits value could be improved (and was).
SKP: The chunk was skipped because of a zlib issue (zlib-rc) with
explanation 'message'
ERR: The read of the file was aborted. The parameters explain why.
$3 status: For 'ERR' the accumulate status code from 'EXIT CODES' above.
This is printed as a 2 digit hexadecimal value
comp-level: The recorded compression level (FLEVEL) of a zlib stream
expressed as a string {supfast,stdfast,default,maximum}
$4 code: The file exit code; where stop was called, as a fairly terse
string {warning,libpng,zlib,invalid,read,write,unexpected}.
file-bits: The zlib window bits recorded in the file.
$5 read-errno: A system errno value from a read translated by strerror(3).
zlib-rc: A zlib return code as a string (see zlib.h).
ok-bits: The smallest zlib window bits value that works.
$6 write-errno:A system errno value from a write translated by strerror(3).
compressed: The count of compressed bytes in the zlib stream, when the
reason is 'SKP'; this is a count of the bytes read from the
stream when the fatal error was encountered.
$7 message: An error message (spaces replaced by _, as in all parameters),
uncompress: The count of bytes from uncompressing the zlib stream; this
may not be the same as the number of bytes in the image.
$8 file: The name of the file (this may contain spaces).
Note that the current test.png that you are shipping is reported as TFB
for "The zlib too far back error was detected and fixed" by pngfix whereas
the copy repaired by optipng reports OK for "No errors were detected in the zlib
stream and optimization was not requested, or was not possible.". So far
I haven't been able to puzzle out how to get pngfix to repair the file but
I notice most are using optipng for that.
Jack
>
>
> $ ldd apps/gdalinfo | grep png
> libpng16.so.16 => /home/even/install-libpng-1.6.2/lib/libpng16.so.16
> (0x00007fda95e39000)
>
> $ gdalinfo data/gdalicon.png -checksum
> Driver: PNG/Portable Network Graphics
> Files: data/gdalicon.png
> Size is 32, 32
> Coordinate System is `'
> Image Structure Metadata:
> INTERLEAVE=PIXEL
> Corner Coordinates:
> Upper Left ( 0.0, 0.0)
> Lower Left ( 0.0, 32.0)
> Upper Right ( 32.0, 0.0)
> Lower Right ( 32.0, 32.0)
> Center ( 16.0, 16.0)
> Band 1 Block=32x1 Type=Byte, ColorInterp=Red
> Checksum=7617
> Mask Flags: PER_DATASET ALPHA
> Band 2 Block=32x1 Type=Byte, ColorInterp=Green
> Checksum=7808
> Mask Flags: PER_DATASET ALPHA
> Band 3 Block=32x1 Type=Byte, ColorInterp=Blue
> Checksum=7377
> Mask Flags: PER_DATASET ALPHA
> Band 4 Block=32x1 Type=Byte, ColorInterp=Alpha
> Checksum=7926
>
> $ gdalinfo ../autotest/gdrivers/data/test.png -checksum
> Driver: PNG/Portable Network Graphics
> Files: ../autotest/gdrivers/data/test.png
> ../autotest/gdrivers/data/test.wld
> Size is 400, 400
> Coordinate System is `'
> GeoTransform =
> 700000.3050000001, 0.38, 0.01
> 4287500.695, -0.01, -0.38
> Corner Coordinates:
> Upper Left ( 700000.305, 4287500.695)
> Lower Left ( 700004.305, 4287348.695)
> Upper Right ( 700152.305, 4287496.695)
> Lower Right ( 700156.305, 4287344.695)
> Center ( 700078.305, 4287422.695)
> Band 1 Block=400x1 Type=Byte, ColorInterp=Palette
> Checksum=57921
> NoData Value=0
> Color Table (RGB with 16 entries)
> 0: 255,255,255,0
> 1: 255,255,208,255
> 2: 255,255,204,255
> 3: 153,204,255,255
> 4: 0,153,255,255
> 5: 102,102,102,255
> 6: 153,153,153,255
> 7: 150,150,150,255
> 8: 0,0,0,255
> 9: 100,160,210,255
> 10: 255,255,255,255
> 11: 226,226,226,255
> 12: 51,51,51,255
> 13: 0,0,0,255
> 14: 0,0,0,255
> 15: 0,0,0,255
>
>
>
> > Jack
> >
> > > On Nov 8, 2013, at 8:02 AM, Jack Howarth <howarth at bromo.med.uc.edu> wrote:
> > > > On Fri, Nov 08, 2013 at 10:31:08AM -0500, Jack Howarth wrote:
> > > >> On Thu, Nov 07, 2013 at 04:34:07PM -0500, Jack Howarth wrote:
> > > >>> The failures on x86_64-apple-darwin13 for the png.py test suite from
> > > >>> auto test for gdal 0.10.1 are the same failures described here...
> > > >>>
> > > >>> https://github.com/licq-im/licq/pull/32
> > > >>> <https://github.com/licq-im/licq/pull/32>
> > > >>>
> > > >>> The failures are completely suppressed by regenerating
> > > >>> autotest/gdrivers/data/test.png with
> > > >>> "optipng -nb -nc -np test.png"...
> > > >>
> > > >> Just to clarify, I used optipng 0.6.3 for this. The newer optipng
> > > >> 0.7.4 release seems more problematic as none of the obvious
> > > >> combination of options can fix the test.png file without changing the
> > > >> checksums and causing additional failures. So you will want to use
> > > >> the optipng 0.6.x release series to fix the bad png files in gdal.
> > > >
> > > > Also, my tests here using optipng 0.7.4 to detect damaged png files
> > > > indicate that only data/gdalicon.png in gdal-0.10.1 needs to be fixed.
> > > >
> > > > optipng -nb -nc -np -fix data/gdalicon.png
> > > > ** Processing: data/gdalicon.png
> > > > Warning:
> > > > 32x32 pixels, 4x8 bits/pixel, RGB+alpha
> > > > Recoverable errors found in input. Fixing...
> > > > Input IDAT size = 1964 bytes
> > > > Input file size = 2021 bytes
> > > >
> > > > Trying:
> > > > zc = 9 zm = 8 zs = 0 f = 0 IDAT size = 240
> > > >
> > > > Selecting parameters:
> > > > zc = 9 zm = 8 zs = 0 f = 0 IDAT size = 240
> > > >
> > > > Output IDAT size = 240 bytes (1724 bytes decrease)
> > > > Output file size = 313 bytes (1708 bytes = 84.51% decrease)
> > > >
> > > > ** Status report
> > > > 1 file(s) have been processed.
> > > > 1 error(s) have been encountered.
> > > > 1 erroneous file(s) have been fixed.
> > > >
> > > >>> % ./png.py
> > > >>>
> > > >>> TEST: png_1 ... success
> > > >>> TEST: png_2 ... success
> > > >>> TEST: png_3 ... success
> > > >>> TEST: png_4 ... success
> > > >>> TEST: png_5 ... success
> > > >>> TEST: png_6 ... success
> > > >>> TEST: png_7 ... success
> > > >>> TEST: png_8 ... success
> > > >>> TEST: png_9 ... success
> > > >>> TEST: png_10 ... success
> > > >>> TEST: png_11 ... success
> > > >>>
> > > >>> Test Script: png
> > > >>> Succeeded: 11
> > > >>> Failed: 0 (0 blew exceptions)
> > > >>> Skipped: 0
> > > >>> Expected fail:0
> > > >>> Duration: 0.08s
> > > >>>
> > > >>> _______________________________________________
> > > >>> gdal-dev mailing list
> > > >>> gdal-dev at lists.osgeo.org
> > > >>> http://lists.osgeo.org/mailman/listinfo/gdal-dev
> > > >>
> > > >> _______________________________________________
> > > >> gdal-dev mailing list
> > > >> gdal-dev at lists.osgeo.org
> > > >> http://lists.osgeo.org/mailman/listinfo/gdal-dev
> > > >
> > > > _______________________________________________
> > > > gdal-dev mailing list
> > > > gdal-dev at lists.osgeo.org
> > > > http://lists.osgeo.org/mailman/listinfo/gdal-dev
> >
> > _______________________________________________
> > gdal-dev mailing list
> > gdal-dev at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/gdal-dev
>
> --
> Geospatial professional services
> http://even.rouault.free.fr/services.html
More information about the gdal-dev
mailing list