[mapserver-users] Corrupted PNG issue

Fri Nov 19 08:44:40 PST 2021

Yes apologies John for hijacking the thread, I thought it could be related to your issue but looking less and less likely.
Steve's suggestion to use the commandline mapserv -nh "QUERY..." approach should rule out a few more issues. 

Even - the same zip cannot be opened on Linux either (using Ark and unzip): "9487 extra bytes at begging or within zipfile". 
The GDAL within the build (3.3.3) creates a zip of the WFS just fine for the same layer/WFS service using ogr2ogr:

ogr2ogr -f "ESRI Shapefile" "H:/Temp/test.shp.zip" WFS:"https://server/mapserver/?service=WFS&REQUEST=GetFeature&version=2.0.0 <https://plmonaghandev.compass.ie/mapserver/?service=WFS&REQUEST=GetFeature&version=2.0.0>" LayerName -overwrite -skipfailures

GDAL 2.4.4 doesn't create zips so it must be MapServer doing the zipping in this version at least?
I'm guessing this points to a MapServer issue, so I'll carry on debugging.

Seth

--
web:http://geographika.co.uk
twitter: @geographika

On Fri, Nov 19, 2021, at 4:11 PM, Even Rouault wrote:
> Ah, GDAL also changed... Zipping is done by GDAL, so could be a GDAL related change (there were adjustments in the past regarding ZIP64 that could result in some bytes being different). Did you try opening the zip with another utility (like 'unzip' on Linux) ?
> 
> This might be then a completely different issue than the one reported by John.
> 
> Le 19/11/2021 à 16:01, Seth G a écrit :
>> Thanks Even, I'll give that a go (the bisect worked well for another issue a few months ago). 
>> 
>> I presume the zipping issue is more likely to be in the MapServer codebase than in GDAL?
>> The GISInternals did switch from GDAL 2.4.4 to GDAL 3.3.3 from the 7.4.3 to 7.6.4 branches.
>> 
>> The zip exports work fine over a web browser rather than command line which is strange. Maybe its related to modifications to the -nh switch. 
>> 
>> Seth
>> 
>> --
>> web:http://geographika.co.uk
>> twitter: @geographika
>> 
>> 
>> On Fri, Nov 19, 2021, at 2:07 PM, Even Rouault wrote:
>>> Could be a msIO_needBinaryStdout() missing somewhere. Seth, if you've the chance to run a git bisect session, that could probably give a strong hint of how to fix that.
>>> 
>>> Le 19/11/2021 à 09:43, Seth G a écrit :
>>>> Possibly unrelated but I ran into a similar issue exporting WFS to zipped shapefiles.
>>>> Working fine in 7-4-3 but broken in 7-6-4 (and current master) - the zip files are corrupt, although they have an identical size. 
>>>> 
>>>> 7-zip reports "Headers Error Unconfirmed start of archive"
>>>> 
>>>> The same command is used for both versions:
>>>> 
>>>> mapserv -nh "QUERY_STRING=map=my.map&service=WFS&REQUEST=GetFeature&TYPENAME=LayerName&version=2.0.0&outputformat=shapezip&srsName=EPSG:3857" > output.zip
>>>> 
>>>> I had a look with a hex editor but the start of the working and corrupt zips seem identical. 
>>>> 
>>>> On the other hand all PNGs / WMS services are fine. 
>>>> 
>>>> Seth
>>>> 
>>>> 
>>>> --
>>>> web:http://geographika.co.uk
>>>> twitter: @geographika
>>>> 
>>>> 
>>>> On Thu, Nov 18, 2021, at 3:53 PM, Steve Lime wrote:
>>>>> Hmmm... I've not run into or heard of this before although I'm not a windows user. I did a quick sanity check with a mapfile here and the latest 7.4 and 7.6 versions. While not exactly the same setup you have in terms of versions, they produce the exact same png image.
>>>>> 
>>>>> What do you get for output if you use mapserv.exe at the command line? So something like:
>>>>> 
>>>>>   mapserv.exe -nh "QUERY_STRING= map=dynamic\5708d96b-c606-4c35-95e7-085fedc1dcce.map&SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&FORMAT=image%2Fpng&TRANSPARENT=true&LAYERS=MAP&TILED=true&WIDTH=512&HEIGHT=512&CRS=EPSG%3A3857&STYLES=&BBOX=-10877294.873093722%2C5536486.832751887%2C-10876071.880641159%2C5537709.82520445" > test.png
>>>>> 
>>>>> That would take PostMan and the web server out of the picture. Is there any chance different versions of libpng are being used? What are you using to manage the tiles?
>>>>> 
>>>>> --Steve
>>>>> 
>>>>> On Wed, Nov 17, 2021 at 4:57 PM John Huotari via MapServer-users <mapserver-users at lists.osgeo.org> wrote:
>>>>>> I’m attempting to upgrade from MapServer 7.6.1 to 7.6.4 via compiled packages obtained from GISInternals.  I’m running it on Windows/IIS and whereas 7.6.1 was generating .PNG tiles perfectly for me, after upgrading to 7.6.4, the .PNGs being created appear to be corrupt.  I can replace the 7.6.4 exe and dlls with 7.6.1 versions and the PNG images generate fine again, so while there are quite a few places that could introduce an issue, with the exception of a change to MapServer everything would be identical in my stack between having the issue in 7.6.4 and not in 7.6.1.
>>>>>>  
>>>>>> The good headers from 7.6.1 look like this
>>>>>>  
>>>>>> 89 50 4e 47 0d 0a 1a 0a  00 00 00 0d 49 48 44 52
>>>>>>  
>>>>>> and the corrupted ones from 7.6.4 look like this
>>>>>>  
>>>>>> 89 50 4e 47 0d 0d 0a 1a 0d 0a 00 00 00 0d 49 48 44 52
>>>>>>  
>>>>>> It appears that the 0a values from the valid header are being converted to 0d 0a.  I might be wrong here, but it appears to me that something is interpreting the 0a as a line feed and given the code is running on Windows, is converting that LF into a CR LF.  The replacement doesn’t seem to be limited to the file header as I see the 7.6.4 version of the file is slightly larger (18,571 bytes instead of 18,407 bytes) and in spot checking, I’ve verified some additional 0d’s exist precede 0a within the data blocks of the .PNG.  Has anyone experienced anything like this or know of any fixes?
>>>>>>  
>>>>>> PNG Images produced on the server with shp2img are just fine, it’s only images produced by making a WMS request to mapserv.exe that have the issue.  An example WMS request would be
>>>>>>  
>>>>>> https://<Server Name Removed>/mapserv.exe?map=dynamic\5708d96b-c606-4c35-95e7-085fedc1dcce.map&SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&FORMAT=image%2Fpng&TRANSPARENT=true&LAYERS=MAP&TILED=true&WIDTH=512&HEIGHT=512&CRS=EPSG%3A3857&STYLES=&BBOX=-10877294.873093722%2C5536486.832751887%2C-10876071.880641159%2C5537709.82520445
>>>>>>  
>>>>>> Maybe I’m completely misdiagnosing the problem as these PNG image files just show up as corrupted within a browser – for example FireFox reports “The image <full URL here> cannot be displayed because it contains errors.”  The way I obtained the actual .PNG images to view in a binary editor was to use PostMan and save the body of the results.  Perhaps PostMan introduced the extra bytes when saving an unrecognizable format file to disk whereas it did not when saving a file it recognized as a valid PNG.  I can’t find anything different between the valid and invalid files beyond the extra 0d’s that have been added though, so I don’t think PostMan or anything else in the chain introduced them.
>>>>>> 
>>>>>> ******************* PLEASE NOTE *******************
>>>>>> This message, along with any attachments, is for the designated recipient(s) only and may contain privileged, proprietary, or otherwise confidential information. If this message has reached you in error, kindly destroy it without review and notify the sender immediately. Any other use of such misdirected e-mail by you is prohibited. Where allowed by local law, electronic communications with Zurich and its affiliates, including e-mail and instant messaging (including content), may be scanned for the purposes of information security and assessment of internal compliance with company policy.
>>>>>> _______________________________________________
>>>>>> MapServer-users mailing list
>>>>>> MapServer-users at lists.osgeo.org
>>>>>> https://lists.osgeo.org/mailman/listinfo/mapserver-users
>>>>> _______________________________________________
>>>>> MapServer-users mailing list
>>>>> MapServer-users at lists.osgeo.org
>>>>> https://lists.osgeo.org/mailman/listinfo/mapserver-users
>>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> MapServer-users mailing list
>>>> MapServer-users at lists.osgeo.org
>>>> https://lists.osgeo.org/mailman/listinfo/mapserver-users
>>>> 
>>>> 
>>> -- 
>>> http://www.spatialys.com
>>> My software is free, but my time generally not.
>>> 
>>> _______________________________________________
>>> MapServer-users mailing list
>>> MapServer-users at lists.osgeo.org
>>> https://lists.osgeo.org/mailman/listinfo/mapserver-users
>>> 
>> 
> -- 
> http://www.spatialys.com
> My software is free, but my time generally not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-users/attachments/20211119/4b6a0ad0/attachment.htm>