[mapserver-dev] Corrupt MapServer Zip outputs

Even Rouault even.rouault at spatialys.com
Tue Nov 23 08:22:25 PST 2021 

Hi Seth,
> I tried with both of these in the Mapfile, and also setting in the 
> command window (SET CPL_CREATE_ZIP64=YES) but still getting a corrupt zip:
>
> CONFIG "CPL_CREATE_ZIP64" "NO"
> CONFIG "CPL_CREATE_ZIP64" "YES"
>
> Is there any ogr command to test the low level zip funcion in GDAL?
No
> I thought "ogr2ogr foo.shp.zip something" command was a higher level 
> one which meant it might not be using the same approach.
I believe it should be representative enough of how MapServer uses GDAL 
zip API.
>
> I tried both the Windows unzip, 7-zip (x64) and opening on Linux with 
> Ark and unzip all with the same errors.
>
> The strangest thing is that saving via IIS through the browser is 
> producing a valid zip using the same OUTPUTFORMAT..
> I'm going to set up the msautotest and roads example in IIS to test if 
> that works.
> Could it be related to a change in the Windows cmd shell?

Maybe? But as I mentioned before, I'd thought this would rather be some 
msIO_needBinaryStdout() call missing. Perhaps IIS already sets stdout to 
binary mode ?

For the sake of testing, what if you add a call to 
msIO_needBinaryStdout() in mapserv.c, let's say at 
https://github.com/MapServer/MapServer/blob/main/mapserv.c#L243 . I'm 
not saying this is necessarily the right fix

Even

>
> Seth
>
> --
> web:http://geographika.co.uk
> twitter: @geographika
>
>
> On Sun, Nov 21, 2021, at 4:54 PM, Even Rouault wrote:
>>
>>
>> Le 21/11/2021 à 16:43, Seth G a écrit :
>>> Steve - in my case WMS are all fine, it is only related to zipped 
>>> outputs using OUTPUTFORMAT blocks.
>>>
>>> I was trying to find a working commit for the bisect, but went back 
>>> to 7-2 without finding one.
>>>
>>> I built MapServer main, and the latest 7-6, 7-4, and 7-2 branches 
>>> and ran the msautotest:
>>>
>>> cd /D D:\GitHub\mapserver\msautotest\wxs
>>> mapserv -nh 
>>> "QUERY_STRING=map=wfs_ogr.map&service=WFS&REQUEST=GetFeature&TYPENAME=road&version=2.0.0&outputformat=shapezip" 
>>> > output.zip
>>>
>>> All zips were invalid. However the same command does work with the 
>>> https://www.gisinternals.com/query.html?content=filelist&file=release-1911-x64-gdal-2-4-4-mapserver-7-4-3.zip 
>>> <https://www.gisinternals.com/query.html?content=filelist&file=release-1911-x64-gdal-2-4-4-mapserver-7-4-3.zip> 
>>> build. It looks like something changed between GDAL 2.4 and 3.3
>>>
>>> Likely function is CPLCreateZip (as this is used by MapServer as 
>>> mentioned by Even in the other thread), found in: 
>>> https://github.com/OSGeo/gdal/commits/v3.3.3/gdal/port/cpl_minizip_zip.cpp 
>>> <https://github.com/OSGeo/gdal/commits/v3.3.3/gdal/port/cpl_minizip_zip.cpp>
>>>
>>> There are only a couple of commits between versions that look like 
>>> they might be relevant?
>>>
>>> https://github.com/OSGeo/gdal/commit/0e159e1ba8ee9e7b4a56bd0b53d16de0acf89485#diff-b156d979e3531903e620bc0616c120584c8e28945247f50b29e81c5a7a49df0b 
>>> <https://github.com/OSGeo/gdal/commit/0e159e1ba8ee9e7b4a56bd0b53d16de0acf89485#diff-b156d979e3531903e620bc0616c120584c8e28945247f50b29e81c5a7a49df0b>
>>>
>>> https://github.com/OSGeo/gdal/commit/edcdc5b0a57b4fab159b14180141ee42a0408830#diff-b156d979e3531903e620bc0616c120584c8e28945247f50b29e81c5a7a49df0b 
>>> <https://github.com/OSGeo/gdal/commit/edcdc5b0a57b4fab159b14180141ee42a0408830#diff-b156d979e3531903e620bc0616c120584c8e28945247f50b29e81c5a7a49df0b>
>>>
>>> Is there a test / output zip file in the GDAL tests that could be 
>>> used to verify?
>>
>> Not necessarily generated by recent GDAL versions, and if they were, 
>> most certainly from Linux builds
>>
>> What is weird is that you reported that .shp.zip files generated by 
>> recent gisinternals version with "ogr2ogr foo.shp.zip something" were 
>> correct.
>>
>> Hum actually I see that .shp.zip are generated with the 
>> CPL_CREATE_ZIP64 environment variable set to NO (the default is YES)
>>
>> Can you try that ?
>>
>> It might be that your .zip reader aren't Zip64 ready, or that GDAL 
>> write them in a slightly invalid way
>>
>>> I have a working and invalid zip from msautotest if that helps.
>> yes, could be useful (but I suspect this is a difference between 
>> regular ZIP vs Zip64 one)
>>
>>>
>>> Seth
>>>
>>>
>>>
>>> --
>>> web:http://geographika.co.uk <http://geographika.co.uk>
>>> twitter: @geographika
>>>
>>>
>>> On Sun, Nov 21, 2021, at 3:51 PM, Steve Lime wrote:
>>>> This seems related...
>>>> https://github.com/MapServer/MapServer/issues/6412 
>>>> <https://github.com/MapServer/MapServer/issues/6412>
>>>>
>>>> WMS fails but shp2img works, as does WFS. Could it be WMS specific? 
>>>> Would be helpful to try mode=map and see if that works.
>>>>
>>>> On Sun, Nov 21, 2021 at 8:45 AM Seth G <sethg at geographika.co.uk 
>>>> <mailto:sethg at geographika.co.uk>> wrote:
>>>>
>>>>     Thanks Jérome. and Even. Sorry I should have checked for the
>>>>     headers in that zip first.
>>>>     I'll attempt the bisecting now!
>>>>
>>>>     --
>>>>     web:http://geographika.co.uk <http://geographika.co.uk>
>>>>     twitter: @geographika
>>>>
>>>>     On Sun, Nov 21, 2021, at 1:12 PM, Even Rouault wrote:
>>>>     > Seth,
>>>>     >
>>>>     > The .zip extension of this test file is a bit misleading, as
>>>>     the HTTP
>>>>     > headers are not stripped (testing HTTP headers is something
>>>>     good to test
>>>>     > in some tests). I've issued
>>>>     > https://github.com/MapServer/MapServer/pull/6440
>>>>     <https://github.com/MapServer/MapServer/pull/6440> to rename
>>>>     the expected
>>>>     > file though to avoid the confusion
>>>>     >
>>>>     > You can obtain a valid .zip file without headers with
>>>>     >
>>>>     > ../../build/mapserv -conf ../etc/mapserv.conf -nh
>>>>     >
>>>>     QUERY_STRING="map=wfs_ogr.map&SERVICE=WFS&VERSION=1.0.0&REQUEST=GetFeature&TYPENAME=road&featureid=road.1521&OUTPUTFORMAT=SHAPEZIP"
>>>>
>>>>     >  > out.zip
>>>>     >
>>>>     > Even
>>>>     >
>>>>     > Le 21/11/2021 à 12:32, Seth G a écrit :
>>>>     >> Hi all,
>>>>     >>
>>>>     >> Following on from the post at
>>>>     https://lists.osgeo.org/pipermail/mapserver-users/2021-November/082429.html
>>>>     <https://lists.osgeo.org/pipermail/mapserver-users/2021-November/082429.html>
>>>>     I've been trying to find where the cause of the corrupt zips
>>>>     may be.
>>>>     >>
>>>>     >> There is a msautotest to create zip files and check against
>>>>     the expected output, however the expected output also seems to
>>>>     be an invalid zip file - I tried on both Windows and Linux.
>>>>     >>
>>>>     >> Even stranger this is both in the main branch (simply
>>>>     download the file with the following URL to test), and also
>>>>     going back in the history to 2016.
>>>>     >>
>>>>     >>
>>>>     https://github.com/MapServer/MapServer/blob/main/msautotest/wxs/expected/wfsogr10_shapezip.zip
>>>>     <https://github.com/MapServer/MapServer/blob/main/msautotest/wxs/expected/wfsogr10_shapezip.zip>
>>>>     >>
>>>>     >> I had a working process using shapezips in a 7.6 release of
>>>>     MapServer so its hard to understand what could have changed,
>>>>     MapServer, GDAL, or zip formats themselves?
>>>>     >> I'll attempt a Git bisect but if the issue is outside
>>>>     MapServer it won't find an issue.
>>>>     >>
>>>>     >> If anyone is able to open the zip file successfully please
>>>>     let me know (tried with Ark and unzip on Linux and Windows
>>>>     inbuilt unzip and 7-zip on Windows). Error seems to be "87
>>>>     extra bytes at beginning or within zip file".
>>>>     >>
>>>>     >> Seth
>>>>     >>
>>>>     >> --
>>>>     >> web:http://geographika.co.uk <http://geographika.co.uk>
>>>>     >> twitter: @geographika
>>>>     >> _______________________________________________
>>>>     >> MapServer-dev mailing list
>>>>     >> MapServer-dev at lists.osgeo.org
>>>>     <mailto:MapServer-dev at lists.osgeo.org>
>>>>     >> https://lists.osgeo.org/mailman/listinfo/mapserver-dev
>>>>     <https://lists.osgeo.org/mailman/listinfo/mapserver-dev>
>>>>     >
>>>>     > --
>>>>     > http://www.spatialys.com <http://www.spatialys.com>
>>>>     > My software is free, but my time generally not.
>>>>     _______________________________________________
>>>>     MapServer-dev mailing list
>>>>     MapServer-dev at lists.osgeo.org
>>>>     <mailto:MapServer-dev at lists.osgeo.org>
>>>>     https://lists.osgeo.org/mailman/listinfo/mapserver-dev
>>>>     <https://lists.osgeo.org/mailman/listinfo/mapserver-dev>
>>>>
>>>
>> -- 
>> http://www.spatialys.com  <http://www.spatialys.com>
>> My software is free, but my time generally not.
>
-- 
http://www.spatialys.com
My software is free, but my time generally not.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-dev/attachments/20211123/cf45bf4c/attachment-0001.html>

More information about the MapServer-dev mailing list