[mapserver-dev] Corrupt MapServer Zip outputs

Tue Nov 23 08:08:29 PST 2021

Hi Even,

I tried with both of these in the Mapfile, and also setting in the command window (SET CPL_CREATE_ZIP64=YES) but still getting a corrupt zip:

CONFIG "CPL_CREATE_ZIP64" "NO"
CONFIG "CPL_CREATE_ZIP64" "YES"

Is there any ogr command to test the low level zip funcion in GDAL?
I thought "ogr2ogr foo.shp.zip something" command was a higher level one which meant it might not be using the same approach. 

I tried both the Windows unzip, 7-zip (x64) and opening on Linux with Ark and unzip all with the same errors.

The strangest thing is that saving via IIS through the browser is producing a valid zip using the same OUTPUTFORMAT..
I'm going to set up the msautotest and roads example in IIS to test if that works. 
Could it be related to a change in the Windows cmd shell?

Seth

--
web:http://geographika.co.uk
twitter: @geographika

On Sun, Nov 21, 2021, at 4:54 PM, Even Rouault wrote:
> 
> 
> Le 21/11/2021 à 16:43, Seth G a écrit :
>> Steve - in my case WMS are all fine, it is only related to zipped outputs using OUTPUTFORMAT blocks. 
>> 
>> I was trying to find a working commit for the bisect, but went back to 7-2 without finding one. 
>> 
>> I built MapServer main, and the latest 7-6, 7-4, and 7-2 branches and ran the msautotest:
>> 
>> cd /D D:\GitHub\mapserver\msautotest\wxs
>> mapserv -nh "QUERY_STRING=map=wfs_ogr.map&service=WFS&REQUEST=GetFeature&TYPENAME=road&version=2.0.0&outputformat=shapezip" > output.zip
>> 
>> All zips were invalid. However the same command does work with the https://www.gisinternals.com/query.html?content=filelist&file=release-1911-x64-gdal-2-4-4-mapserver-7-4-3.zip build. It looks like something changed between GDAL 2.4 and 3.3
>> 
>> Likely function is CPLCreateZip (as this is used by MapServer as mentioned by Even in the other thread), found in: https://github.com/OSGeo/gdal/commits/v3.3.3/gdal/port/cpl_minizip_zip.cpp
>> 
>> There are only a couple of commits between versions that look like they might be relevant?
>> 
>> https://github.com/OSGeo/gdal/commit/0e159e1ba8ee9e7b4a56bd0b53d16de0acf89485#diff-b156d979e3531903e620bc0616c120584c8e28945247f50b29e81c5a7a49df0b
>> 
>> https://github.com/OSGeo/gdal/commit/edcdc5b0a57b4fab159b14180141ee42a0408830#diff-b156d979e3531903e620bc0616c120584c8e28945247f50b29e81c5a7a49df0b
>> 
>> Is there a test / output zip file in the GDAL tests that could be used to verify?
> Not necessarily generated by recent GDAL versions, and if they were, most certainly from Linux builds
> 
> What is weird is that you reported that .shp.zip files generated by recent gisinternals version with "ogr2ogr foo.shp.zip something" were correct. 
> 
> Hum actually I see that .shp.zip are generated with the CPL_CREATE_ZIP64 environment variable set to NO (the default is YES)
> 
> Can you try that ?
> 
> It might be that your .zip reader aren't Zip64 ready, or that GDAL write them in a slightly invalid way
> 
>> I have a working and invalid zip from msautotest if that helps. 
> yes, could be useful (but I suspect this is a difference between regular ZIP vs Zip64 one)
> 
>> 
>> Seth
>> 
>> 
>> 
>> --
>> web:http://geographika.co.uk
>> twitter: @geographika
>> 
>> 
>> On Sun, Nov 21, 2021, at 3:51 PM, Steve Lime wrote:
>>> This seems related... 
>>> https://github.com/MapServer/MapServer/issues/6412 
>>> 
>>> WMS fails but shp2img works, as does WFS. Could it be WMS specific? Would be helpful to try mode=map and see if that works.
>>> 
>>> On Sun, Nov 21, 2021 at 8:45 AM Seth G <sethg at geographika.co.uk> wrote:
>>>> Thanks Jérome. and Even. Sorry I should have checked for the headers in that zip first.
>>>> I'll attempt the bisecting now!
>>>> 
>>>> --
>>>> web:http://geographika.co.uk
>>>> twitter: @geographika
>>>> 
>>>> On Sun, Nov 21, 2021, at 1:12 PM, Even Rouault wrote:
>>>> > Seth,
>>>> >
>>>> > The .zip extension of this test file is a bit misleading, as the HTTP 
>>>> > headers are not stripped (testing HTTP headers is something good to test 
>>>> > in some tests). I've issued 
>>>> > https://github.com/MapServer/MapServer/pull/6440 to rename the expected 
>>>> > file though to avoid the confusion
>>>> >
>>>> > You can obtain a valid .zip file without headers with
>>>> >
>>>> > ../../build/mapserv -conf ../etc/mapserv.conf -nh 
>>>> > QUERY_STRING="map=wfs_ogr.map&SERVICE=WFS&VERSION=1.0.0&REQUEST=GetFeature&TYPENAME=road&featureid=road.1521&OUTPUTFORMAT=SHAPEZIP" 
>>>> >  > out.zip
>>>> >
>>>> > Even
>>>> >
>>>> > Le 21/11/2021 à 12:32, Seth G a écrit :
>>>> >> Hi all,
>>>> >>
>>>> >> Following on from the post at https://lists.osgeo.org/pipermail/mapserver-users/2021-November/082429.html I've been trying to find where the cause of the corrupt zips may be.
>>>> >>
>>>> >> There is a msautotest to create zip files and check against the expected output, however the expected output also seems to be an invalid zip file - I tried on both Windows and Linux.
>>>> >>
>>>> >> Even stranger this is both in the main branch (simply download the file with the following URL to test), and also going back in the history to 2016.
>>>> >>
>>>> >> https://github.com/MapServer/MapServer/blob/main/msautotest/wxs/expected/wfsogr10_shapezip.zip
>>>> >>
>>>> >> I had a working process using shapezips in a 7.6 release of MapServer so its hard to understand what could have changed, MapServer, GDAL, or zip formats themselves?
>>>> >> I'll attempt a Git bisect but if the issue is outside MapServer it won't find an issue.
>>>> >>
>>>> >> If anyone is able to open the zip file successfully please let me know (tried with Ark and unzip on Linux and Windows inbuilt unzip and 7-zip on Windows). Error seems to be "87 extra bytes at beginning or within zip file".
>>>> >>
>>>> >> Seth
>>>> >>
>>>> >> --
>>>> >> web:http://geographika.co.uk
>>>> >> twitter: @geographika
>>>> >> _______________________________________________
>>>> >> MapServer-dev mailing list
>>>> >> MapServer-dev at lists.osgeo.org
>>>> >> https://lists.osgeo.org/mailman/listinfo/mapserver-dev
>>>> >
>>>> > -- 
>>>> > http://www.spatialys.com
>>>> > My software is free, but my time generally not.
>>>> _______________________________________________
>>>> MapServer-dev mailing list
>>>> MapServer-dev at lists.osgeo.org
>>>> https://lists.osgeo.org/mailman/listinfo/mapserver-dev
>> 
> -- 
> http://www.spatialys.com
> My software is free, but my time generally not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-dev/attachments/20211123/b941b379/attachment.html>