[mapserver-dev] Corrupt MapServer Zip outputs

Even Rouault even.rouault at spatialys.com
Sun Nov 21 07:54:39 PST 2021 

Le 21/11/2021 à 16:43, Seth G a écrit :
> Steve - in my case WMS are all fine, it is only related to zipped 
> outputs using OUTPUTFORMAT blocks.
>
> I was trying to find a working commit for the bisect, but went back to 
> 7-2 without finding one.
>
> I built MapServer main, and the latest 7-6, 7-4, and 7-2 branches and 
> ran the msautotest:
>
> cd /D D:\GitHub\mapserver\msautotest\wxs
> mapserv -nh 
> "QUERY_STRING=map=wfs_ogr.map&service=WFS&REQUEST=GetFeature&TYPENAME=road&version=2.0.0&outputformat=shapezip" 
> > output.zip
>
> All zips were invalid. However the same command does work with the 
> https://www.gisinternals.com/query.html?content=filelist&file=release-1911-x64-gdal-2-4-4-mapserver-7-4-3.zip 
> <https://www.gisinternals.com/query.html?content=filelist&file=release-1911-x64-gdal-2-4-4-mapserver-7-4-3.zip> 
> build. It looks like something changed between GDAL 2.4 and 3.3
>
> Likely function is CPLCreateZip (as this is used by MapServer as 
> mentioned by Even in the other thread), found in: 
> https://github.com/OSGeo/gdal/commits/v3.3.3/gdal/port/cpl_minizip_zip.cpp 
> <https://github.com/OSGeo/gdal/commits/v3.3.3/gdal/port/cpl_minizip_zip.cpp>
>
> There are only a couple of commits between versions that look like 
> they might be relevant?
>
> https://github.com/OSGeo/gdal/commit/0e159e1ba8ee9e7b4a56bd0b53d16de0acf89485#diff-b156d979e3531903e620bc0616c120584c8e28945247f50b29e81c5a7a49df0b 
> <https://github.com/OSGeo/gdal/commit/0e159e1ba8ee9e7b4a56bd0b53d16de0acf89485#diff-b156d979e3531903e620bc0616c120584c8e28945247f50b29e81c5a7a49df0b>
>
> https://github.com/OSGeo/gdal/commit/edcdc5b0a57b4fab159b14180141ee42a0408830#diff-b156d979e3531903e620bc0616c120584c8e28945247f50b29e81c5a7a49df0b 
> <https://github.com/OSGeo/gdal/commit/edcdc5b0a57b4fab159b14180141ee42a0408830#diff-b156d979e3531903e620bc0616c120584c8e28945247f50b29e81c5a7a49df0b>
>
> Is there a test / output zip file in the GDAL tests that could be used 
> to verify?

Not necessarily generated by recent GDAL versions, and if they were, 
most certainly from Linux builds

What is weird is that you reported that .shp.zip files generated by 
recent gisinternals version with "ogr2ogr foo.shp.zip something" were 
correct.

Hum actually I see that .shp.zip are generated with the CPL_CREATE_ZIP64 
environment variable set to NO (the default is YES)

Can you try that ?

It might be that your .zip reader aren't Zip64 ready, or that GDAL write 
them in a slightly invalid way

> I have a working and invalid zip from msautotest if that helps.
yes, could be useful (but I suspect this is a difference between regular 
ZIP vs Zip64 one)
>
> Seth
>
>
>
> --
> web:http://geographika.co.uk
> twitter: @geographika
>
>
> On Sun, Nov 21, 2021, at 3:51 PM, Steve Lime wrote:
>> This seems related...
>> https://github.com/MapServer/MapServer/issues/6412 
>> <https://github.com/MapServer/MapServer/issues/6412>
>>
>> WMS fails but shp2img works, as does WFS. Could it be WMS specific? 
>> Would be helpful to try mode=map and see if that works.
>>
>> On Sun, Nov 21, 2021 at 8:45 AM Seth G <sethg at geographika.co.uk 
>> <mailto:sethg at geographika.co.uk>> wrote:
>>
>>     Thanks Jérome. and Even. Sorry I should have checked for the
>>     headers in that zip first.
>>     I'll attempt the bisecting now!
>>
>>     --
>>     web:http://geographika.co.uk <http://geographika.co.uk>
>>     twitter: @geographika
>>
>>     On Sun, Nov 21, 2021, at 1:12 PM, Even Rouault wrote:
>>     > Seth,
>>     >
>>     > The .zip extension of this test file is a bit misleading, as
>>     the HTTP
>>     > headers are not stripped (testing HTTP headers is something
>>     good to test
>>     > in some tests). I've issued
>>     > https://github.com/MapServer/MapServer/pull/6440
>>     <https://github.com/MapServer/MapServer/pull/6440> to rename the
>>     expected
>>     > file though to avoid the confusion
>>     >
>>     > You can obtain a valid .zip file without headers with
>>     >
>>     > ../../build/mapserv -conf ../etc/mapserv.conf -nh
>>     >
>>     QUERY_STRING="map=wfs_ogr.map&SERVICE=WFS&VERSION=1.0.0&REQUEST=GetFeature&TYPENAME=road&featureid=road.1521&OUTPUTFORMAT=SHAPEZIP"
>>
>>     >  > out.zip
>>     >
>>     > Even
>>     >
>>     > Le 21/11/2021 à 12:32, Seth G a écrit :
>>     >> Hi all,
>>     >>
>>     >> Following on from the post at
>>     https://lists.osgeo.org/pipermail/mapserver-users/2021-November/082429.html
>>     <https://lists.osgeo.org/pipermail/mapserver-users/2021-November/082429.html>
>>     I've been trying to find where the cause of the corrupt zips may be.
>>     >>
>>     >> There is a msautotest to create zip files and check against
>>     the expected output, however the expected output also seems to be
>>     an invalid zip file - I tried on both Windows and Linux.
>>     >>
>>     >> Even stranger this is both in the main branch (simply download
>>     the file with the following URL to test), and also going back in
>>     the history to 2016.
>>     >>
>>     >>
>>     https://github.com/MapServer/MapServer/blob/main/msautotest/wxs/expected/wfsogr10_shapezip.zip
>>     <https://github.com/MapServer/MapServer/blob/main/msautotest/wxs/expected/wfsogr10_shapezip.zip>
>>     >>
>>     >> I had a working process using shapezips in a 7.6 release of
>>     MapServer so its hard to understand what could have changed,
>>     MapServer, GDAL, or zip formats themselves?
>>     >> I'll attempt a Git bisect but if the issue is outside
>>     MapServer it won't find an issue.
>>     >>
>>     >> If anyone is able to open the zip file successfully please let
>>     me know (tried with Ark and unzip on Linux and Windows inbuilt
>>     unzip and 7-zip on Windows). Error seems to be "87 extra bytes at
>>     beginning or within zip file".
>>     >>
>>     >> Seth
>>     >>
>>     >> --
>>     >> web:http://geographika.co.uk <http://geographika.co.uk>
>>     >> twitter: @geographika
>>     >> _______________________________________________
>>     >> MapServer-dev mailing list
>>     >> MapServer-dev at lists.osgeo.org
>>     <mailto:MapServer-dev at lists.osgeo.org>
>>     >> https://lists.osgeo.org/mailman/listinfo/mapserver-dev
>>     <https://lists.osgeo.org/mailman/listinfo/mapserver-dev>
>>     >
>>     > --
>>     > http://www.spatialys.com <http://www.spatialys.com>
>>     > My software is free, but my time generally not.
>>     _______________________________________________
>>     MapServer-dev mailing list
>>     MapServer-dev at lists.osgeo.org <mailto:MapServer-dev at lists.osgeo.org>
>>     https://lists.osgeo.org/mailman/listinfo/mapserver-dev
>>     <https://lists.osgeo.org/mailman/listinfo/mapserver-dev>
>>
>
-- 
http://www.spatialys.com
My software is free, but my time generally not.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-dev/attachments/20211121/c3f1f339/attachment-0001.html>

More information about the MapServer-dev mailing list