<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
Hi Seth,<br>
<blockquote type="cite"
cite="mid:8615ba91-49e4-4b53-ad2b-db5d80a5f729@www.fastmail.com">
<div>I tried with both of these in the Mapfile, and also setting
in the command window (SET CPL_CREATE_ZIP64=YES) but still
getting a corrupt zip:<br>
</div>
<div><br>
</div>
<div>CONFIG "CPL_CREATE_ZIP64" "NO"<br>
</div>
<div>CONFIG "CPL_CREATE_ZIP64" "YES"</div>
<div><br>
</div>
<div>Is there any ogr command to test the low level zip funcion in
GDAL?<br>
</div>
</blockquote>
No<br>
<blockquote type="cite"
cite="mid:8615ba91-49e4-4b53-ad2b-db5d80a5f729@www.fastmail.com">
<div>I thought "ogr2ogr foo.shp.zip something" command was a
higher level one which meant it might not be using the same
approach. <br>
</div>
</blockquote>
I believe it should be representative enough of how MapServer uses
GDAL zip API.<br>
<blockquote type="cite"
cite="mid:8615ba91-49e4-4b53-ad2b-db5d80a5f729@www.fastmail.com">
<div><br>
</div>
<div>I tried both the Windows unzip, 7-zip (x64) and opening on
Linux with Ark and unzip all with the same errors.<br>
</div>
<div><br>
</div>
<div>The strangest thing is that saving via IIS through the
browser is producing a valid zip using the same OUTPUTFORMAT..<br>
</div>
<div>I'm going to set up the msautotest and roads example in IIS
to test if that works. <br>
</div>
<div>Could it be related to a change in the Windows cmd shell?<br>
</div>
</blockquote>
<p>Maybe? But as I mentioned before, I'd thought this would rather
be some msIO_needBinaryStdout() call missing. Perhaps IIS already
sets stdout to binary mode ?</p>
<p>For the sake of testing, what if you add a call to
msIO_needBinaryStdout() in mapserv.c, let's say at
<a class="moz-txt-link-freetext" href="https://github.com/MapServer/MapServer/blob/main/mapserv.c#L243">https://github.com/MapServer/MapServer/blob/main/mapserv.c#L243</a> .
I'm not saying this is necessarily the right fix<br>
</p>
<p>Even<br>
</p>
<blockquote type="cite"
cite="mid:8615ba91-49e4-4b53-ad2b-db5d80a5f729@www.fastmail.com">
<div><br>
</div>
<div>Seth<br>
</div>
<div><br>
</div>
<div id="sig62266145">
<div class="signature">--<br>
</div>
<div class="signature">web:<a class="moz-txt-link-freetext" href="http://geographika.co.uk">http://geographika.co.uk</a><br>
</div>
<div class="signature">twitter: @geographika<br>
</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>On Sun, Nov 21, 2021, at 4:54 PM, Even Rouault wrote:<br>
</div>
<blockquote type="cite" id="qt" style="">
<p><br>
</p>
<div class="qt-moz-cite-prefix">Le 21/11/2021 à 16:43, Seth G a
écrit :<br>
</div>
<blockquote type="cite"
cite="mid:77f2ae3e-6514-4ae3-81b7-a9dbf0956255@www.fastmail.com">
<div>Steve - in my case WMS are all fine, it is only related
to zipped outputs using OUTPUTFORMAT blocks. <br>
</div>
<div><br>
</div>
<div>I was trying to find a working commit for the bisect, but
went back to 7-2 without finding one. <br>
</div>
<div><br>
</div>
<div>I built MapServer main, and the latest 7-6, 7-4, and 7-2
branches and ran the msautotest:<br>
</div>
<div><br>
</div>
<div>cd /D D:\GitHub\mapserver\msautotest\wxs<br>
</div>
<div>mapserv -nh
"QUERY_STRING=map=wfs_ogr.map&service=WFS&REQUEST=GetFeature&TYPENAME=road&version=2.0.0&outputformat=shapezip"
> output.zip<br>
</div>
<div><br>
</div>
<div>All zips were invalid. However the same command does work
with the <a
href="https://www.gisinternals.com/query.html?content=filelist&file=release-1911-x64-gdal-2-4-4-mapserver-7-4-3.zip"
moz-do-not-send="true">https://www.gisinternals.com/query.html?content=filelist&file=release-1911-x64-gdal-2-4-4-mapserver-7-4-3.zip</a>
build. It looks like something changed between GDAL 2.4 and
3.3<br>
</div>
<div><br>
</div>
<div>Likely function is <span class="qt-pl-c1">CPLCreateZip
(as this is used by MapServer as mentioned by Even in the
other thread</span>), found in: <a
href="https://github.com/OSGeo/gdal/commits/v3.3.3/gdal/port/cpl_minizip_zip.cpp"
moz-do-not-send="true">https://github.com/OSGeo/gdal/commits/v3.3.3/gdal/port/cpl_minizip_zip.cpp</a><br>
</div>
<div><br>
</div>
<div>There are only a couple of commits between versions that
look like they might be relevant?<br>
</div>
<div><br>
</div>
<div><a
href="https://github.com/OSGeo/gdal/commit/0e159e1ba8ee9e7b4a56bd0b53d16de0acf89485#diff-b156d979e3531903e620bc0616c120584c8e28945247f50b29e81c5a7a49df0b"
moz-do-not-send="true">https://github.com/OSGeo/gdal/commit/0e159e1ba8ee9e7b4a56bd0b53d16de0acf89485#diff-b156d979e3531903e620bc0616c120584c8e28945247f50b29e81c5a7a49df0b</a><br>
</div>
<div><br>
</div>
<div><a
href="https://github.com/OSGeo/gdal/commit/edcdc5b0a57b4fab159b14180141ee42a0408830#diff-b156d979e3531903e620bc0616c120584c8e28945247f50b29e81c5a7a49df0b"
moz-do-not-send="true">https://github.com/OSGeo/gdal/commit/edcdc5b0a57b4fab159b14180141ee42a0408830#diff-b156d979e3531903e620bc0616c120584c8e28945247f50b29e81c5a7a49df0b</a><br>
</div>
<div><br>
</div>
<div>Is there a test / output zip file in the GDAL tests that
could be used to verify?<br>
</div>
</blockquote>
<p>Not necessarily generated by recent GDAL versions, and if
they were, most certainly from Linux builds<br>
</p>
<p>What is weird is that you reported that .shp.zip files
generated by recent gisinternals version with "ogr2ogr
foo.shp.zip something" were correct. <br>
</p>
<p>Hum actually I see that .shp.zip are generated with the
CPL_CREATE_ZIP64 environment variable set to NO (the default
is YES)<br>
</p>
<p>Can you try that ?<br>
</p>
<p>It might be that your .zip reader aren't Zip64 ready, or that
GDAL write them in a slightly invalid way<br>
</p>
<blockquote type="cite"
cite="mid:77f2ae3e-6514-4ae3-81b7-a9dbf0956255@www.fastmail.com">
<div>I have a working and invalid zip from msautotest if that
helps. <br>
</div>
</blockquote>
<div>yes, could be useful (but I suspect this is a difference
between regular ZIP vs Zip64 one)<br>
</div>
<div> <br>
</div>
<blockquote type="cite"
cite="mid:77f2ae3e-6514-4ae3-81b7-a9dbf0956255@www.fastmail.com">
<div><br>
</div>
<div>Seth<br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div id="qt-sig62266145">
<div class="qt-signature">--<br>
</div>
<div class="qt-signature">web:<a
class="qt-moz-txt-link-freetext"
href="http://geographika.co.uk" moz-do-not-send="true">http://geographika.co.uk</a><br>
</div>
<div class="qt-signature">twitter: @geographika<br>
</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>On Sun, Nov 21, 2021, at 3:51 PM, Steve Lime wrote:<br>
</div>
<blockquote type="cite" id="qt-qt" style="">
<div dir="auto">
<div>This seems related... <br>
</div>
<div dir="auto"><a
href="https://github.com/MapServer/MapServer/issues/6412"
moz-do-not-send="true">https://github.com/MapServer/MapServer/issues/6412</a> <br>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">WMS fails but shp2img works, as does WFS.
Could it be WMS specific? Would be helpful to try
mode=map and see if that works.<br>
</div>
</div>
<div>
<div><br>
</div>
<div class="qt-qt-gmail_quote">
<div dir="ltr" class="qt-qt-gmail_attr">On Sun, Nov 21,
2021 at 8:45 AM Seth G <<a
href="mailto:sethg@geographika.co.uk"
moz-do-not-send="true">sethg@geographika.co.uk</a>>
wrote:<br>
</div>
<blockquote class="qt-qt-gmail_quote"
style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-color:rgb(204,
204,
204);border-left-style:solid;border-left-width:1px;padding-left:1ex;">
<div>Thanks Jérome. and Even. Sorry I should have
checked for the headers in that zip first.<br>
</div>
<div>I'll attempt the bisecting now!<br>
</div>
<div><br>
</div>
<div>--<br>
</div>
<div>web:<a href="http://geographika.co.uk"
rel="noreferrer" target="_blank"
moz-do-not-send="true">http://geographika.co.uk</a><br>
</div>
<div>twitter: @geographika<br>
</div>
<div><br>
</div>
<div>On Sun, Nov 21, 2021, at 1:12 PM, Even Rouault
wrote:<br>
</div>
<div>> Seth,<br>
</div>
<div>><br>
</div>
<div>> The .zip extension of this test file is a
bit misleading, as the HTTP <br>
</div>
<div>> headers are not stripped (testing HTTP
headers is something good to test <br>
</div>
<div>> in some tests). I've issued <br>
</div>
<div>> <a
href="https://github.com/MapServer/MapServer/pull/6440"
rel="noreferrer" target="_blank"
moz-do-not-send="true">https://github.com/MapServer/MapServer/pull/6440</a>
to rename the expected <br>
</div>
<div>> file though to avoid the confusion<br>
</div>
<div>><br>
</div>
<div>> You can obtain a valid .zip file without
headers with<br>
</div>
<div>><br>
</div>
<div>> ../../build/mapserv -conf
../etc/mapserv.conf -nh <br>
</div>
<div>>
QUERY_STRING="map=wfs_ogr.map&SERVICE=WFS&VERSION=1.0.0&REQUEST=GetFeature&TYPENAME=road&featureid=road.1521&OUTPUTFORMAT=SHAPEZIP"
<br>
</div>
<div>> > out.zip<br>
</div>
<div>><br>
</div>
<div>> Even<br>
</div>
<div>><br>
</div>
<div>> Le 21/11/2021 à 12:32, Seth G a écrit :<br>
</div>
<div>>> Hi all,<br>
</div>
<div>>><br>
</div>
<div>>> Following on from the post at <a
href="https://lists.osgeo.org/pipermail/mapserver-users/2021-November/082429.html"
rel="noreferrer" target="_blank"
moz-do-not-send="true">https://lists.osgeo.org/pipermail/mapserver-users/2021-November/082429.html</a>
I've been trying to find where the cause of the
corrupt zips may be.<br>
</div>
<div>>><br>
</div>
<div>>> There is a msautotest to create zip
files and check against the expected output, however
the expected output also seems to be an invalid zip
file - I tried on both Windows and Linux.<br>
</div>
<div>>><br>
</div>
<div>>> Even stranger this is both in the main
branch (simply download the file with the following
URL to test), and also going back in the history to
2016.<br>
</div>
<div>>><br>
</div>
<div>>> <a
href="https://github.com/MapServer/MapServer/blob/main/msautotest/wxs/expected/wfsogr10_shapezip.zip"
rel="noreferrer" target="_blank"
moz-do-not-send="true">https://github.com/MapServer/MapServer/blob/main/msautotest/wxs/expected/wfsogr10_shapezip.zip</a><br>
</div>
<div>>><br>
</div>
<div>>> I had a working process using shapezips
in a 7.6 release of MapServer so its hard to
understand what could have changed, MapServer, GDAL,
or zip formats themselves?<br>
</div>
<div>>> I'll attempt a Git bisect but if the
issue is outside MapServer it won't find an issue.<br>
</div>
<div>>><br>
</div>
<div>>> If anyone is able to open the zip file
successfully please let me know (tried with Ark and
unzip on Linux and Windows inbuilt unzip and 7-zip
on Windows). Error seems to be "87 extra bytes at
beginning or within zip file".<br>
</div>
<div>>><br>
</div>
<div>>> Seth<br>
</div>
<div>>><br>
</div>
<div>>> --<br>
</div>
<div>>> web:<a href="http://geographika.co.uk"
rel="noreferrer" target="_blank"
moz-do-not-send="true">http://geographika.co.uk</a><br>
</div>
<div>>> twitter: @geographika<br>
</div>
<div>>>
_______________________________________________<br>
</div>
<div>>> MapServer-dev mailing list<br>
</div>
<div>>> <a
href="mailto:MapServer-dev@lists.osgeo.org"
target="_blank" moz-do-not-send="true">MapServer-dev@lists.osgeo.org</a><br>
</div>
<div>>> <a
href="https://lists.osgeo.org/mailman/listinfo/mapserver-dev"
rel="noreferrer" target="_blank"
moz-do-not-send="true">https://lists.osgeo.org/mailman/listinfo/mapserver-dev</a><br>
</div>
<div>><br>
</div>
<div>> -- <br>
</div>
<div>> <a href="http://www.spatialys.com"
rel="noreferrer" target="_blank"
moz-do-not-send="true">http://www.spatialys.com</a><br>
</div>
<div>> My software is free, but my time generally
not.<br>
</div>
<div>_______________________________________________<br>
</div>
<div>MapServer-dev mailing list<br>
</div>
<div><a href="mailto:MapServer-dev@lists.osgeo.org"
target="_blank" moz-do-not-send="true">MapServer-dev@lists.osgeo.org</a><br>
</div>
<div><a
href="https://lists.osgeo.org/mailman/listinfo/mapserver-dev"
rel="noreferrer" target="_blank"
moz-do-not-send="true">https://lists.osgeo.org/mailman/listinfo/mapserver-dev</a><br>
</div>
</blockquote>
</div>
</div>
</blockquote>
<div><br>
</div>
</blockquote>
<pre class="qt-moz-signature" cols="72">--
<a class="qt-moz-txt-link-freetext" href="http://www.spatialys.com" moz-do-not-send="true">http://www.spatialys.com</a>
My software is free, but my time generally not.
</pre>
</blockquote>
<div><br>
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
<a class="moz-txt-link-freetext" href="http://www.spatialys.com">http://www.spatialys.com</a>
My software is free, but my time generally not.</pre>
</body>
</html>