From jmckenna at gatewaygeomatics.com  Tue Nov  2 06:18:52 2021
From: jmckenna at gatewaygeomatics.com (Jeff McKenna)
Date: Tue, 2 Nov 2021 10:18:52 -0300
Subject: [mapserver-dev] testing new mail archive process
Message-ID: <9af41833-27d8-2bde-3fd5-b96543ff93c4@gatewaygeomatics.com>

*please ignore*


-- 
Jeff McKenna
GatewayGeo: Developers of MS4W, MapServer Consulting and Training
co-founder of FOSS4G
http://gatewaygeo.com/

From bclay1297 at gmail.com  Tue Nov  9 06:48:32 2021
From: bclay1297 at gmail.com (Bruce Clay)
Date: Tue, 9 Nov 2021 09:48:32 -0500
Subject: [mapserver-dev] how to use tiles from gdal2tiles
Message-ID: <CAHemx53eLKU63rQKNvET3=7k+WNHOW+iNh9DYAuHxDPWQkn1pA@mail.gmail.com>

My apologies to those on the GDAL list as well as here I posted ther a
couple of days ago but did not get any replies.

We have tile sets created by gdal2tiles.py as well as from other tools.  We
need to display the tiles i web using frameworks such as such as cesium or
worldwindjs.  We are using the latest mapserver from ms4w on windows 10.
iI we use seed_cache we get a tileset quite different from that of
gdal2tiles.  we configured the mapcache.xml file for those datasets and
they appear to work ok in QGIS.  Using the same configuration style for the
slippy map tiles lets them show in a get capabilities but not in QGIS nor a
simple one tile query.

the mapcache,xml entries are basicallt as follows

   <cache name="tileset_a_cache" type="disk">
     <base>C:/data_static/MapTiles</base>
  </cache>

  <source name="tileset_a_src" type="wms">
  <getmap>
    <params>
      <FORMAT>image/png</FORMAT>
      <LAYERS>TileSetA</LAYERS>
      <MAP>/data/config/map/nasaww.map</MAP>
    </params>
  </getmap>
  <http>
    <url>http://localhost/cgi-bin/mapserv.exe?</url>
  </http>
</source>

<tileset name=" tileset_a  ">
   <source> tileset_a  _src</source>
        <cache> tileset_a  _cache</cache>
        <grid>GoogleMapsCompatible</grid>
        <grid>g</grid>
        <format>PNG</format>
        <metatile>8 8</metatile>
    </tileset>


If I query wmts capabilities the dataset show in the list

http://localhost/mapcache/wmts?service=wmts&request=getcapabilities&version=1.0.0

<Layer>
<ows:Title>tileset_a</ows:Title>
<ows:Identifier> tileset_a  </ows:Identifier>
<Style isDefault="true">
<ows:Identifier>default</ows:Identifier>
</Style>
<Format>image/png</Format>
<TileMatrixSetLink>
<TileMatrixSet>GoogleMapsCompatible</TileMatrixSet>
</TileMatrixSetLink>
<TileMatrixSetLink>
<TileMatrixSet>g</TileMatrixSet>
</TileMatrixSetLink>
<ResourceURL format="image/png" resourceType="tile" template="
http://localhost/mapcache/wmts/1.0.0/tileset_a/default/{TileMatrixSet}/{TileMatrix}/{TileRow}/{TileCol}.png
"/>
</Layer>

if I add a wmts layer using the following in QGIS the layer shows in the
list
http://localhost/mapcache/wmts?service=wmts&request=getcapabilities&version=1.0.0

no tiles are displayed and the server error log has several entries of
WMS server error. Invalid layer(s) given in the LAYERS parameter.

if I try to add the tiles as andXYZ layer using the following no tiles are
displayed

http://localhost/mapcache/tms/1.0.0/tileset_a/{x}/{y}/{z}.png
but no errors are added to the error log

When some of the other application generate the slippy maps we do not have
the source images so we cannot add that to the map file

What is the proper way to use the tiles generated by gdal2tiles.py with
mapserver?

Bruce
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-dev/attachments/20211109/9e57b1dc/attachment.html>

From bclay1297 at gmail.com  Wed Nov 10 06:05:14 2021
From: bclay1297 at gmail.com (Bruce Clay)
Date: Wed, 10 Nov 2021 09:05:14 -0500
Subject: [mapserver-dev] can mapserver serve xyz
Message-ID: <CAHemx53539qF8H5es5SwNpSEMrsu3mM8=A3zOQy-OtYsmA+6GA@mail.gmail.com>

Can mapserver serve xyz (mapbox or others) type tiles?

If I try to add a xyz layer in qgis using
http://localhost/cgi-bin/mapserv.exe?map=/data/config/map/nasaww.map&MODE=tile&TILEMODE=gmap&LAYERS=bluemarble/{z}/{x}/{y}.png
I always get a black set of tiles

if i query
http://localhost/cgi-bin/mapserv.exe?map=/data/config/map/nasaww.map&MODE=tile&TILEMODE=gmap&TILE=0+0+0&LAYERS=bluemarble
I get a black tile

if I change the layer name to a non-existent layer I still get a black tile
and no errors are reported in the mapserver log file.

if I click on the openlayers.html file in the root directory of the tileset
it displays valid images.

Online documentation refers to a tile mode but does not explain how to
configure the map file to support xyz.

Bruce
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-dev/attachments/20211110/a708bb47/attachment.html>

From richard.greenwood at gmail.com  Wed Nov 10 09:44:37 2021
From: richard.greenwood at gmail.com (Richard Greenwood)
Date: Wed, 10 Nov 2021 10:44:37 -0700
Subject: [mapserver-dev] can mapserver serve xyz
In-Reply-To: <CAHemx53539qF8H5es5SwNpSEMrsu3mM8=A3zOQy-OtYsmA+6GA@mail.gmail.com>
References: <CAHemx53539qF8H5es5SwNpSEMrsu3mM8=A3zOQy-OtYsmA+6GA@mail.gmail.com>
Message-ID: <CAHBySPY8BP153hyQL=5gmug7fmSWuDNHviYc6xESgSwNUVBcSQ@mail.gmail.com>

I'm not sure what you're trying to do so apologizes if I'm stating the
obvious, but MapServer is a rendering engine that takes raw data (vector
and/or raster) and returns a rasterized representation of that data based
on the request. That may be a single image or multiple tiles depending on
the request. If you already have tiles then you don't need MapServer. Your
client (QGIS, OpenLayers, etc.) can fetch those tiles directly. Neither of
the query strings that you've pasted into the email look valid to me so I'm
not surprised that you're not getting any results.

On Wed, Nov 10, 2021 at 7:05 AM Bruce Clay <bclay1297 at gmail.com> wrote:

> Can mapserver serve xyz (mapbox or others) type tiles?
>
> If I try to add a xyz layer in qgis using
> http://localhost/cgi-bin/mapserv.exe?map=/data/config/map/nasaww.map&MODE=tile&TILEMODE=gmap&LAYERS=bluemarble/{z}/{x}/{y}.png
> I always get a black set of tiles
>
> if i query
> http://localhost/cgi-bin/mapserv.exe?map=/data/config/map/nasaww.map&MODE=tile&TILEMODE=gmap&TILE=0+0+0&LAYERS=bluemarble
> I get a black tile
>
> if I change the layer name to a non-existent layer I still get a black
> tile and no errors are reported in the mapserver log file.
>
> if I click on the openlayers.html file in the root directory of the
> tileset it displays valid images.
>
> Online documentation refers to a tile mode but does not explain how to
> configure the map file to support xyz.
>
> Bruce
> _______________________________________________
> MapServer-dev mailing list
> MapServer-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/mapserver-dev
>


-- 
Richard W. Greenwood
www.greenwoodmap.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-dev/attachments/20211110/9597e3b5/attachment.html>

From bclay1297 at gmail.com  Wed Nov 10 12:21:23 2021
From: bclay1297 at gmail.com (Bruce Clay)
Date: Wed, 10 Nov 2021 15:21:23 -0500
Subject: [mapserver-dev] can mapserver serve xyz
In-Reply-To: <CAHBySPY8BP153hyQL=5gmug7fmSWuDNHviYc6xESgSwNUVBcSQ@mail.gmail.com>
References: <CAHemx53539qF8H5es5SwNpSEMrsu3mM8=A3zOQy-OtYsmA+6GA@mail.gmail.com>
 <CAHBySPY8BP153hyQL=5gmug7fmSWuDNHviYc6xESgSwNUVBcSQ@mail.gmail.com>
Message-ID: <CAHemx52x0WTe_=0TtuSPxLwE4b0-WhJjGbvMDmahw5BxiQcz8g@mail.gmail.com>

Richard:
  Thanks for your reply.  We have several datasets that I think follow the
slippy map directory structure.  We want them on a server  accessible to
remote thin clients.  We have WMTS datasets running on mapserver but that
uses a different directory structure and those are accessible through a TMS
or WMTS mapcaches query.

How does the remote thin client access the files from a
folder C:\data_static\MapTiles\BlueMarbleTiles where the tms.xml file
lives.  Some of the tilesets we generated with gdal2tiles.py and they have
a tilemapresource.xml file instead.  Those files have an openlayers.html
file in the base of the tileset that will display the teils when clicked on
but again that is from the local system.


Bruce


On Wed, Nov 10, 2021 at 12:44 PM Richard Greenwood <
richard.greenwood at gmail.com> wrote:

> I'm not sure what you're trying to do so apologizes if I'm stating the
> obvious, but MapServer is a rendering engine that takes raw data (vector
> and/or raster) and returns a rasterized representation of that data based
> on the request. That may be a single image or multiple tiles depending on
> the request. If you already have tiles then you don't need MapServer. Your
> client (QGIS, OpenLayers, etc.) can fetch those tiles directly. Neither of
> the query strings that you've pasted into the email look valid to me so I'm
> not surprised that you're not getting any results.
>
> On Wed, Nov 10, 2021 at 7:05 AM Bruce Clay <bclay1297 at gmail.com> wrote:
>
>> Can mapserver serve xyz (mapbox or others) type tiles?
>>
>> If I try to add a xyz layer in qgis using
>> http://localhost/cgi-bin/mapserv.exe?map=/data/config/map/nasaww.map&MODE=tile&TILEMODE=gmap&LAYERS=bluemarble/{z}/{x}/{y}.png
>> I always get a black set of tiles
>>
>> if i query
>> http://localhost/cgi-bin/mapserv.exe?map=/data/config/map/nasaww.map&MODE=tile&TILEMODE=gmap&TILE=0+0+0&LAYERS=bluemarble
>> I get a black tile
>>
>> if I change the layer name to a non-existent layer I still get a black
>> tile and no errors are reported in the mapserver log file.
>>
>> if I click on the openlayers.html file in the root directory of the
>> tileset it displays valid images.
>>
>> Online documentation refers to a tile mode but does not explain how to
>> configure the map file to support xyz.
>>
>> Bruce
>> _______________________________________________
>> MapServer-dev mailing list
>> MapServer-dev at lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/mapserver-dev
>>
>
>
> --
> Richard W. Greenwood
> www.greenwoodmap.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-dev/attachments/20211110/4eba2ce9/attachment.html>

From sethg at geographika.co.uk  Wed Nov 10 12:29:08 2021
From: sethg at geographika.co.uk (Seth G)
Date: Wed, 10 Nov 2021 21:29:08 +0100
Subject: [mapserver-dev] can mapserver serve xyz
In-Reply-To: <CAHemx52x0WTe_=0TtuSPxLwE4b0-WhJjGbvMDmahw5BxiQcz8g@mail.gmail.com>
References: <CAHemx53539qF8H5es5SwNpSEMrsu3mM8=A3zOQy-OtYsmA+6GA@mail.gmail.com>
 <CAHBySPY8BP153hyQL=5gmug7fmSWuDNHviYc6xESgSwNUVBcSQ@mail.gmail.com>
 <CAHemx52x0WTe_=0TtuSPxLwE4b0-WhJjGbvMDmahw5BxiQcz8g@mail.gmail.com>
Message-ID: <975bdc28-e85f-4b7a-b9b6-6571c52e756b@www.fastmail.com>

Hi Bruce,

If you already have the tiles in an XYZ file structure you can simply put them in a folder on the web server (Apache, IIS etc.) and OpenLayer can access them as in the example at https://openlayers.org/en/latest/examples/xyz.html
OpenLayers just fills in the correcy xyz parameters based on map location e.g. https://c.tile.thunderforest.com/cycle/13/3997/2556.png

The approach using http://localhost/cgi-bin/mapserv.exe?map=/data/config/map/nasaww.map&MODE=tile&TILEMODE=gmap&LAYERS=bluemarble/{z}/{x}/{y}.png <http://localhost/cgi-bin/mapserv.exe?map=/data/config/map/nasaww.map&MODE=tile&TILEMODE=gmap&LAYERS=bluemarble/%7Bz%7D/%7Bx%7D/%7By%7D.png> can be used to generate new tiles from datasets using MapServer as long as projections etc. are set correctly.

Seth

--
web:http://geographika.co.uk
twitter: @geographika


On Wed, Nov 10, 2021, at 9:21 PM, Bruce Clay wrote:
> Richard:
>   Thanks for your reply.  We have several datasets that I think follow the slippy map directory structure.  We want them on a server  accessible to remote thin clients.  We have WMTS datasets running on mapserver but that uses a different directory structure and those are accessible through a TMS or WMTS mapcaches query.  
> 
> How does the remote thin client access the files from a folder C:\data_static\MapTiles\BlueMarbleTiles where the tms.xml file lives.  Some of the tilesets we generated with gdal2tiles.py and they have a tilemapresource.xml file instead.  Those files have an openlayers.html file in the base of the tileset that will display the teils when clicked on but again that is from the local system.
> 
> 
> Bruce
> 
> 
> On Wed, Nov 10, 2021 at 12:44 PM Richard Greenwood <richard.greenwood at gmail.com> wrote:
>> I'm not sure what you're trying to do so apologizes if I'm stating the obvious, but MapServer is a rendering engine that takes raw data (vector and/or raster) and returns a rasterized representation of that data based on the request. That may be a single image or multiple tiles depending on the request. If you already have tiles then you don't need MapServer. Your client (QGIS, OpenLayers, etc.) can fetch those tiles directly. Neither of the query strings that you've pasted into the email look valid to me so I'm not surprised that you're not getting any results.
>> 
>> On Wed, Nov 10, 2021 at 7:05 AM Bruce Clay <bclay1297 at gmail.com> wrote:
>>> Can mapserver serve xyz (mapbox or others) type tiles?
>>> 
>>> If I try to add a xyz layer in qgis using http://localhost/cgi-bin/mapserv.exe?map=/data/config/map/nasaww.map&MODE=tile&TILEMODE=gmap&LAYERS=bluemarble/{z}/{x}/{y}.png <http://localhost/cgi-bin/mapserv.exe?map=/data/config/map/nasaww.map&MODE=tile&TILEMODE=gmap&LAYERS=bluemarble/%7Bz%7D/%7Bx%7D/%7By%7D.png> I always get a black set of tiles
>>> 
>>> if i query http://localhost/cgi-bin/mapserv.exe?map=/data/config/map/nasaww.map&MODE=tile&TILEMODE=gmap&TILE=0+0+0&LAYERS=bluemarble I get a black tile
>>> 
>>> if I change the layer name to a non-existent layer I still get a black tile and no errors are reported in the mapserver log file.
>>> 
>>> if I click on the openlayers.html file in the root directory of the tileset it displays valid images.
>>> 
>>> Online documentation refers to a tile mode but does not explain how to configure the map file to support xyz.  
>>> 
>>> Bruce
>>> _______________________________________________
>>> MapServer-dev mailing list
>>> MapServer-dev at lists.osgeo.org
>>> https://lists.osgeo.org/mailman/listinfo/mapserver-dev
>> 
>> 
>> -- 
>> Richard W. Greenwood
>> www.greenwoodmap.com
> _______________________________________________
> MapServer-dev mailing list
> MapServer-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/mapserver-dev
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-dev/attachments/20211110/b8e2bc2a/attachment.html>

From bclay1297 at gmail.com  Mon Nov 15 09:17:10 2021
From: bclay1297 at gmail.com (Bruce Clay)
Date: Mon, 15 Nov 2021 12:17:10 -0500
Subject: [mapserver-dev] could not load nexrad data
Message-ID: <CAHemx52bN5ex8DziELMX-CDMQb8Q96e41H1oLts9vMD_8Z3ZWw@mail.gmail.com>

I am trying to pull nexrad data from mapserver  to QGIS but it always fails
with the following error as reported in the mapsever error log

[Mon Nov 15 12:07:41 2021].244000 msDrawRasterLayerLow(nexrad_wmst):
entering.
[Mon Nov 15 12:07:41 2021].286000 msPostGISLayerWhichShapes(): Error
(ERROR:  invalid input syntax for type timestamp:
"2021-11-15T10_49:39Z:00:00"
LINE 1: ... ("file_time" between date_trunc('hour',timestamp '2021-11-1...
                                                             ^
) executing query: SELECT
"file_path"::text,"file_time"::text,ST_AsBinary(("geom"),'NDR') as
geom,"oid"::text FROM imagery.nexrad WHERE "geom" &&
ST_GeomFromText('POLYGON((-79.9499051833123
24.0232142857143,-79.9499051833123 49.9767857142857,-66.0213337547408
49.9767857142857,-66.0213337547408 24.0232142857143,-79.9499051833123
24.0232142857143))',find_srid('','imagery.nexrad','geom')) AND ("file_time"
between date_trunc('hour',timestamp '2021-11-15T10_49:39Z:00:00') and
date_trunc('hour',timestamp '2021-11-15T10_49:39Z:00:00') + interval '1
hour' - interval '1 second')
[Mon Nov 15 12:07:41 2021].286000 msPostGISLayerWhichShapes(): Query error.
Error executing query. Check server logs
[Mon Nov 15 12:07:41 2021].287000 msDrawMap(): Image handling error. Failed
to draw layer named 'nexrad_wmst'.

I followed the examples I could find on line as best I could.  My map file
entries are as follows.


#  Tile Index
LAYER
 STATUS ON
 NAME "time_idx"
 TYPE POLYGON
 DATA "geom from imagery.nexrad"
  PROJECTION
   "init=epsg:4326"
 END
 METADATA
   "wms_title" "TIME INDEX"
   "wms_srs"   "EPSG:4326"
   "wms_extent" "-126 24 -66 50"
   "wms_timeextent" "2003-08-01/2006-12-31/PT5M"
   "wms_timeitem" "file_time" #column in postgis table of type timestamp
   "wms_timedefault" "2021-11-15T10_49:39Z"
   "wms_enable_request" "*"
 END

 CONNECTIONTYPE postgis
 CONNECTION "host=127.0.0.1 port=5433 dbname=weather user=postgres
password=<pwd>"

END

# raster layer
LAYER
 NAME "nexrad_wmst"
 TYPE RASTER
 STATUS ON
 DEBUG ON
 PROJECTION
   "init=epsg:4326"
 END
 METADATA
   "wms_title" "NEXRAD BASE REF WMS-T"
   "wms_srs"   "EPSG:4326"
   "wms_extent" "-126 24 -66 50"
   "wms_timeextent" "2021-11-14/2021-11-16/PT5M"
   "wms_timeitem" "file_time" #datetime is a column in postgis table of
type timestamp
   "wms_timedefault" "2021-11-15T10_49:39Z"
   "wms_enable_request" "*"
 END
 OFFSITE 0 0 0
 TILEITEM "file_path" #filepath is a column in postgis table with varchar
of the filepath to each image
 TILEINDEX "time_idx"
 FILTER (`[file_time]` = `2021-11-15 10:49:39`)
END

The data in the database looks like the following

station_id file_path timestamp min_lon min_lat max_lon max_lat
KMVX C:\data_dynamic\weather\ridge2_111521\KMVX_L3_BDHC_20211114_230548.tif
2021-11-14 23:07 -88.82200000 34.42055000 -78.82255000 44.42000000

I do not know where the time format shown in the error message comes from,
presumably from QGIS.  the error occurs when  try to add the nexrad data as
WMS.

The goal is to sequence through the NEXRAD images over time in QGIS.

Any thoughts would be greatly appreciated.

Bruce
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-dev/attachments/20211115/ae319177/attachment.html>

From bclay1297 at gmail.com  Tue Nov 16 04:38:26 2021
From: bclay1297 at gmail.com (Bruce Clay)
Date: Tue, 16 Nov 2021 07:38:26 -0500
Subject: [mapserver-dev] WMST Get Capabilities opens OpenLayerViewer
Message-ID: <CAHemx50OuC-AxmFB3b29kqkH6T3-YUU_QR1Ym9QPGr0YkvVJOQ@mail.gmail.com>

I am using the mapserver from the latest  MS4w.  When I query WFS and WMS
GetCapabilities I get a normal xml reply.  When I do a  WMST
GetCapabilities on the same server the OpenLayersViewer opens but is
empty.  Viewing the web page source reveals java script but no xml
information.

The query follows
http://localhost/cgi-bin/mapserv.exe?map=/data/config/map/nasaww.map&SERVICE=WMST&VERSION=1.1.1&REQUEST=GetCapabilities

Bruce
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-dev/attachments/20211116/aa08d72d/attachment.html>

From jukka.rahkonen at maanmittauslaitos.fi  Tue Nov 16 06:42:52 2021
From: jukka.rahkonen at maanmittauslaitos.fi (Rahkonen Jukka (MML))
Date: Tue, 16 Nov 2021 14:42:52 +0000
Subject: [mapserver-dev] WMST Get Capabilities opens OpenLayerViewer
Message-ID: <32acb8d696be40089702d4ec10faff7b@maanmittauslaitos.fi>

Hi,

Mapserver-users list is where users can ask questions and give answers.

There is no reason to believe that Mapserver supports WMTS at all https://www.mapserver.org/documentation.html#ogc.
Also in the query you used &SERVICE=WMST&version=1.1.1. ?WMST? must be a typo and WMTS has only one published version 1.0.0. The error message from Mapserver is not especially bright, Geoservers do that better.
https://demo.geo-solutions.it/geoserver/ows?service=WMTS&version=1.0.0&request=GetCapabilities
Result:
<ows:Exception exceptionCode="InvalidParameterValue" locator="service">
<ows:ExceptionText>No service: ( WMTS )</ows:ExceptionText>
</ows:Exception>

You must install MapCache and configure it to use Mapserver as a WMS source. MapCache can do WMTS.

-Jukka Rahkonen-



L?hett?j?: MapServer-dev <mapserver-dev-bounces at lists.osgeo.org> Puolesta Bruce Clay
L?hetetty: tiistai 16. marraskuuta 2021 14.38
Vastaanottaja: MapServer Dev Mailing List <mapserver-dev at lists.osgeo.org>
Aihe: [mapserver-dev] WMST Get Capabilities opens OpenLayerViewer

I am using the mapserver from the latest  MS4w.  When I query WFS and WMS GetCapabilities I get a normal xml reply.  When I do a  WMST GetCapabilities on the same server the OpenLayersViewer opens but is empty.  Viewing the web page source reveals java script but no xml information.

The query follows
http://localhost/cgi-bin/mapserv.exe?map=/data/config/map/nasaww.map&SERVICE=WMST&VERSION=1.1.1&REQUEST=GetCapabilities

Bruce
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-dev/attachments/20211116/0850591d/attachment.html>

From bclay1297 at gmail.com  Tue Nov 16 07:04:14 2021
From: bclay1297 at gmail.com (Bruce Clay)
Date: Tue, 16 Nov 2021 10:04:14 -0500
Subject: [mapserver-dev] WMST Get Capabilities opens OpenLayerViewer
In-Reply-To: <32acb8d696be40089702d4ec10faff7b@maanmittauslaitos.fi>
References: <32acb8d696be40089702d4ec10faff7b@maanmittauslaitos.fi>
Message-ID: <CAHemx53O3VNBeRRUbm02JjCPuFONYRAMpzSmFUnvGwhLt6R1Pw@mail.gmail.com>

Jukka:
  Thanks for your reply.  What I am working on is WMS-T as defined on
https://mapserver.org/ogc/wms_time.html.  I do have mapcache installed and
it does support WMTS as you mentioned but I am trying to time sequence
weather radar images.

I have continued digging since my last post and have updated results.

If I post
http://localhost/cgi-bin/mapserv.exe?MAP=/data/config/map/nasaww.map&LAYERS=nexrad_wmst&SERVICE=WMST&VERSION=1.1.1&REQUEST=GetMap&STYLES=&FORMAT=image/jpeg&SRS=EPSG:4326&BBOX=-180,-90.0,180.0,90.0&WIDTH=1000&HEIGHT=500&TIME=2021-11-15T11:41:58Z

the OpenLayersViewer appears empty.  The Mapserver error log show  an entry
like the following for all images in the dataset as defined in the map file
using a postgis index entry along with a raster layer entry referring to
the time index

[Tue Nov 16 09:55:15 2021].828000 msDrawRasterLayerLow(nexrad_wmst):
entering.
[Tue Nov 16 09:55:15 2021].969000 msDrawRasterLayerLow(nexrad_wmst):
Filename is:
C:\data_dynamic\weather\ridge2_111521\KMVX_L3_BDHC_20211114_230548.tif
[Tue Nov 16 09:55:15 2021].969000 msDrawRasterLayerLow(nexrad_wmst): Path
is: C:\data_dynamic\weather\ridge2_111521\KMVX_L3_BDHC_20211114_230548.tif
[Tue Nov 16 09:55:15 2021].970000 msDrawRasterLayerGDAL(): Entering
transform.
[Tue Nov 16 09:55:15 2021].970000 msDrawRasterLayerGDAL():
src=0,0,7407,7407, dst=172,183,23,23
[Tue Nov 16 09:55:15 2021].970000 msDrawRasterLayerGDAL(): source raster PL
(-298.823,-139.181) for dst PL (172,183).
[Tue Nov 16 09:55:15 2021].970000 msDrawRasterLayerGDAL():
red,green,blue,alpha bands = 1,2,3,4
[Tue Nov 16 09:55:15 2021].972000 msDrawRasterLayerLow(nexrad_wmst):
Filename is:
C:\data_dynamic\weather\ridge2_111521\KMVX_L3_BDHC_20211114_231019.tif
[Tue Nov 16 09:55:15 2021].972000 msDrawRasterLayerLow(nexrad_wmst): Path
is: C:\data_dynamic\weather\ridge2_111521\KMVX_L3_BDHC_20211114_231019.tif

So mapserver is definitely seeing the map file entries otherwise it would
not know the file name its not returning the image.  For the query shown
above it should have returned a single image.  I would not expect any file
names to go in the log when I do a GetCapabilities query.

Bruce


On Tue, Nov 16, 2021 at 9:42 AM Rahkonen Jukka (MML) <
jukka.rahkonen at maanmittauslaitos.fi> wrote:

> Hi,
>
>
>
> Mapserver-users list is where users can ask questions and give answers.
>
>
>
> There is no reason to believe that Mapserver supports WMTS at all
> https://www.mapserver.org/documentation.html#ogc.
>
> Also in the query you used &SERVICE=WMST&version=1.1.1. ?WMST? must be a
> typo and WMTS has only one published version 1.0.0. The error message from
> Mapserver is not especially bright, Geoservers do that better.
>
>
> https://demo.geo-solutions.it/geoserver/ows?service=WMTS&version=1.0.0&request=GetCapabilities
> Result:
> <ows:Exception exceptionCode="InvalidParameterValue" locator="service">
>
> <ows:ExceptionText>No service: ( WMTS )</ows:ExceptionText>
>
> </ows:Exception>
>
>
>
> You must install MapCache and configure it to use Mapserver as a WMS
> source. MapCache can do WMTS.
>
>
>
> -Jukka Rahkonen-
>
>
>
>
>
>
>
> *L?hett?j?:* MapServer-dev <mapserver-dev-bounces at lists.osgeo.org> *Puolesta
> *Bruce Clay
> *L?hetetty:* tiistai 16. marraskuuta 2021 14.38
> *Vastaanottaja:* MapServer Dev Mailing List <mapserver-dev at lists.osgeo.org
> >
> *Aihe:* [mapserver-dev] WMST Get Capabilities opens OpenLayerViewer
>
>
>
> I am using the mapserver from the latest  MS4w.  When I query WFS and WMS
> GetCapabilities I get a normal xml reply.  When I do a  WMST
> GetCapabilities on the same server the OpenLayersViewer opens but is
> empty.  Viewing the web page source reveals java script but no xml
> information.
>
>
>
> The query follows
>
>
> http://localhost/cgi-bin/mapserv.exe?map=/data/config/map/nasaww.map&SERVICE=WMST&VERSION=1.1.1&REQUEST=GetCapabilities
>
>
>
> Bruce
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-dev/attachments/20211116/ba6e0d14/attachment-0001.html>

From bclay1297 at gmail.com  Tue Nov 16 08:45:35 2021
From: bclay1297 at gmail.com (Bruce Clay)
Date: Tue, 16 Nov 2021 11:45:35 -0500
Subject: [mapserver-dev] WMST Get Capabilities opens OpenLayerViewer
In-Reply-To: <32acb8d696be40089702d4ec10faff7b@maanmittauslaitos.fi>
References: <32acb8d696be40089702d4ec10faff7b@maanmittauslaitos.fi>
Message-ID: <CAHemx51K-Z=wFY4x91zf_k5AN_u7JFU-+8Ap2c96sXSMjZCfHQ@mail.gmail.com>

Jukka:
  Your email got me thinking so I took the t off and made it a regular WMS
query and I can get an image.  It looks like the WMS-T is just a WMS with a
time field, not a protocol that will sequence the images..  So it looks
like what we might need to  do is a postgis select with a time window or at
least starting point then make separate queries to get the actual images.
Does that sound right?

QGIS has a TimeManager plugin but it is updating the time display but not
showing the images.  That is a different issue for a different forum.

Thanks

Bruce




On Tue, Nov 16, 2021 at 9:42 AM Rahkonen Jukka (MML) <
jukka.rahkonen at maanmittauslaitos.fi> wrote:

> Hi,
>
>
>
> Mapserver-users list is where users can ask questions and give answers.
>
>
>
> There is no reason to believe that Mapserver supports WMTS at all
> https://www.mapserver.org/documentation.html#ogc.
>
> Also in the query you used &SERVICE=WMST&version=1.1.1. ?WMST? must be a
> typo and WMTS has only one published version 1.0.0. The error message from
> Mapserver is not especially bright, Geoservers do that better.
>
>
> https://demo.geo-solutions.it/geoserver/ows?service=WMTS&version=1.0.0&request=GetCapabilities
> Result:
> <ows:Exception exceptionCode="InvalidParameterValue" locator="service">
>
> <ows:ExceptionText>No service: ( WMTS )</ows:ExceptionText>
>
> </ows:Exception>
>
>
>
> You must install MapCache and configure it to use Mapserver as a WMS
> source. MapCache can do WMTS.
>
>
>
> -Jukka Rahkonen-
>
>
>
>
>
>
>
> *L?hett?j?:* MapServer-dev <mapserver-dev-bounces at lists.osgeo.org> *Puolesta
> *Bruce Clay
> *L?hetetty:* tiistai 16. marraskuuta 2021 14.38
> *Vastaanottaja:* MapServer Dev Mailing List <mapserver-dev at lists.osgeo.org
> >
> *Aihe:* [mapserver-dev] WMST Get Capabilities opens OpenLayerViewer
>
>
>
> I am using the mapserver from the latest  MS4w.  When I query WFS and WMS
> GetCapabilities I get a normal xml reply.  When I do a  WMST
> GetCapabilities on the same server the OpenLayersViewer opens but is
> empty.  Viewing the web page source reveals java script but no xml
> information.
>
>
>
> The query follows
>
>
> http://localhost/cgi-bin/mapserv.exe?map=/data/config/map/nasaww.map&SERVICE=WMST&VERSION=1.1.1&REQUEST=GetCapabilities
>
>
>
> Bruce
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-dev/attachments/20211116/cee501ff/attachment.html>

From jukka.rahkonen at maanmittauslaitos.fi  Tue Nov 16 08:53:10 2021
From: jukka.rahkonen at maanmittauslaitos.fi (Rahkonen Jukka (MML))
Date: Tue, 16 Nov 2021 16:53:10 +0000
Subject: [mapserver-dev] WMST Get Capabilities opens OpenLayerViewer
Message-ID: <69688924f58d4c1e896682e5f3867945@maanmittauslaitos.fi>

Hi,

WMTS means Web Map Tile Service and you can find the standard from here https://www.ogc.org/standards/wmts.  WMS-T is time enabled Web Map Service (WMS) and how to use the time dimension is documented in WMS standards https://www.ogc.org/standards/wms. Are you interested in tiles or times?

-Jukka Rahkonen-

L?hett?j?: Bruce Clay <bclay1297 at gmail.com>
L?hetetty: tiistai 16. marraskuuta 2021 18.46
Vastaanottaja: Rahkonen Jukka (MML) <jukka.rahkonen at maanmittauslaitos.fi>
Kopio: MapServer Dev Mailing List <mapserver-dev at lists.osgeo.org>
Aihe: Re: [mapserver-dev] WMST Get Capabilities opens OpenLayerViewer

Jukka:
  Your email got me thinking so I took the t off and made it a regular WMS query and I can get an image.  It looks like the WMS-T is just a WMS with a time field, not a protocol that will sequence the images..  So it looks like what we might need to  do is a postgis select with a time window or at least starting point then make separate queries to get the actual images. Does that sound right?

QGIS has a TimeManager plugin but it is updating the time display but not showing the images.  That is a different issue for a different forum.

Thanks

Bruce




On Tue, Nov 16, 2021 at 9:42 AM Rahkonen Jukka (MML) <jukka.rahkonen at maanmittauslaitos.fi<mailto:jukka.rahkonen at maanmittauslaitos.fi>> wrote:
Hi,

Mapserver-users list is where users can ask questions and give answers.

There is no reason to believe that Mapserver supports WMTS at all https://www.mapserver.org/documentation.html#ogc.
Also in the query you used &SERVICE=WMST&version=1.1.1. ?WMST? must be a typo and WMTS has only one published version 1.0.0. The error message from Mapserver is not especially bright, Geoservers do that better.
https://demo.geo-solutions.it/geoserver/ows?service=WMTS&version=1.0.0&request=GetCapabilities
Result:
<ows:Exception exceptionCode="InvalidParameterValue" locator="service">
<ows:ExceptionText>No service: ( WMTS )</ows:ExceptionText>
</ows:Exception>

You must install MapCache and configure it to use Mapserver as a WMS source. MapCache can do WMTS.

-Jukka Rahkonen-



L?hett?j?: MapServer-dev <mapserver-dev-bounces at lists.osgeo.org<mailto:mapserver-dev-bounces at lists.osgeo.org>> Puolesta Bruce Clay
L?hetetty: tiistai 16. marraskuuta 2021 14.38
Vastaanottaja: MapServer Dev Mailing List <mapserver-dev at lists.osgeo.org<mailto:mapserver-dev at lists.osgeo.org>>
Aihe: [mapserver-dev] WMST Get Capabilities opens OpenLayerViewer

I am using the mapserver from the latest  MS4w.  When I query WFS and WMS GetCapabilities I get a normal xml reply.  When I do a  WMST GetCapabilities on the same server the OpenLayersViewer opens but is empty.  Viewing the web page source reveals java script but no xml information.

The query follows
http://localhost/cgi-bin/mapserv.exe?map=/data/config/map/nasaww.map&SERVICE=WMST&VERSION=1.1.1&REQUEST=GetCapabilities

Bruce
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-dev/attachments/20211116/8bb88705/attachment-0001.html>

From bclay1297 at gmail.com  Tue Nov 16 09:26:09 2021
From: bclay1297 at gmail.com (Bruce Clay)
Date: Tue, 16 Nov 2021 12:26:09 -0500
Subject: [mapserver-dev] WMST Get Capabilities opens OpenLayerViewer
In-Reply-To: <69688924f58d4c1e896682e5f3867945@maanmittauslaitos.fi>
References: <69688924f58d4c1e896682e5f3867945@maanmittauslaitos.fi>
Message-ID: <CAHemx53Kwta6CSC892OYxJUxEM0=kS3hY8_=0bgTc9NiVBaxcA@mail.gmail.com>

Jukka:

Yes I am trying to time sequence weather images from a mapserver time
enabled layer

Bruce


On Tue, Nov 16, 2021 at 11:53 AM Rahkonen Jukka (MML) <
jukka.rahkonen at maanmittauslaitos.fi> wrote:

> Hi,
>
>
>
> WMTS means Web Map Tile Service and you can find the standard from here
> https://www.ogc.org/standards/wmts.  WMS-T is time enabled Web Map
> Service (WMS) and how to use the time dimension is documented in WMS
> standards https://www.ogc.org/standards/wms. Are you interested in tiles
> or times?
>
>
>
> -Jukka Rahkonen-
>
>
>
> *L?hett?j?:* Bruce Clay <bclay1297 at gmail.com>
> *L?hetetty:* tiistai 16. marraskuuta 2021 18.46
> *Vastaanottaja:* Rahkonen Jukka (MML) <jukka.rahkonen at maanmittauslaitos.fi
> >
> *Kopio:* MapServer Dev Mailing List <mapserver-dev at lists.osgeo.org>
> *Aihe:* Re: [mapserver-dev] WMST Get Capabilities opens OpenLayerViewer
>
>
>
> Jukka:
>
>   Your email got me thinking so I took the t off and made it a regular WMS
> query and I can get an image.  It looks like the WMS-T is just a WMS with a
> time field, not a protocol that will sequence the images..  So it looks
> like what we might need to  do is a postgis select with a time window or at
> least starting point then make separate queries to get the actual images.
> Does that sound right?
>
>
>
> QGIS has a TimeManager plugin but it is updating the time display but not
> showing the images.  That is a different issue for a different forum.
>
>
>
> Thanks
>
>
>
> Bruce
>
>
>
>
>
>
>
>
>
> On Tue, Nov 16, 2021 at 9:42 AM Rahkonen Jukka (MML) <
> jukka.rahkonen at maanmittauslaitos.fi> wrote:
>
> Hi,
>
>
>
> Mapserver-users list is where users can ask questions and give answers.
>
>
>
> There is no reason to believe that Mapserver supports WMTS at all
> https://www.mapserver.org/documentation.html#ogc.
>
> Also in the query you used &SERVICE=WMST&version=1.1.1. ?WMST? must be a
> typo and WMTS has only one published version 1.0.0. The error message from
> Mapserver is not especially bright, Geoservers do that better.
>
>
> https://demo.geo-solutions.it/geoserver/ows?service=WMTS&version=1.0.0&request=GetCapabilities
> Result:
> <ows:Exception exceptionCode="InvalidParameterValue" locator="service">
>
> <ows:ExceptionText>No service: ( WMTS )</ows:ExceptionText>
>
> </ows:Exception>
>
>
>
> You must install MapCache and configure it to use Mapserver as a WMS
> source. MapCache can do WMTS.
>
>
>
> -Jukka Rahkonen-
>
>
>
>
>
>
>
> *L?hett?j?:* MapServer-dev <mapserver-dev-bounces at lists.osgeo.org> *Puolesta
> *Bruce Clay
> *L?hetetty:* tiistai 16. marraskuuta 2021 14.38
> *Vastaanottaja:* MapServer Dev Mailing List <mapserver-dev at lists.osgeo.org
> >
> *Aihe:* [mapserver-dev] WMST Get Capabilities opens OpenLayerViewer
>
>
>
> I am using the mapserver from the latest  MS4w.  When I query WFS and WMS
> GetCapabilities I get a normal xml reply.  When I do a  WMST
> GetCapabilities on the same server the OpenLayersViewer opens but is
> empty.  Viewing the web page source reveals java script but no xml
> information.
>
>
>
> The query follows
>
>
> http://localhost/cgi-bin/mapserv.exe?map=/data/config/map/nasaww.map&SERVICE=WMST&VERSION=1.1.1&REQUEST=GetCapabilities
>
>
>
> Bruce
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-dev/attachments/20211116/d264d782/attachment.html>

From jukka.rahkonen at maanmittauslaitos.fi  Tue Nov 16 09:54:32 2021
From: jukka.rahkonen at maanmittauslaitos.fi (Rahkonen Jukka (MML))
Date: Tue, 16 Nov 2021 17:54:32 +0000
Subject: [mapserver-dev] WMST Get Capabilities opens OpenLayerViewer
Message-ID: <1266eb850dfb45739ffcbeeebc43baf1@maanmittauslaitos.fi>

Hi,

That will work fine. This service is using WMS-T boosted by an OGC API Features service that delivers some metadata https://kartta.paikkatietoikkuna.fi/?zoomLevel=1&coord=525406_7159061&mapLayers=801+100+default,3400+100+ortokuva:indeksi&timeseries=1950&noSavedState=true&showIntro=false&lang=en

-Jukka Rahkonen-

L?hett?j?: Bruce Clay <bclay1297 at gmail.com>
L?hetetty: tiistai 16. marraskuuta 2021 19.26
Vastaanottaja: Rahkonen Jukka (MML) <jukka.rahkonen at maanmittauslaitos.fi>
Kopio: MapServer Dev Mailing List <mapserver-dev at lists.osgeo.org>
Aihe: Re: [mapserver-dev] WMST Get Capabilities opens OpenLayerViewer

Jukka:

Yes I am trying to time sequence weather images from a mapserver time enabled layer

Bruce


On Tue, Nov 16, 2021 at 11:53 AM Rahkonen Jukka (MML) <jukka.rahkonen at maanmittauslaitos.fi<mailto:jukka.rahkonen at maanmittauslaitos.fi>> wrote:
Hi,

WMTS means Web Map Tile Service and you can find the standard from here https://www.ogc.org/standards/wmts.  WMS-T is time enabled Web Map Service (WMS) and how to use the time dimension is documented in WMS standards https://www.ogc.org/standards/wms. Are you interested in tiles or times?

-Jukka Rahkonen-

L?hett?j?: Bruce Clay <bclay1297 at gmail.com<mailto:bclay1297 at gmail.com>>
L?hetetty: tiistai 16. marraskuuta 2021 18.46
Vastaanottaja: Rahkonen Jukka (MML) <jukka.rahkonen at maanmittauslaitos.fi<mailto:jukka.rahkonen at maanmittauslaitos.fi>>
Kopio: MapServer Dev Mailing List <mapserver-dev at lists.osgeo.org<mailto:mapserver-dev at lists.osgeo.org>>
Aihe: Re: [mapserver-dev] WMST Get Capabilities opens OpenLayerViewer

Jukka:
  Your email got me thinking so I took the t off and made it a regular WMS query and I can get an image.  It looks like the WMS-T is just a WMS with a time field, not a protocol that will sequence the images..  So it looks like what we might need to  do is a postgis select with a time window or at least starting point then make separate queries to get the actual images. Does that sound right?

QGIS has a TimeManager plugin but it is updating the time display but not showing the images.  That is a different issue for a different forum.

Thanks

Bruce




On Tue, Nov 16, 2021 at 9:42 AM Rahkonen Jukka (MML) <jukka.rahkonen at maanmittauslaitos.fi<mailto:jukka.rahkonen at maanmittauslaitos.fi>> wrote:
Hi,

Mapserver-users list is where users can ask questions and give answers.

There is no reason to believe that Mapserver supports WMTS at all https://www.mapserver.org/documentation.html#ogc.
Also in the query you used &SERVICE=WMST&version=1.1.1. ?WMST? must be a typo and WMTS has only one published version 1.0.0. The error message from Mapserver is not especially bright, Geoservers do that better.
https://demo.geo-solutions.it/geoserver/ows?service=WMTS&version=1.0.0&request=GetCapabilities
Result:
<ows:Exception exceptionCode="InvalidParameterValue" locator="service">
<ows:ExceptionText>No service: ( WMTS )</ows:ExceptionText>
</ows:Exception>

You must install MapCache and configure it to use Mapserver as a WMS source. MapCache can do WMTS.

-Jukka Rahkonen-



L?hett?j?: MapServer-dev <mapserver-dev-bounces at lists.osgeo.org<mailto:mapserver-dev-bounces at lists.osgeo.org>> Puolesta Bruce Clay
L?hetetty: tiistai 16. marraskuuta 2021 14.38
Vastaanottaja: MapServer Dev Mailing List <mapserver-dev at lists.osgeo.org<mailto:mapserver-dev at lists.osgeo.org>>
Aihe: [mapserver-dev] WMST Get Capabilities opens OpenLayerViewer

I am using the mapserver from the latest  MS4w.  When I query WFS and WMS GetCapabilities I get a normal xml reply.  When I do a  WMST GetCapabilities on the same server the OpenLayersViewer opens but is empty.  Viewing the web page source reveals java script but no xml information.

The query follows
http://localhost/cgi-bin/mapserv.exe?map=/data/config/map/nasaww.map&SERVICE=WMST&VERSION=1.1.1&REQUEST=GetCapabilities

Bruce
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-dev/attachments/20211116/e77a7269/attachment-0001.html>

From sethg at geographika.co.uk  Sun Nov 21 03:32:12 2021
From: sethg at geographika.co.uk (Seth G)
Date: Sun, 21 Nov 2021 12:32:12 +0100
Subject: [mapserver-dev] Corrupt MapServer Zip outputs
Message-ID: <b68a56e5-f7d9-478f-9e6e-9d38f330701e@www.fastmail.com>

Hi all,

Following on from the post at https://lists.osgeo.org/pipermail/mapserver-users/2021-November/082429.html I've been trying to find where the cause of the corrupt zips may be. 

There is a msautotest to create zip files and check against the expected output, however the expected output also seems to be an invalid zip file - I tried on both Windows and Linux.

Even stranger this is both in the main branch (simply download the file with the following URL to test), and also going back in the history to 2016.

https://github.com/MapServer/MapServer/blob/main/msautotest/wxs/expected/wfsogr10_shapezip.zip

I had a working process using shapezips in a 7.6 release of MapServer so its hard to understand what could have changed, MapServer, GDAL, or zip formats themselves?
I'll attempt a Git bisect but if the issue is outside MapServer it won't find an issue. 

If anyone is able to open the zip file successfully please let me know (tried with Ark and unzip on Linux and Windows inbuilt unzip and 7-zip on Windows). Error seems to be "87 extra bytes at beginning or within zip file". 

Seth

--
web:http://geographika.co.uk
twitter: @geographika

From jbo-ads at mailo.com  Sun Nov 21 03:49:00 2021
From: jbo-ads at mailo.com (jbo-ads at mailo.com)
Date: Sun, 21 Nov 2021 12:49:00 +0100
Subject: [mapserver-dev] Corrupt MapServer Zip outputs
In-Reply-To: <b68a56e5-f7d9-478f-9e6e-9d38f330701e@www.fastmail.com>
References: <b68a56e5-f7d9-478f-9e6e-9d38f330701e@www.fastmail.com>
Message-ID: <c9183223-8510-59f2-5078-468bac818e88@mailo.com>

Hi all,

On MacOS Monterey, /usr/bin/unzip gives:

87 extra bytes at beginning or within zipfile
 ? (attempting to process anyway)
 ? inflating: road.dbf
 ? inflating: road.prj
 ? inflating: road.shp
 ? inflating: road.shx

And below is an excerpt of "ogrinfo -al road.shp":

Layer name: road
Metadata:
 ? DBF_DATE_LAST_UPDATE=2018-06-18
Geometry: Line String
Feature Count: 1
Extent: (2576803.500000, 399102.187500) - (2589745.500000, 402556.250000)

J?rome.


Le 21/11/2021 ? 12:32, Seth G a ?crit?:
> Hi all,
>
> Following on from the post at https://lists.osgeo.org/pipermail/mapserver-users/2021-November/082429.html I've been trying to find where the cause of the corrupt zips may be.
>
> There is a msautotest to create zip files and check against the expected output, however the expected output also seems to be an invalid zip file - I tried on both Windows and Linux.
>
> Even stranger this is both in the main branch (simply download the file with the following URL to test), and also going back in the history to 2016.
>
> https://github.com/MapServer/MapServer/blob/main/msautotest/wxs/expected/wfsogr10_shapezip.zip
>
> I had a working process using shapezips in a 7.6 release of MapServer so its hard to understand what could have changed, MapServer, GDAL, or zip formats themselves?
> I'll attempt a Git bisect but if the issue is outside MapServer it won't find an issue.
>
> If anyone is able to open the zip file successfully please let me know (tried with Ark and unzip on Linux and Windows inbuilt unzip and 7-zip on Windows). Error seems to be "87 extra bytes at beginning or within zip file".
>
> Seth
>
> --
> web:http://geographika.co.uk
> twitter: @geographika
> _______________________________________________
> MapServer-dev mailing list
> MapServer-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/mapserver-dev




From even.rouault at spatialys.com  Sun Nov 21 04:12:56 2021
From: even.rouault at spatialys.com (Even Rouault)
Date: Sun, 21 Nov 2021 13:12:56 +0100
Subject: [mapserver-dev] Corrupt MapServer Zip outputs
In-Reply-To: <b68a56e5-f7d9-478f-9e6e-9d38f330701e@www.fastmail.com>
References: <b68a56e5-f7d9-478f-9e6e-9d38f330701e@www.fastmail.com>
Message-ID: <9ea20c6b-9888-b88e-451b-0bbb56bcbc5f@spatialys.com>

Seth,

The .zip extension of this test file is a bit misleading, as the HTTP 
headers are not stripped (testing HTTP headers is something good to test 
in some tests). I've issued 
https://github.com/MapServer/MapServer/pull/6440 to rename the expected 
file though to avoid the confusion

You can obtain a valid .zip file without headers with

../../build/mapserv -conf ../etc/mapserv.conf -nh 
QUERY_STRING="map=wfs_ogr.map&SERVICE=WFS&VERSION=1.0.0&REQUEST=GetFeature&TYPENAME=road&featureid=road.1521&OUTPUTFORMAT=SHAPEZIP" 
 > out.zip

Even

Le 21/11/2021 ? 12:32, Seth G a ?crit?:
> Hi all,
>
> Following on from the post at https://lists.osgeo.org/pipermail/mapserver-users/2021-November/082429.html I've been trying to find where the cause of the corrupt zips may be.
>
> There is a msautotest to create zip files and check against the expected output, however the expected output also seems to be an invalid zip file - I tried on both Windows and Linux.
>
> Even stranger this is both in the main branch (simply download the file with the following URL to test), and also going back in the history to 2016.
>
> https://github.com/MapServer/MapServer/blob/main/msautotest/wxs/expected/wfsogr10_shapezip.zip
>
> I had a working process using shapezips in a 7.6 release of MapServer so its hard to understand what could have changed, MapServer, GDAL, or zip formats themselves?
> I'll attempt a Git bisect but if the issue is outside MapServer it won't find an issue.
>
> If anyone is able to open the zip file successfully please let me know (tried with Ark and unzip on Linux and Windows inbuilt unzip and 7-zip on Windows). Error seems to be "87 extra bytes at beginning or within zip file".
>
> Seth
>
> --
> web:http://geographika.co.uk
> twitter: @geographika
> _______________________________________________
> MapServer-dev mailing list
> MapServer-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/mapserver-dev

-- 
http://www.spatialys.com
My software is free, but my time generally not.


From sethg at geographika.co.uk  Sun Nov 21 06:40:05 2021
From: sethg at geographika.co.uk (Seth G)
Date: Sun, 21 Nov 2021 15:40:05 +0100
Subject: [mapserver-dev] Corrupt MapServer Zip outputs
In-Reply-To: <9ea20c6b-9888-b88e-451b-0bbb56bcbc5f@spatialys.com>
References: <b68a56e5-f7d9-478f-9e6e-9d38f330701e@www.fastmail.com>
 <9ea20c6b-9888-b88e-451b-0bbb56bcbc5f@spatialys.com>
Message-ID: <d10fb76a-0405-4ab9-96c8-a6adf4a660b3@www.fastmail.com>

Thanks J?rome. and Even. Sorry I should have checked for the headers in that zip first.
I'll attempt the bisecting now!

--
web:http://geographika.co.uk
twitter: @geographika

On Sun, Nov 21, 2021, at 1:12 PM, Even Rouault wrote:
> Seth,
>
> The .zip extension of this test file is a bit misleading, as the HTTP 
> headers are not stripped (testing HTTP headers is something good to test 
> in some tests). I've issued 
> https://github.com/MapServer/MapServer/pull/6440 to rename the expected 
> file though to avoid the confusion
>
> You can obtain a valid .zip file without headers with
>
> ../../build/mapserv -conf ../etc/mapserv.conf -nh 
> QUERY_STRING="map=wfs_ogr.map&SERVICE=WFS&VERSION=1.0.0&REQUEST=GetFeature&TYPENAME=road&featureid=road.1521&OUTPUTFORMAT=SHAPEZIP" 
>  > out.zip
>
> Even
>
> Le 21/11/2021 ? 12:32, Seth G a ?crit?:
>> Hi all,
>>
>> Following on from the post at https://lists.osgeo.org/pipermail/mapserver-users/2021-November/082429.html I've been trying to find where the cause of the corrupt zips may be.
>>
>> There is a msautotest to create zip files and check against the expected output, however the expected output also seems to be an invalid zip file - I tried on both Windows and Linux.
>>
>> Even stranger this is both in the main branch (simply download the file with the following URL to test), and also going back in the history to 2016.
>>
>> https://github.com/MapServer/MapServer/blob/main/msautotest/wxs/expected/wfsogr10_shapezip.zip
>>
>> I had a working process using shapezips in a 7.6 release of MapServer so its hard to understand what could have changed, MapServer, GDAL, or zip formats themselves?
>> I'll attempt a Git bisect but if the issue is outside MapServer it won't find an issue.
>>
>> If anyone is able to open the zip file successfully please let me know (tried with Ark and unzip on Linux and Windows inbuilt unzip and 7-zip on Windows). Error seems to be "87 extra bytes at beginning or within zip file".
>>
>> Seth
>>
>> --
>> web:http://geographika.co.uk
>> twitter: @geographika
>> _______________________________________________
>> MapServer-dev mailing list
>> MapServer-dev at lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/mapserver-dev
>
> -- 
> http://www.spatialys.com
> My software is free, but my time generally not.

From sdlime at gmail.com  Sun Nov 21 06:51:59 2021
From: sdlime at gmail.com (Steve Lime)
Date: Sun, 21 Nov 2021 08:51:59 -0600
Subject: [mapserver-dev] Corrupt MapServer Zip outputs
In-Reply-To: <d10fb76a-0405-4ab9-96c8-a6adf4a660b3@www.fastmail.com>
References: <b68a56e5-f7d9-478f-9e6e-9d38f330701e@www.fastmail.com>
 <9ea20c6b-9888-b88e-451b-0bbb56bcbc5f@spatialys.com>
 <d10fb76a-0405-4ab9-96c8-a6adf4a660b3@www.fastmail.com>
Message-ID: <CAMrKZ99GHeX9gE7WRyL5DZ6fYwZCg4gNg=v+_RSi6=y95N5LSw@mail.gmail.com>

This seems related...
https://github.com/MapServer/MapServer/issues/6412

WMS fails but shp2img works, as does WFS. Could it be WMS specific? Would
be helpful to try mode=map and see if that works.

On Sun, Nov 21, 2021 at 8:45 AM Seth G <sethg at geographika.co.uk> wrote:

> Thanks J?rome. and Even. Sorry I should have checked for the headers in
> that zip first.
> I'll attempt the bisecting now!
>
> --
> web:http://geographika.co.uk
> twitter: @geographika
>
> On Sun, Nov 21, 2021, at 1:12 PM, Even Rouault wrote:
> > Seth,
> >
> > The .zip extension of this test file is a bit misleading, as the HTTP
> > headers are not stripped (testing HTTP headers is something good to test
> > in some tests). I've issued
> > https://github.com/MapServer/MapServer/pull/6440 to rename the expected
> > file though to avoid the confusion
> >
> > You can obtain a valid .zip file without headers with
> >
> > ../../build/mapserv -conf ../etc/mapserv.conf -nh
> >
> QUERY_STRING="map=wfs_ogr.map&SERVICE=WFS&VERSION=1.0.0&REQUEST=GetFeature&TYPENAME=road&featureid=road.1521&OUTPUTFORMAT=SHAPEZIP"
>
> >  > out.zip
> >
> > Even
> >
> > Le 21/11/2021 ? 12:32, Seth G a ?crit :
> >> Hi all,
> >>
> >> Following on from the post at
> https://lists.osgeo.org/pipermail/mapserver-users/2021-November/082429.html
> I've been trying to find where the cause of the corrupt zips may be.
> >>
> >> There is a msautotest to create zip files and check against the
> expected output, however the expected output also seems to be an invalid
> zip file - I tried on both Windows and Linux.
> >>
> >> Even stranger this is both in the main branch (simply download the file
> with the following URL to test), and also going back in the history to 2016.
> >>
> >>
> https://github.com/MapServer/MapServer/blob/main/msautotest/wxs/expected/wfsogr10_shapezip.zip
> >>
> >> I had a working process using shapezips in a 7.6 release of MapServer
> so its hard to understand what could have changed, MapServer, GDAL, or zip
> formats themselves?
> >> I'll attempt a Git bisect but if the issue is outside MapServer it
> won't find an issue.
> >>
> >> If anyone is able to open the zip file successfully please let me know
> (tried with Ark and unzip on Linux and Windows inbuilt unzip and 7-zip on
> Windows). Error seems to be "87 extra bytes at beginning or within zip
> file".
> >>
> >> Seth
> >>
> >> --
> >> web:http://geographika.co.uk
> >> twitter: @geographika
> >> _______________________________________________
> >> MapServer-dev mailing list
> >> MapServer-dev at lists.osgeo.org
> >> https://lists.osgeo.org/mailman/listinfo/mapserver-dev
> >
> > --
> > http://www.spatialys.com
> > My software is free, but my time generally not.
> _______________________________________________
> MapServer-dev mailing list
> MapServer-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/mapserver-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-dev/attachments/20211121/b97737a7/attachment-0001.html>

From sethg at geographika.co.uk  Sun Nov 21 07:43:57 2021
From: sethg at geographika.co.uk (Seth G)
Date: Sun, 21 Nov 2021 16:43:57 +0100
Subject: [mapserver-dev] Corrupt MapServer Zip outputs
In-Reply-To: <CAMrKZ99GHeX9gE7WRyL5DZ6fYwZCg4gNg=v+_RSi6=y95N5LSw@mail.gmail.com>
References: <b68a56e5-f7d9-478f-9e6e-9d38f330701e@www.fastmail.com>
 <9ea20c6b-9888-b88e-451b-0bbb56bcbc5f@spatialys.com>
 <d10fb76a-0405-4ab9-96c8-a6adf4a660b3@www.fastmail.com>
 <CAMrKZ99GHeX9gE7WRyL5DZ6fYwZCg4gNg=v+_RSi6=y95N5LSw@mail.gmail.com>
Message-ID: <77f2ae3e-6514-4ae3-81b7-a9dbf0956255@www.fastmail.com>

Steve - in my case WMS are all fine, it is only related to zipped outputs using OUTPUTFORMAT blocks. 

I was trying to find a working commit for the bisect, but went back to 7-2 without finding one. 

I built MapServer main, and the latest 7-6, 7-4, and 7-2 branches and ran the msautotest:

cd /D D:\GitHub\mapserver\msautotest\wxs
mapserv -nh "QUERY_STRING=map=wfs_ogr.map&service=WFS&REQUEST=GetFeature&TYPENAME=road&version=2.0.0&outputformat=shapezip" > output.zip

All zips were invalid. However the same command does work with the https://www.gisinternals.com/query.html?content=filelist&file=release-1911-x64-gdal-2-4-4-mapserver-7-4-3.zip build. It looks like something changed between GDAL 2.4 and 3.3

Likely function is CPLCreateZip (as this is used by MapServer as mentioned by Even in the other thread), found in: https://github.com/OSGeo/gdal/commits/v3.3.3/gdal/port/cpl_minizip_zip.cpp

There are only a couple of commits between versions that look like they might be relevant?

https://github.com/OSGeo/gdal/commit/0e159e1ba8ee9e7b4a56bd0b53d16de0acf89485#diff-b156d979e3531903e620bc0616c120584c8e28945247f50b29e81c5a7a49df0b

https://github.com/OSGeo/gdal/commit/edcdc5b0a57b4fab159b14180141ee42a0408830#diff-b156d979e3531903e620bc0616c120584c8e28945247f50b29e81c5a7a49df0b

Is there a test / output zip file in the GDAL tests that could be used to verify?
I have a working and invalid zip from msautotest if that helps. 

Seth



--
web:http://geographika.co.uk
twitter: @geographika


On Sun, Nov 21, 2021, at 3:51 PM, Steve Lime wrote:
> This seems related... 
> https://github.com/MapServer/MapServer/issues/6412 
> 
> WMS fails but shp2img works, as does WFS. Could it be WMS specific? Would be helpful to try mode=map and see if that works.
> 
> On Sun, Nov 21, 2021 at 8:45 AM Seth G <sethg at geographika.co.uk> wrote:
>> Thanks J?rome. and Even. Sorry I should have checked for the headers in that zip first.
>> I'll attempt the bisecting now!
>> 
>> --
>> web:http://geographika.co.uk
>> twitter: @geographika
>> 
>> On Sun, Nov 21, 2021, at 1:12 PM, Even Rouault wrote:
>> > Seth,
>> >
>> > The .zip extension of this test file is a bit misleading, as the HTTP 
>> > headers are not stripped (testing HTTP headers is something good to test 
>> > in some tests). I've issued 
>> > https://github.com/MapServer/MapServer/pull/6440 to rename the expected 
>> > file though to avoid the confusion
>> >
>> > You can obtain a valid .zip file without headers with
>> >
>> > ../../build/mapserv -conf ../etc/mapserv.conf -nh 
>> > QUERY_STRING="map=wfs_ogr.map&SERVICE=WFS&VERSION=1.0.0&REQUEST=GetFeature&TYPENAME=road&featureid=road.1521&OUTPUTFORMAT=SHAPEZIP" 
>> >  > out.zip
>> >
>> > Even
>> >
>> > Le 21/11/2021 ? 12:32, Seth G a ?crit :
>> >> Hi all,
>> >>
>> >> Following on from the post at https://lists.osgeo.org/pipermail/mapserver-users/2021-November/082429.html I've been trying to find where the cause of the corrupt zips may be.
>> >>
>> >> There is a msautotest to create zip files and check against the expected output, however the expected output also seems to be an invalid zip file - I tried on both Windows and Linux.
>> >>
>> >> Even stranger this is both in the main branch (simply download the file with the following URL to test), and also going back in the history to 2016.
>> >>
>> >> https://github.com/MapServer/MapServer/blob/main/msautotest/wxs/expected/wfsogr10_shapezip.zip
>> >>
>> >> I had a working process using shapezips in a 7.6 release of MapServer so its hard to understand what could have changed, MapServer, GDAL, or zip formats themselves?
>> >> I'll attempt a Git bisect but if the issue is outside MapServer it won't find an issue.
>> >>
>> >> If anyone is able to open the zip file successfully please let me know (tried with Ark and unzip on Linux and Windows inbuilt unzip and 7-zip on Windows). Error seems to be "87 extra bytes at beginning or within zip file".
>> >>
>> >> Seth
>> >>
>> >> --
>> >> web:http://geographika.co.uk
>> >> twitter: @geographika
>> >> _______________________________________________
>> >> MapServer-dev mailing list
>> >> MapServer-dev at lists.osgeo.org
>> >> https://lists.osgeo.org/mailman/listinfo/mapserver-dev
>> >
>> > -- 
>> > http://www.spatialys.com
>> > My software is free, but my time generally not.
>> _______________________________________________
>> MapServer-dev mailing list
>> MapServer-dev at lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/mapserver-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-dev/attachments/20211121/9f9f5bed/attachment.html>

From even.rouault at spatialys.com  Sun Nov 21 07:54:39 2021
From: even.rouault at spatialys.com (Even Rouault)
Date: Sun, 21 Nov 2021 16:54:39 +0100
Subject: [mapserver-dev] Corrupt MapServer Zip outputs
In-Reply-To: <77f2ae3e-6514-4ae3-81b7-a9dbf0956255@www.fastmail.com>
References: <b68a56e5-f7d9-478f-9e6e-9d38f330701e@www.fastmail.com>
 <9ea20c6b-9888-b88e-451b-0bbb56bcbc5f@spatialys.com>
 <d10fb76a-0405-4ab9-96c8-a6adf4a660b3@www.fastmail.com>
 <CAMrKZ99GHeX9gE7WRyL5DZ6fYwZCg4gNg=v+_RSi6=y95N5LSw@mail.gmail.com>
 <77f2ae3e-6514-4ae3-81b7-a9dbf0956255@www.fastmail.com>
Message-ID: <67372c2a-8e30-2a3f-7248-349bb5044738@spatialys.com>


Le 21/11/2021 ? 16:43, Seth G a ?crit?:
> Steve - in my case WMS are all fine, it is only related to zipped 
> outputs using OUTPUTFORMAT blocks.
>
> I was trying to find a working commit for the bisect, but went back to 
> 7-2 without finding one.
>
> I built MapServer main, and the latest 7-6, 7-4, and 7-2 branches and 
> ran the msautotest:
>
> cd /D D:\GitHub\mapserver\msautotest\wxs
> mapserv -nh 
> "QUERY_STRING=map=wfs_ogr.map&service=WFS&REQUEST=GetFeature&TYPENAME=road&version=2.0.0&outputformat=shapezip" 
> > output.zip
>
> All zips were invalid. However the same command does work with the 
> https://www.gisinternals.com/query.html?content=filelist&file=release-1911-x64-gdal-2-4-4-mapserver-7-4-3.zip 
> <https://www.gisinternals.com/query.html?content=filelist&file=release-1911-x64-gdal-2-4-4-mapserver-7-4-3.zip> 
> build. It looks like something changed between GDAL 2.4 and 3.3
>
> Likely function is CPLCreateZip (as this is used by MapServer as 
> mentioned by Even in the other thread), found in: 
> https://github.com/OSGeo/gdal/commits/v3.3.3/gdal/port/cpl_minizip_zip.cpp 
> <https://github.com/OSGeo/gdal/commits/v3.3.3/gdal/port/cpl_minizip_zip.cpp>
>
> There are only a couple of commits between versions that look like 
> they might be relevant?
>
> https://github.com/OSGeo/gdal/commit/0e159e1ba8ee9e7b4a56bd0b53d16de0acf89485#diff-b156d979e3531903e620bc0616c120584c8e28945247f50b29e81c5a7a49df0b 
> <https://github.com/OSGeo/gdal/commit/0e159e1ba8ee9e7b4a56bd0b53d16de0acf89485#diff-b156d979e3531903e620bc0616c120584c8e28945247f50b29e81c5a7a49df0b>
>
> https://github.com/OSGeo/gdal/commit/edcdc5b0a57b4fab159b14180141ee42a0408830#diff-b156d979e3531903e620bc0616c120584c8e28945247f50b29e81c5a7a49df0b 
> <https://github.com/OSGeo/gdal/commit/edcdc5b0a57b4fab159b14180141ee42a0408830#diff-b156d979e3531903e620bc0616c120584c8e28945247f50b29e81c5a7a49df0b>
>
> Is there a test / output zip file in the GDAL tests that could be used 
> to verify?

Not necessarily generated by recent GDAL versions, and if they were, 
most certainly from Linux builds

What is weird is that you reported that .shp.zip files generated by 
recent gisinternals version with "ogr2ogr foo.shp.zip something" were 
correct.

Hum actually I see that .shp.zip are generated with the CPL_CREATE_ZIP64 
environment variable set to NO (the default is YES)

Can you try that ?

It might be that your .zip reader aren't Zip64 ready, or that GDAL write 
them in a slightly invalid way

> I have a working and invalid zip from msautotest if that helps.
yes, could be useful (but I suspect this is a difference between regular 
ZIP vs Zip64 one)
>
> Seth
>
>
>
> --
> web:http://geographika.co.uk
> twitter: @geographika
>
>
> On Sun, Nov 21, 2021, at 3:51 PM, Steve Lime wrote:
>> This seems related...
>> https://github.com/MapServer/MapServer/issues/6412 
>> <https://github.com/MapServer/MapServer/issues/6412>
>>
>> WMS fails but shp2img works, as does WFS. Could it be WMS specific? 
>> Would be helpful to try mode=map and see if that works.
>>
>> On Sun, Nov 21, 2021 at 8:45 AM Seth G <sethg at geographika.co.uk 
>> <mailto:sethg at geographika.co.uk>> wrote:
>>
>>     Thanks J?rome. and Even. Sorry I should have checked for the
>>     headers in that zip first.
>>     I'll attempt the bisecting now!
>>
>>     --
>>     web:http://geographika.co.uk <http://geographika.co.uk>
>>     twitter: @geographika
>>
>>     On Sun, Nov 21, 2021, at 1:12 PM, Even Rouault wrote:
>>     > Seth,
>>     >
>>     > The .zip extension of this test file is a bit misleading, as
>>     the HTTP
>>     > headers are not stripped (testing HTTP headers is something
>>     good to test
>>     > in some tests). I've issued
>>     > https://github.com/MapServer/MapServer/pull/6440
>>     <https://github.com/MapServer/MapServer/pull/6440> to rename the
>>     expected
>>     > file though to avoid the confusion
>>     >
>>     > You can obtain a valid .zip file without headers with
>>     >
>>     > ../../build/mapserv -conf ../etc/mapserv.conf -nh
>>     >
>>     QUERY_STRING="map=wfs_ogr.map&SERVICE=WFS&VERSION=1.0.0&REQUEST=GetFeature&TYPENAME=road&featureid=road.1521&OUTPUTFORMAT=SHAPEZIP"
>>
>>     >? > out.zip
>>     >
>>     > Even
>>     >
>>     > Le 21/11/2021 ? 12:32, Seth G a ?crit?:
>>     >> Hi all,
>>     >>
>>     >> Following on from the post at
>>     https://lists.osgeo.org/pipermail/mapserver-users/2021-November/082429.html
>>     <https://lists.osgeo.org/pipermail/mapserver-users/2021-November/082429.html>
>>     I've been trying to find where the cause of the corrupt zips may be.
>>     >>
>>     >> There is a msautotest to create zip files and check against
>>     the expected output, however the expected output also seems to be
>>     an invalid zip file - I tried on both Windows and Linux.
>>     >>
>>     >> Even stranger this is both in the main branch (simply download
>>     the file with the following URL to test), and also going back in
>>     the history to 2016.
>>     >>
>>     >>
>>     https://github.com/MapServer/MapServer/blob/main/msautotest/wxs/expected/wfsogr10_shapezip.zip
>>     <https://github.com/MapServer/MapServer/blob/main/msautotest/wxs/expected/wfsogr10_shapezip.zip>
>>     >>
>>     >> I had a working process using shapezips in a 7.6 release of
>>     MapServer so its hard to understand what could have changed,
>>     MapServer, GDAL, or zip formats themselves?
>>     >> I'll attempt a Git bisect but if the issue is outside
>>     MapServer it won't find an issue.
>>     >>
>>     >> If anyone is able to open the zip file successfully please let
>>     me know (tried with Ark and unzip on Linux and Windows inbuilt
>>     unzip and 7-zip on Windows). Error seems to be "87 extra bytes at
>>     beginning or within zip file".
>>     >>
>>     >> Seth
>>     >>
>>     >> --
>>     >> web:http://geographika.co.uk <http://geographika.co.uk>
>>     >> twitter: @geographika
>>     >> _______________________________________________
>>     >> MapServer-dev mailing list
>>     >> MapServer-dev at lists.osgeo.org
>>     <mailto:MapServer-dev at lists.osgeo.org>
>>     >> https://lists.osgeo.org/mailman/listinfo/mapserver-dev
>>     <https://lists.osgeo.org/mailman/listinfo/mapserver-dev>
>>     >
>>     > --
>>     > http://www.spatialys.com <http://www.spatialys.com>
>>     > My software is free, but my time generally not.
>>     _______________________________________________
>>     MapServer-dev mailing list
>>     MapServer-dev at lists.osgeo.org <mailto:MapServer-dev at lists.osgeo.org>
>>     https://lists.osgeo.org/mailman/listinfo/mapserver-dev
>>     <https://lists.osgeo.org/mailman/listinfo/mapserver-dev>
>>
>
-- 
http://www.spatialys.com
My software is free, but my time generally not.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-dev/attachments/20211121/c3f1f339/attachment-0001.html>

From sdlime at gmail.com  Mon Nov 22 08:14:38 2021
From: sdlime at gmail.com (Steve Lime)
Date: Mon, 22 Nov 2021 10:14:38 -0600
Subject: [mapserver-dev] Corrupt MapServer Zip outputs
In-Reply-To: <77f2ae3e-6514-4ae3-81b7-a9dbf0956255@www.fastmail.com>
References: <b68a56e5-f7d9-478f-9e6e-9d38f330701e@www.fastmail.com>
 <9ea20c6b-9888-b88e-451b-0bbb56bcbc5f@spatialys.com>
 <d10fb76a-0405-4ab9-96c8-a6adf4a660b3@www.fastmail.com>
 <CAMrKZ99GHeX9gE7WRyL5DZ6fYwZCg4gNg=v+_RSi6=y95N5LSw@mail.gmail.com>
 <77f2ae3e-6514-4ae3-81b7-a9dbf0956255@www.fastmail.com>
Message-ID: <CAMrKZ9-M=HiQwjzyE+vcVS0rUzB8Q_XQ3V3LMKt1ofHbnQjP6A@mail.gmail.com>

On Sun, Nov 21, 2021 at 9:44 AM Seth G <sethg at geographika.co.uk> wrote:

> Steve - in my case WMS are all fine, it is only related to zipped outputs
> using OUTPUTFORMAT blocks.
>

Sorry, I was thinking in reference to the corrupted PNG issue. Is the
corrupted ZIP output related? --Steve
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-dev/attachments/20211122/df1ce087/attachment.html>

From sethg at geographika.co.uk  Tue Nov 23 08:08:29 2021
From: sethg at geographika.co.uk (Seth G)
Date: Tue, 23 Nov 2021 17:08:29 +0100
Subject: [mapserver-dev] Corrupt MapServer Zip outputs
In-Reply-To: <67372c2a-8e30-2a3f-7248-349bb5044738@spatialys.com>
References: <b68a56e5-f7d9-478f-9e6e-9d38f330701e@www.fastmail.com>
 <9ea20c6b-9888-b88e-451b-0bbb56bcbc5f@spatialys.com>
 <d10fb76a-0405-4ab9-96c8-a6adf4a660b3@www.fastmail.com>
 <CAMrKZ99GHeX9gE7WRyL5DZ6fYwZCg4gNg=v+_RSi6=y95N5LSw@mail.gmail.com>
 <77f2ae3e-6514-4ae3-81b7-a9dbf0956255@www.fastmail.com>
 <67372c2a-8e30-2a3f-7248-349bb5044738@spatialys.com>
Message-ID: <8615ba91-49e4-4b53-ad2b-db5d80a5f729@www.fastmail.com>

Hi Even,

I tried with both of these in the Mapfile, and also setting in the command window (SET CPL_CREATE_ZIP64=YES) but still getting a corrupt zip:

CONFIG "CPL_CREATE_ZIP64" "NO"
CONFIG "CPL_CREATE_ZIP64" "YES"

Is there any ogr command to test the low level zip funcion in GDAL?
I thought "ogr2ogr foo.shp.zip something" command was a higher level one which meant it might not be using the same approach. 

I tried both the Windows unzip, 7-zip (x64) and opening on Linux with Ark and unzip all with the same errors.

The strangest thing is that saving via IIS through the browser is producing a valid zip using the same OUTPUTFORMAT..
I'm going to set up the msautotest and roads example in IIS to test if that works. 
Could it be related to a change in the Windows cmd shell?

Seth

--
web:http://geographika.co.uk
twitter: @geographika


On Sun, Nov 21, 2021, at 4:54 PM, Even Rouault wrote:
> 
> 
> Le 21/11/2021 ? 16:43, Seth G a ?crit :
>> Steve - in my case WMS are all fine, it is only related to zipped outputs using OUTPUTFORMAT blocks. 
>> 
>> I was trying to find a working commit for the bisect, but went back to 7-2 without finding one. 
>> 
>> I built MapServer main, and the latest 7-6, 7-4, and 7-2 branches and ran the msautotest:
>> 
>> cd /D D:\GitHub\mapserver\msautotest\wxs
>> mapserv -nh "QUERY_STRING=map=wfs_ogr.map&service=WFS&REQUEST=GetFeature&TYPENAME=road&version=2.0.0&outputformat=shapezip" > output.zip
>> 
>> All zips were invalid. However the same command does work with the https://www.gisinternals.com/query.html?content=filelist&file=release-1911-x64-gdal-2-4-4-mapserver-7-4-3.zip build. It looks like something changed between GDAL 2.4 and 3.3
>> 
>> Likely function is CPLCreateZip (as this is used by MapServer as mentioned by Even in the other thread), found in: https://github.com/OSGeo/gdal/commits/v3.3.3/gdal/port/cpl_minizip_zip.cpp
>> 
>> There are only a couple of commits between versions that look like they might be relevant?
>> 
>> https://github.com/OSGeo/gdal/commit/0e159e1ba8ee9e7b4a56bd0b53d16de0acf89485#diff-b156d979e3531903e620bc0616c120584c8e28945247f50b29e81c5a7a49df0b
>> 
>> https://github.com/OSGeo/gdal/commit/edcdc5b0a57b4fab159b14180141ee42a0408830#diff-b156d979e3531903e620bc0616c120584c8e28945247f50b29e81c5a7a49df0b
>> 
>> Is there a test / output zip file in the GDAL tests that could be used to verify?
> Not necessarily generated by recent GDAL versions, and if they were, most certainly from Linux builds
> 
> What is weird is that you reported that .shp.zip files generated by recent gisinternals version with "ogr2ogr foo.shp.zip something" were correct. 
> 
> Hum actually I see that .shp.zip are generated with the CPL_CREATE_ZIP64 environment variable set to NO (the default is YES)
> 
> Can you try that ?
> 
> It might be that your .zip reader aren't Zip64 ready, or that GDAL write them in a slightly invalid way
> 
>> I have a working and invalid zip from msautotest if that helps. 
> yes, could be useful (but I suspect this is a difference between regular ZIP vs Zip64 one)
> 
>> 
>> Seth
>> 
>> 
>> 
>> --
>> web:http://geographika.co.uk
>> twitter: @geographika
>> 
>> 
>> On Sun, Nov 21, 2021, at 3:51 PM, Steve Lime wrote:
>>> This seems related... 
>>> https://github.com/MapServer/MapServer/issues/6412 
>>> 
>>> WMS fails but shp2img works, as does WFS. Could it be WMS specific? Would be helpful to try mode=map and see if that works.
>>> 
>>> On Sun, Nov 21, 2021 at 8:45 AM Seth G <sethg at geographika.co.uk> wrote:
>>>> Thanks J?rome. and Even. Sorry I should have checked for the headers in that zip first.
>>>> I'll attempt the bisecting now!
>>>> 
>>>> --
>>>> web:http://geographika.co.uk
>>>> twitter: @geographika
>>>> 
>>>> On Sun, Nov 21, 2021, at 1:12 PM, Even Rouault wrote:
>>>> > Seth,
>>>> >
>>>> > The .zip extension of this test file is a bit misleading, as the HTTP 
>>>> > headers are not stripped (testing HTTP headers is something good to test 
>>>> > in some tests). I've issued 
>>>> > https://github.com/MapServer/MapServer/pull/6440 to rename the expected 
>>>> > file though to avoid the confusion
>>>> >
>>>> > You can obtain a valid .zip file without headers with
>>>> >
>>>> > ../../build/mapserv -conf ../etc/mapserv.conf -nh 
>>>> > QUERY_STRING="map=wfs_ogr.map&SERVICE=WFS&VERSION=1.0.0&REQUEST=GetFeature&TYPENAME=road&featureid=road.1521&OUTPUTFORMAT=SHAPEZIP" 
>>>> >  > out.zip
>>>> >
>>>> > Even
>>>> >
>>>> > Le 21/11/2021 ? 12:32, Seth G a ?crit :
>>>> >> Hi all,
>>>> >>
>>>> >> Following on from the post at https://lists.osgeo.org/pipermail/mapserver-users/2021-November/082429.html I've been trying to find where the cause of the corrupt zips may be.
>>>> >>
>>>> >> There is a msautotest to create zip files and check against the expected output, however the expected output also seems to be an invalid zip file - I tried on both Windows and Linux.
>>>> >>
>>>> >> Even stranger this is both in the main branch (simply download the file with the following URL to test), and also going back in the history to 2016.
>>>> >>
>>>> >> https://github.com/MapServer/MapServer/blob/main/msautotest/wxs/expected/wfsogr10_shapezip.zip
>>>> >>
>>>> >> I had a working process using shapezips in a 7.6 release of MapServer so its hard to understand what could have changed, MapServer, GDAL, or zip formats themselves?
>>>> >> I'll attempt a Git bisect but if the issue is outside MapServer it won't find an issue.
>>>> >>
>>>> >> If anyone is able to open the zip file successfully please let me know (tried with Ark and unzip on Linux and Windows inbuilt unzip and 7-zip on Windows). Error seems to be "87 extra bytes at beginning or within zip file".
>>>> >>
>>>> >> Seth
>>>> >>
>>>> >> --
>>>> >> web:http://geographika.co.uk
>>>> >> twitter: @geographika
>>>> >> _______________________________________________
>>>> >> MapServer-dev mailing list
>>>> >> MapServer-dev at lists.osgeo.org
>>>> >> https://lists.osgeo.org/mailman/listinfo/mapserver-dev
>>>> >
>>>> > -- 
>>>> > http://www.spatialys.com
>>>> > My software is free, but my time generally not.
>>>> _______________________________________________
>>>> MapServer-dev mailing list
>>>> MapServer-dev at lists.osgeo.org
>>>> https://lists.osgeo.org/mailman/listinfo/mapserver-dev
>> 
> -- 
> http://www.spatialys.com
> My software is free, but my time generally not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-dev/attachments/20211123/b941b379/attachment.html>

From even.rouault at spatialys.com  Tue Nov 23 08:22:25 2021
From: even.rouault at spatialys.com (Even Rouault)
Date: Tue, 23 Nov 2021 17:22:25 +0100
Subject: [mapserver-dev] Corrupt MapServer Zip outputs
In-Reply-To: <8615ba91-49e4-4b53-ad2b-db5d80a5f729@www.fastmail.com>
References: <b68a56e5-f7d9-478f-9e6e-9d38f330701e@www.fastmail.com>
 <9ea20c6b-9888-b88e-451b-0bbb56bcbc5f@spatialys.com>
 <d10fb76a-0405-4ab9-96c8-a6adf4a660b3@www.fastmail.com>
 <CAMrKZ99GHeX9gE7WRyL5DZ6fYwZCg4gNg=v+_RSi6=y95N5LSw@mail.gmail.com>
 <77f2ae3e-6514-4ae3-81b7-a9dbf0956255@www.fastmail.com>
 <67372c2a-8e30-2a3f-7248-349bb5044738@spatialys.com>
 <8615ba91-49e4-4b53-ad2b-db5d80a5f729@www.fastmail.com>
Message-ID: <77d0e235-0111-9d99-92a0-ed55d1bfe6b1@spatialys.com>

Hi Seth,
> I tried with both of these in the Mapfile, and also setting in the 
> command window (SET CPL_CREATE_ZIP64=YES) but still getting a corrupt zip:
>
> CONFIG "CPL_CREATE_ZIP64" "NO"
> CONFIG "CPL_CREATE_ZIP64" "YES"
>
> Is there any ogr command to test the low level zip funcion in GDAL?
No
> I thought "ogr2ogr foo.shp.zip something" command was a higher level 
> one which meant it might not be using the same approach.
I believe it should be representative enough of how MapServer uses GDAL 
zip API.
>
> I tried both the Windows unzip, 7-zip (x64) and opening on Linux with 
> Ark and unzip all with the same errors.
>
> The strangest thing is that saving via IIS through the browser is 
> producing a valid zip using the same OUTPUTFORMAT..
> I'm going to set up the msautotest and roads example in IIS to test if 
> that works.
> Could it be related to a change in the Windows cmd shell?

Maybe? But as I mentioned before, I'd thought this would rather be some 
msIO_needBinaryStdout() call missing. Perhaps IIS already sets stdout to 
binary mode ?

For the sake of testing, what if you add a call to 
msIO_needBinaryStdout() in mapserv.c, let's say at 
https://github.com/MapServer/MapServer/blob/main/mapserv.c#L243 . I'm 
not saying this is necessarily the right fix

Even

>
> Seth
>
> --
> web:http://geographika.co.uk
> twitter: @geographika
>
>
> On Sun, Nov 21, 2021, at 4:54 PM, Even Rouault wrote:
>>
>>
>> Le 21/11/2021 ? 16:43, Seth G a ?crit?:
>>> Steve - in my case WMS are all fine, it is only related to zipped 
>>> outputs using OUTPUTFORMAT blocks.
>>>
>>> I was trying to find a working commit for the bisect, but went back 
>>> to 7-2 without finding one.
>>>
>>> I built MapServer main, and the latest 7-6, 7-4, and 7-2 branches 
>>> and ran the msautotest:
>>>
>>> cd /D D:\GitHub\mapserver\msautotest\wxs
>>> mapserv -nh 
>>> "QUERY_STRING=map=wfs_ogr.map&service=WFS&REQUEST=GetFeature&TYPENAME=road&version=2.0.0&outputformat=shapezip" 
>>> > output.zip
>>>
>>> All zips were invalid. However the same command does work with the 
>>> https://www.gisinternals.com/query.html?content=filelist&file=release-1911-x64-gdal-2-4-4-mapserver-7-4-3.zip 
>>> <https://www.gisinternals.com/query.html?content=filelist&file=release-1911-x64-gdal-2-4-4-mapserver-7-4-3.zip> 
>>> build. It looks like something changed between GDAL 2.4 and 3.3
>>>
>>> Likely function is CPLCreateZip (as this is used by MapServer as 
>>> mentioned by Even in the other thread), found in: 
>>> https://github.com/OSGeo/gdal/commits/v3.3.3/gdal/port/cpl_minizip_zip.cpp 
>>> <https://github.com/OSGeo/gdal/commits/v3.3.3/gdal/port/cpl_minizip_zip.cpp>
>>>
>>> There are only a couple of commits between versions that look like 
>>> they might be relevant?
>>>
>>> https://github.com/OSGeo/gdal/commit/0e159e1ba8ee9e7b4a56bd0b53d16de0acf89485#diff-b156d979e3531903e620bc0616c120584c8e28945247f50b29e81c5a7a49df0b 
>>> <https://github.com/OSGeo/gdal/commit/0e159e1ba8ee9e7b4a56bd0b53d16de0acf89485#diff-b156d979e3531903e620bc0616c120584c8e28945247f50b29e81c5a7a49df0b>
>>>
>>> https://github.com/OSGeo/gdal/commit/edcdc5b0a57b4fab159b14180141ee42a0408830#diff-b156d979e3531903e620bc0616c120584c8e28945247f50b29e81c5a7a49df0b 
>>> <https://github.com/OSGeo/gdal/commit/edcdc5b0a57b4fab159b14180141ee42a0408830#diff-b156d979e3531903e620bc0616c120584c8e28945247f50b29e81c5a7a49df0b>
>>>
>>> Is there a test / output zip file in the GDAL tests that could be 
>>> used to verify?
>>
>> Not necessarily generated by recent GDAL versions, and if they were, 
>> most certainly from Linux builds
>>
>> What is weird is that you reported that .shp.zip files generated by 
>> recent gisinternals version with "ogr2ogr foo.shp.zip something" were 
>> correct.
>>
>> Hum actually I see that .shp.zip are generated with the 
>> CPL_CREATE_ZIP64 environment variable set to NO (the default is YES)
>>
>> Can you try that ?
>>
>> It might be that your .zip reader aren't Zip64 ready, or that GDAL 
>> write them in a slightly invalid way
>>
>>> I have a working and invalid zip from msautotest if that helps.
>> yes, could be useful (but I suspect this is a difference between 
>> regular ZIP vs Zip64 one)
>>
>>>
>>> Seth
>>>
>>>
>>>
>>> --
>>> web:http://geographika.co.uk <http://geographika.co.uk>
>>> twitter: @geographika
>>>
>>>
>>> On Sun, Nov 21, 2021, at 3:51 PM, Steve Lime wrote:
>>>> This seems related...
>>>> https://github.com/MapServer/MapServer/issues/6412 
>>>> <https://github.com/MapServer/MapServer/issues/6412>
>>>>
>>>> WMS fails but shp2img works, as does WFS. Could it be WMS specific? 
>>>> Would be helpful to try mode=map and see if that works.
>>>>
>>>> On Sun, Nov 21, 2021 at 8:45 AM Seth G <sethg at geographika.co.uk 
>>>> <mailto:sethg at geographika.co.uk>> wrote:
>>>>
>>>>     Thanks J?rome. and Even. Sorry I should have checked for the
>>>>     headers in that zip first.
>>>>     I'll attempt the bisecting now!
>>>>
>>>>     --
>>>>     web:http://geographika.co.uk <http://geographika.co.uk>
>>>>     twitter: @geographika
>>>>
>>>>     On Sun, Nov 21, 2021, at 1:12 PM, Even Rouault wrote:
>>>>     > Seth,
>>>>     >
>>>>     > The .zip extension of this test file is a bit misleading, as
>>>>     the HTTP
>>>>     > headers are not stripped (testing HTTP headers is something
>>>>     good to test
>>>>     > in some tests). I've issued
>>>>     > https://github.com/MapServer/MapServer/pull/6440
>>>>     <https://github.com/MapServer/MapServer/pull/6440> to rename
>>>>     the expected
>>>>     > file though to avoid the confusion
>>>>     >
>>>>     > You can obtain a valid .zip file without headers with
>>>>     >
>>>>     > ../../build/mapserv -conf ../etc/mapserv.conf -nh
>>>>     >
>>>>     QUERY_STRING="map=wfs_ogr.map&SERVICE=WFS&VERSION=1.0.0&REQUEST=GetFeature&TYPENAME=road&featureid=road.1521&OUTPUTFORMAT=SHAPEZIP"
>>>>
>>>>     >? > out.zip
>>>>     >
>>>>     > Even
>>>>     >
>>>>     > Le 21/11/2021 ? 12:32, Seth G a ?crit?:
>>>>     >> Hi all,
>>>>     >>
>>>>     >> Following on from the post at
>>>>     https://lists.osgeo.org/pipermail/mapserver-users/2021-November/082429.html
>>>>     <https://lists.osgeo.org/pipermail/mapserver-users/2021-November/082429.html>
>>>>     I've been trying to find where the cause of the corrupt zips
>>>>     may be.
>>>>     >>
>>>>     >> There is a msautotest to create zip files and check against
>>>>     the expected output, however the expected output also seems to
>>>>     be an invalid zip file - I tried on both Windows and Linux.
>>>>     >>
>>>>     >> Even stranger this is both in the main branch (simply
>>>>     download the file with the following URL to test), and also
>>>>     going back in the history to 2016.
>>>>     >>
>>>>     >>
>>>>     https://github.com/MapServer/MapServer/blob/main/msautotest/wxs/expected/wfsogr10_shapezip.zip
>>>>     <https://github.com/MapServer/MapServer/blob/main/msautotest/wxs/expected/wfsogr10_shapezip.zip>
>>>>     >>
>>>>     >> I had a working process using shapezips in a 7.6 release of
>>>>     MapServer so its hard to understand what could have changed,
>>>>     MapServer, GDAL, or zip formats themselves?
>>>>     >> I'll attempt a Git bisect but if the issue is outside
>>>>     MapServer it won't find an issue.
>>>>     >>
>>>>     >> If anyone is able to open the zip file successfully please
>>>>     let me know (tried with Ark and unzip on Linux and Windows
>>>>     inbuilt unzip and 7-zip on Windows). Error seems to be "87
>>>>     extra bytes at beginning or within zip file".
>>>>     >>
>>>>     >> Seth
>>>>     >>
>>>>     >> --
>>>>     >> web:http://geographika.co.uk <http://geographika.co.uk>
>>>>     >> twitter: @geographika
>>>>     >> _______________________________________________
>>>>     >> MapServer-dev mailing list
>>>>     >> MapServer-dev at lists.osgeo.org
>>>>     <mailto:MapServer-dev at lists.osgeo.org>
>>>>     >> https://lists.osgeo.org/mailman/listinfo/mapserver-dev
>>>>     <https://lists.osgeo.org/mailman/listinfo/mapserver-dev>
>>>>     >
>>>>     > --
>>>>     > http://www.spatialys.com <http://www.spatialys.com>
>>>>     > My software is free, but my time generally not.
>>>>     _______________________________________________
>>>>     MapServer-dev mailing list
>>>>     MapServer-dev at lists.osgeo.org
>>>>     <mailto:MapServer-dev at lists.osgeo.org>
>>>>     https://lists.osgeo.org/mailman/listinfo/mapserver-dev
>>>>     <https://lists.osgeo.org/mailman/listinfo/mapserver-dev>
>>>>
>>>
>> -- 
>> http://www.spatialys.com  <http://www.spatialys.com>
>> My software is free, but my time generally not.
>
-- 
http://www.spatialys.com
My software is free, but my time generally not.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-dev/attachments/20211123/cf45bf4c/attachment-0001.html>

From sethg at geographika.co.uk  Tue Nov 23 10:07:37 2021
From: sethg at geographika.co.uk (Seth G)
Date: Tue, 23 Nov 2021 19:07:37 +0100
Subject: [mapserver-dev] Corrupt MapServer Zip outputs
In-Reply-To: <77d0e235-0111-9d99-92a0-ed55d1bfe6b1@spatialys.com>
References: <b68a56e5-f7d9-478f-9e6e-9d38f330701e@www.fastmail.com>
 <9ea20c6b-9888-b88e-451b-0bbb56bcbc5f@spatialys.com>
 <d10fb76a-0405-4ab9-96c8-a6adf4a660b3@www.fastmail.com>
 <CAMrKZ99GHeX9gE7WRyL5DZ6fYwZCg4gNg=v+_RSi6=y95N5LSw@mail.gmail.com>
 <77f2ae3e-6514-4ae3-81b7-a9dbf0956255@www.fastmail.com>
 <67372c2a-8e30-2a3f-7248-349bb5044738@spatialys.com>
 <8615ba91-49e4-4b53-ad2b-db5d80a5f729@www.fastmail.com>
 <77d0e235-0111-9d99-92a0-ed55d1bfe6b1@spatialys.com>
Message-ID: <889ed711-7cc8-4fc8-ba93-7fd22a921d68@www.fastmail.com>

Hi Even,

No difference when adding in that function call, but I think it is on the right track so will try putting that call in in other places and debugging. 

>From the int msIO_needBinaryStdout() comments:

/*
/*      But don't do it we are using FastCGI.  We will take care of     */
/*      doing it in the libfcgi library in that case for the normal     */
/*      cgi case, and for the fastcgi case the _setmode() call          */
/*      causes a crash.                                                 */

I've no problems in my server setups as I'm using FastCGI. As this is on the commandline this is triggered, and I'm guessing it is the same issue for PNGs on the other thread if it is running under CGI rather than FastCGI.

Seth

--
web:http://geographika.co.uk
twitter: @geographika


On Tue, Nov 23, 2021, at 5:22 PM, Even Rouault wrote:
> Hi Seth,
> 
>> I tried with both of these in the Mapfile, and also setting in the command window (SET CPL_CREATE_ZIP64=YES) but still getting a corrupt zip:
>> 
>> CONFIG "CPL_CREATE_ZIP64" "NO"
>> CONFIG "CPL_CREATE_ZIP64" "YES"
>> 
>> Is there any ogr command to test the low level zip funcion in GDAL?
> No
> 
>> I thought "ogr2ogr foo.shp.zip something" command was a higher level one which meant it might not be using the same approach. 
> I believe it should be representative enough of how MapServer uses GDAL zip API.
> 
>> 
>> I tried both the Windows unzip, 7-zip (x64) and opening on Linux with Ark and unzip all with the same errors.
>> 
>> The strangest thing is that saving via IIS through the browser is producing a valid zip using the same OUTPUTFORMAT..
>> I'm going to set up the msautotest and roads example in IIS to test if that works. 
>> Could it be related to a change in the Windows cmd shell?
> Maybe? But as I mentioned before, I'd thought this would rather be some msIO_needBinaryStdout() call missing. Perhaps IIS already sets stdout to binary mode ?
> 
> For the sake of testing, what if you add a call to msIO_needBinaryStdout() in mapserv.c, let's say at https://github.com/MapServer/MapServer/blob/main/mapserv.c#L243 . I'm not saying this is necessarily the right fix
> 
> Even
> 
>> 
>> Seth
>> 
>> --
>> web:http://geographika.co.uk
>> twitter: @geographika
>> 
>> 
>> On Sun, Nov 21, 2021, at 4:54 PM, Even Rouault wrote:
>>> 
>>> 
>>> Le 21/11/2021 ? 16:43, Seth G a ?crit :
>>>> Steve - in my case WMS are all fine, it is only related to zipped outputs using OUTPUTFORMAT blocks. 
>>>> 
>>>> I was trying to find a working commit for the bisect, but went back to 7-2 without finding one. 
>>>> 
>>>> I built MapServer main, and the latest 7-6, 7-4, and 7-2 branches and ran the msautotest:
>>>> 
>>>> cd /D D:\GitHub\mapserver\msautotest\wxs
>>>> mapserv -nh "QUERY_STRING=map=wfs_ogr.map&service=WFS&REQUEST=GetFeature&TYPENAME=road&version=2.0.0&outputformat=shapezip" > output.zip
>>>> 
>>>> All zips were invalid. However the same command does work with the https://www.gisinternals.com/query.html?content=filelist&file=release-1911-x64-gdal-2-4-4-mapserver-7-4-3.zip build. It looks like something changed between GDAL 2.4 and 3.3
>>>> 
>>>> Likely function is CPLCreateZip (as this is used by MapServer as mentioned by Even in the other thread), found in: https://github.com/OSGeo/gdal/commits/v3.3.3/gdal/port/cpl_minizip_zip.cpp
>>>> 
>>>> There are only a couple of commits between versions that look like they might be relevant?
>>>> 
>>>> https://github.com/OSGeo/gdal/commit/0e159e1ba8ee9e7b4a56bd0b53d16de0acf89485#diff-b156d979e3531903e620bc0616c120584c8e28945247f50b29e81c5a7a49df0b
>>>> 
>>>> https://github.com/OSGeo/gdal/commit/edcdc5b0a57b4fab159b14180141ee42a0408830#diff-b156d979e3531903e620bc0616c120584c8e28945247f50b29e81c5a7a49df0b
>>>> 
>>>> Is there a test / output zip file in the GDAL tests that could be used to verify?
>>> Not necessarily generated by recent GDAL versions, and if they were, most certainly from Linux builds
>>> 
>>> What is weird is that you reported that .shp.zip files generated by recent gisinternals version with "ogr2ogr foo.shp.zip something" were correct. 
>>> 
>>> Hum actually I see that .shp.zip are generated with the CPL_CREATE_ZIP64 environment variable set to NO (the default is YES)
>>> 
>>> Can you try that ?
>>> 
>>> It might be that your .zip reader aren't Zip64 ready, or that GDAL write them in a slightly invalid way
>>> 
>>>> I have a working and invalid zip from msautotest if that helps. 
>>> yes, could be useful (but I suspect this is a difference between regular ZIP vs Zip64 one)
>>> 
>>>> 
>>>> Seth
>>>> 
>>>> 
>>>> 
>>>> --
>>>> web:http://geographika.co.uk
>>>> twitter: @geographika
>>>> 
>>>> 
>>>> On Sun, Nov 21, 2021, at 3:51 PM, Steve Lime wrote:
>>>>> This seems related... 
>>>>> https://github.com/MapServer/MapServer/issues/6412 
>>>>> 
>>>>> WMS fails but shp2img works, as does WFS. Could it be WMS specific? Would be helpful to try mode=map and see if that works.
>>>>> 
>>>>> On Sun, Nov 21, 2021 at 8:45 AM Seth G <sethg at geographika.co.uk> wrote:
>>>>>> Thanks J?rome. and Even. Sorry I should have checked for the headers in that zip first.
>>>>>> I'll attempt the bisecting now!
>>>>>> 
>>>>>> --
>>>>>> web:http://geographika.co.uk
>>>>>> twitter: @geographika
>>>>>> 
>>>>>> On Sun, Nov 21, 2021, at 1:12 PM, Even Rouault wrote:
>>>>>> > Seth,
>>>>>> >
>>>>>> > The .zip extension of this test file is a bit misleading, as the HTTP 
>>>>>> > headers are not stripped (testing HTTP headers is something good to test 
>>>>>> > in some tests). I've issued 
>>>>>> > https://github.com/MapServer/MapServer/pull/6440 to rename the expected 
>>>>>> > file though to avoid the confusion
>>>>>> >
>>>>>> > You can obtain a valid .zip file without headers with
>>>>>> >
>>>>>> > ../../build/mapserv -conf ../etc/mapserv.conf -nh 
>>>>>> > QUERY_STRING="map=wfs_ogr.map&SERVICE=WFS&VERSION=1.0.0&REQUEST=GetFeature&TYPENAME=road&featureid=road.1521&OUTPUTFORMAT=SHAPEZIP" 
>>>>>> >  > out.zip
>>>>>> >
>>>>>> > Even
>>>>>> >
>>>>>> > Le 21/11/2021 ? 12:32, Seth G a ?crit :
>>>>>> >> Hi all,
>>>>>> >>
>>>>>> >> Following on from the post at https://lists.osgeo.org/pipermail/mapserver-users/2021-November/082429.html I've been trying to find where the cause of the corrupt zips may be.
>>>>>> >>
>>>>>> >> There is a msautotest to create zip files and check against the expected output, however the expected output also seems to be an invalid zip file - I tried on both Windows and Linux.
>>>>>> >>
>>>>>> >> Even stranger this is both in the main branch (simply download the file with the following URL to test), and also going back in the history to 2016.
>>>>>> >>
>>>>>> >> https://github.com/MapServer/MapServer/blob/main/msautotest/wxs/expected/wfsogr10_shapezip.zip
>>>>>> >>
>>>>>> >> I had a working process using shapezips in a 7.6 release of MapServer so its hard to understand what could have changed, MapServer, GDAL, or zip formats themselves?
>>>>>> >> I'll attempt a Git bisect but if the issue is outside MapServer it won't find an issue.
>>>>>> >>
>>>>>> >> If anyone is able to open the zip file successfully please let me know (tried with Ark and unzip on Linux and Windows inbuilt unzip and 7-zip on Windows). Error seems to be "87 extra bytes at beginning or within zip file".
>>>>>> >>
>>>>>> >> Seth
>>>>>> >>
>>>>>> >> --
>>>>>> >> web:http://geographika.co.uk
>>>>>> >> twitter: @geographika
>>>>>> >> _______________________________________________
>>>>>> >> MapServer-dev mailing list
>>>>>> >> MapServer-dev at lists.osgeo.org
>>>>>> >> https://lists.osgeo.org/mailman/listinfo/mapserver-dev
>>>>>> >
>>>>>> > -- 
>>>>>> > http://www.spatialys.com
>>>>>> > My software is free, but my time generally not.
>>>>>> _______________________________________________
>>>>>> MapServer-dev mailing list
>>>>>> MapServer-dev at lists.osgeo.org
>>>>>> https://lists.osgeo.org/mailman/listinfo/mapserver-dev
>>>> 
>>> -- 
>>> http://www.spatialys.com
>>> My software is free, but my time generally not.
>>> 
>> 
> -- 
> http://www.spatialys.com
> My software is free, but my time generally not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-dev/attachments/20211123/11773240/attachment.html>

From sdlime at gmail.com  Tue Nov 23 16:30:48 2021
From: sdlime at gmail.com (Steve Lime)
Date: Tue, 23 Nov 2021 18:30:48 -0600
Subject: [mapserver-dev] PSC Meeting Thursday
Message-ID: <CAMrKZ9-4vNE6+pJ_jn4Yg2iP1szzwdDs4=Zh1oceFZhzjUM-gQ@mail.gmail.com>

We have one scheduled in a couple of days. However it?s a holiday here and
I won?t be around. Can folks do next Thursday (12/2) instead? Hoping to get
caught up on outstanding pull requests by then. ?Steve
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-dev/attachments/20211123/bb7bf87b/attachment.html>

From jmckenna at gatewaygeomatics.com  Tue Nov 23 17:22:48 2021
From: jmckenna at gatewaygeomatics.com (Jeff McKenna)
Date: Tue, 23 Nov 2021 21:22:48 -0400
Subject: [mapserver-dev] PSC Meeting Thursday
In-Reply-To: <CAMrKZ9-4vNE6+pJ_jn4Yg2iP1szzwdDs4=Zh1oceFZhzjUM-gQ@mail.gmail.com>
References: <CAMrKZ9-4vNE6+pJ_jn4Yg2iP1szzwdDs4=Zh1oceFZhzjUM-gQ@mail.gmail.com>
Message-ID: <283d1d3c-cc4c-88e3-0e62-477be8a39a91@gatewaygeomatics.com>

Thanks Steve, I was going to suggest that.  Note the time will be:

  - 8am Minny, 3pm Paris, 4pm Helsinki
  - https://github.com/MapServer/MapServer/wiki/PSC-Meeting-2021-12-02

(which is actually 14:00 UTC)

-jeff



On 2021-11-23 8:30 p.m., Steve Lime wrote:
> We have one scheduled in a couple of days. However it?s a holiday here 
> and I won?t be around. Can folks do next Thursday (12/2) instead? Hoping 
> to get caught up on outstanding pull requests by then. ?Steve




From sethg at geographika.co.uk  Wed Nov 24 10:38:16 2021
From: sethg at geographika.co.uk (Seth G)
Date: Wed, 24 Nov 2021 19:38:16 +0100
Subject: [mapserver-dev] PSC Meeting Thursday
In-Reply-To: <283d1d3c-cc4c-88e3-0e62-477be8a39a91@gatewaygeomatics.com>
References: <CAMrKZ9-4vNE6+pJ_jn4Yg2iP1szzwdDs4=Zh1oceFZhzjUM-gQ@mail.gmail.com>
 <283d1d3c-cc4c-88e3-0e62-477be8a39a91@gatewaygeomatics.com>
Message-ID: <359968a0-c13d-41aa-9a52-ecb2c301ec31@www.fastmail.com>

Sounds good to me,

Seth

--
web:http://geographika.co.uk
twitter: @geographika

On Wed, Nov 24, 2021, at 2:22 AM, Jeff McKenna wrote:
> Thanks Steve, I was going to suggest that.  Note the time will be:
>
>   - 8am Minny, 3pm Paris, 4pm Helsinki
>   - https://github.com/MapServer/MapServer/wiki/PSC-Meeting-2021-12-02
>
> (which is actually 14:00 UTC)
>
> -jeff
>
>
>
> On 2021-11-23 8:30 p.m., Steve Lime wrote:
>> We have one scheduled in a couple of days. However it?s a holiday here 
>> and I won?t be around. Can folks do next Thursday (12/2) instead? Hoping 
>> to get caught up on outstanding pull requests by then. ?Steve
>
>
>
> _______________________________________________
> MapServer-dev mailing list
> MapServer-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/mapserver-dev