From nathan.tyler.lowry at gmail.com Mon Mar 1 11:04:40 2021 From: nathan.tyler.lowry at gmail.com (Nathan L) Date: Mon, 1 Mar 2021 12:04:40 -0700 Subject: [mapserver-users] Fwd: MapServer PostGIS: WMS Layers Without Features In-Reply-To: References: Message-ID: > > ---------- Forwarded message --------- From: Nathan L >> nathan.tyler.lowry at gmail.com Date: Tue, Feb 23, 2021 at 1:04 PM Subject: >> MapServer PostGIS: WMS Layers Without Features To: GISCO ListServ >> List at giscolorado.org >> >> Hi, ya'll! I'm in need of some help. >> >> I've prepared an instance of PostGIS and MapServer on the same machine. I >> have loaded data into PostGIS and can view these features successfully in >> thick clients (ArcGIS Pro, QGIS). >> >> My MapServer appears to be creating a successful WMS service with two >> layers. However, these two layers, called from PostGIS, are empty of any >> features. The same two layers were the features successfully confirmed to >> be in the PostGIS database, above. Nevertheless, my WMS layers from >> MapServer show up empty. >> >> What am I missing? >> >> My Mapfile: https://pastebin.com/tRTJKTxA >> >> My ms_error.txt: https://pastebin.com/daJJMEkJ >> >> My PostgreSQL log file: https://pastebin.com/EN2LzheE >> >> My WMS Returns: https://pastebin.com/NR0W0Pg7 >> >> Nathan Tyler Lowry >> >> Comprehensive GIS Solutions >> >> (719) 208-7092 Desk >> >> (719) 963-3571 Cell >> >> (719) 960-2084 Fax >> >> Nathan.Tyler.Lowry at gmail.com >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lars.fricke at skendata.de Tue Mar 2 01:09:28 2021 From: lars.fricke at skendata.de (Lars Fricke) Date: Tue, 2 Mar 2021 10:09:28 +0100 Subject: [mapserver-users] Fwd: MapServer PostGIS: WMS Layers Without Features In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From sdlime at gmail.com Tue Mar 2 07:06:11 2021 From: sdlime at gmail.com (Steve Lime) Date: Tue, 2 Mar 2021 09:06:11 -0600 Subject: [mapserver-users] Fwd: MapServer PostGIS: WMS Layers Without Features In-Reply-To: References: Message-ID: What are you using to view the WMS service? What do you get if you take WMS configuration out of the equation and just do a call like so: http://localhost/cgi-bin/mapserv.exe?map=[your mapfile]&mode=map&layers=all That should get you an image with all the features in all the layers for your default extent. --Steve On Mon, Mar 1, 2021 at 1:05 PM Nathan L wrote: > ---------- Forwarded message --------- From: Nathan L >>> nathan.tyler.lowry at gmail.com Date: Tue, Feb 23, 2021 at 1:04 PM >>> Subject: MapServer PostGIS: WMS Layers Without Features To: GISCO ListServ >>> List at giscolorado.org >>> >>> Hi, ya'll! I'm in need of some help. >>> >>> I've prepared an instance of PostGIS and MapServer on the same machine. >>> I have loaded data into PostGIS and can view these features successfully in >>> thick clients (ArcGIS Pro, QGIS). >>> >>> My MapServer appears to be creating a successful WMS service with two >>> layers. However, these two layers, called from PostGIS, are empty of any >>> features. The same two layers were the features successfully confirmed to >>> be in the PostGIS database, above. Nevertheless, my WMS layers from >>> MapServer show up empty. >>> >>> What am I missing? >>> >>> My Mapfile: https://pastebin.com/tRTJKTxA >>> >>> My ms_error.txt: https://pastebin.com/daJJMEkJ >>> >>> My PostgreSQL log file: https://pastebin.com/EN2LzheE >>> >>> My WMS Returns: https://pastebin.com/NR0W0Pg7 >>> >>> Nathan Tyler Lowry >>> >>> Comprehensive GIS Solutions >>> >>> (719) 208-7092 Desk >>> >>> (719) 963-3571 Cell >>> >>> (719) 960-2084 Fax >>> >>> Nathan.Tyler.Lowry at gmail.com >>> >> _______________________________________________ > mapserver-users mailing list > mapserver-users at lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/mapserver-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jukka.rahkonen at maanmittauslaitos.fi Tue Mar 2 07:25:54 2021 From: jukka.rahkonen at maanmittauslaitos.fi (Rahkonen Jukka (MML)) Date: Tue, 2 Mar 2021 15:25:54 +0000 Subject: [mapserver-users] Fwd: MapServer PostGIS: WMS Layers Without Features Message-ID: Hi, First thing that comes into my mind is that for some reason Mapserver may believe that the request is outside the area where you have data. I would double check the projections and extents. -Jukka Rahkonen- L?hett?j?: mapserver-users Puolesta Steve Lime L?hetetty: tiistai 2. maaliskuuta 2021 17.06 Vastaanottaja: Nathan L Kopio: Mapserver Aihe: Re: [mapserver-users] Fwd: MapServer PostGIS: WMS Layers Without Features What are you using to view the WMS service? What do you get if you take WMS configuration out of the equation and just do a call like so: http://localhost/cgi-bin/mapserv.exe?map=[your mapfile]&mode=map&layers=all That should get you an image with all the features in all the layers for your default extent. --Steve On Mon, Mar 1, 2021 at 1:05 PM Nathan L > wrote: ---------- Forwarded message --------- From: Nathan L nathan.tyler.lowry at gmail.com Date: Tue, Feb 23, 2021 at 1:04 PM Subject: MapServer PostGIS: WMS Layers Without Features To: GISCO ListServ List at giscolorado.org Hi, ya'll! I'm in need of some help. I've prepared an instance of PostGIS and MapServer on the same machine. I have loaded data into PostGIS and can view these features successfully in thick clients (ArcGIS Pro, QGIS). My MapServer appears to be creating a successful WMS service with two layers. However, these two layers, called from PostGIS, are empty of any features. The same two layers were the features successfully confirmed to be in the PostGIS database, above. Nevertheless, my WMS layers from MapServer show up empty. What am I missing? My Mapfile: https://pastebin.com/tRTJKTxA My ms_error.txt: https://pastebin.com/daJJMEkJ My PostgreSQL log file: https://pastebin.com/EN2LzheE My WMS Returns: https://pastebin.com/NR0W0Pg7 Nathan Tyler Lowry Comprehensive GIS Solutions (719) 208-7092 Desk (719) 963-3571 Cell (719) 960-2084 Fax Nathan.Tyler.Lowry at gmail.com _______________________________________________ mapserver-users mailing list mapserver-users at lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From jvaldezch at gmail.com Tue Mar 2 13:26:12 2021 From: jvaldezch at gmail.com (Jaime Valdez) Date: Tue, 2 Mar 2021 15:26:12 -0600 Subject: [mapserver-users] How to use MS_ENCRYPTION_KEY with AWS connection Message-ID: Hi, I'm trying to setup a mapfile by using and encrypting AWS credentials into CONFIG parameters, but it is not working. CONFIG "AWS_ACCESS_KEY_ID" "{ ... }" CONFIG "AWS_SECRET_ACCESS_KEY" "{ ... }" CONFIG "AWS_DEFAULT_REGION" "{ ... }" When I change into plain AWS credentials works. I have use MS_ENCRYPTION_KEY in the past when connecting to PostGIS. Kind regards. --- Jaime E. Valdez ? + 52 442 168 8006 -------------- next part -------------- An HTML attachment was scrubbed... URL: From sethg at geographika.co.uk Tue Mar 2 14:07:27 2021 From: sethg at geographika.co.uk (Seth G) Date: Tue, 02 Mar 2021 23:07:27 +0100 Subject: [mapserver-users] =?utf-8?q?How_to_use_MS=5FENCRYPTION=5FKEY_wit?= =?utf-8?q?h_AWS_connection?= In-Reply-To: References: Message-ID: <35f352af-8a50-47cf-bb1b-18621788d648@www.fastmail.com> Hi Jaime, I believe the encryption key can only be used for some connection types (I know for example it is not implemented currently for SQL Server). Docs report the following as supported: OGR Oracle Spatial PostGIS SDE https://mapserver.org/utilities/msencrypt.html#purpose Regards, Seth -- web:http://geographika.co.uk twitter: @geographika On Tue, Mar 2, 2021, at 10:26 PM, Jaime Valdez wrote: > Hi, > I'm trying to setup a mapfile by using and encrypting AWS credentials into CONFIG parameters, but it is not working. > `CONFIG "AWS_ACCESS_KEY_ID" "{ ... }"` > `CONFIG "AWS_SECRET_ACCESS_KEY" "{ ... }"` > `CONFIG "AWS_DEFAULT_REGION" "{ ... }"` > When I change into plain AWS credentials works. I have use MS_ENCRYPTION_KEY in the past when connecting to PostGIS. > Kind regards. > --- > Jaime E. Valdez ? + 52 442 168 8006 > _______________________________________________ > mapserver-users mailing list > mapserver-users at lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/mapserver-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at home.gofferje.net Sun Mar 14 03:43:59 2021 From: lists at home.gofferje.net (Stefan Gofferje) Date: Sun, 14 Mar 2021 12:43:59 +0200 Subject: [mapserver-users] Hello from Finland Message-ID: Hello all! I just wanted to quickly introduce myself. I just started tinkering with mapserver to see if I can make it work as a new backend for my weather website. I'm running www.saakeskus.fi as a free website for stormchasers in Finland since 2007. At the moment, it's tons of pre-rendered maps which are done with GrADS in a Joomla website. A while ago I started reworking the web part based on Leaflet and and started writing new map creation scripts in bash with GDAL. That works pretty nicely but results in one huge mapfile for each parameter at each time... And then I stumbled over mapserver... :). I have a test server running on docker on my home server at the moment and I'm making (I think) pretty good progress understanding how it works. I don't really know a lot about GIS stuff but I got my first working WMS maps from grib files out of it after about one and a half day of tinkering. On the old version of my website, I published almost all my GrADS scripts because there wasn't much information on the web on how to do nice weather maps with GrADS when I started. If I can make mapserver work for me, I'm planning to do the same with mapserver map files. Looking forward to the community! Have a beautiful Sunday! Stefan -- (o_ Stefan Gofferje | SCLT, MCP, CCSA //\ Reg'd Linux User #247167 | VCP #2263 V_/_ https://www.gofferje.net | https://www.saakeskus.fi From iljya.kalai at gmail.com Sun Mar 14 08:51:34 2021 From: iljya.kalai at gmail.com (Iljya Kalai) Date: Sun, 14 Mar 2021 16:51:34 +0100 Subject: [mapserver-users] Hello from Finland In-Reply-To: References: Message-ID: Welcome Stefan! Your plan sounds great and I look forward to it. I'm also somewhat new here :) Iljya On Sun., Mar. 14, 2021, 11:53 Stefan Gofferje, wrote: > Hello all! > > I just wanted to quickly introduce myself. I just started tinkering with > mapserver to see if I can make it work as a new backend for my weather > website. I'm running www.saakeskus.fi as a free website for stormchasers > in Finland since 2007. At the moment, it's tons of pre-rendered maps > which are done with GrADS in a Joomla website. A while ago I started > reworking the web part based on Leaflet and and started writing new map > creation scripts in bash with GDAL. That works pretty nicely but results > in one huge mapfile for each parameter at each time... > And then I stumbled over mapserver... :). > I have a test server running on docker on my home server at the moment > and I'm making (I think) pretty good progress understanding how it > works. I don't really know a lot about GIS stuff but I got my first > working WMS maps from grib files out of it after about one and a half > day of tinkering. > On the old version of my website, I published almost all my GrADS > scripts because there wasn't much information on the web on how to do > nice weather maps with GrADS when I started. If I can make mapserver > work for me, I'm planning to do the same with mapserver map files. > > Looking forward to the community! > Have a beautiful Sunday! > Stefan > > -- > (o_ Stefan Gofferje | SCLT, MCP, CCSA > //\ Reg'd Linux User #247167 | VCP #2263 > V_/_ https://www.gofferje.net | https://www.saakeskus.fi > > _______________________________________________ > mapserver-users mailing list > mapserver-users at lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/mapserver-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jmckenna at gatewaygeomatics.com Mon Mar 15 05:35:34 2021 From: jmckenna at gatewaygeomatics.com (Jeff McKenna) Date: Mon, 15 Mar 2021 09:35:34 -0300 Subject: [mapserver-users] Hello from Finland In-Reply-To: References: Message-ID: <1a29c34c-8ad1-29a7-df7e-8c28bf18a1fa@gatewaygeomatics.com> Thanks Stefan for this wonderful introduction, it's nice to hear of positive experiences to give us all more motivation to continue to helping to improve MapServer! Welcome to both you and Iliya to the community. Keep us posted on your activities! Oh, once your MapServer backend is plugged-in be sure to add your weather site to the MapServer gallery, so we can help new users see what can be done... https://github.com/mapserver/mapserver/wiki/MapServer-Site-Gallery Hello to Finland from the east coast of Canada! -jeff -- Jeff McKenna GatewayGeo: Developers of MS4W, MapServer Consulting and Training co-founder of FOSS4G http://gatewaygeo.com/ On 2021-03-14 7:43 a.m., Stefan Gofferje wrote: > Hello all! > > I just wanted to quickly introduce myself. I just started tinkering with > mapserver to see if I can make it work as a new backend for my weather > website. I'm running www.saakeskus.fi as a free website for stormchasers > in Finland since 2007. At the moment, it's tons of pre-rendered maps > which are done with GrADS in a Joomla website. A while ago I started > reworking the web part based on Leaflet and and started writing new map > creation scripts in bash with GDAL. That works pretty nicely but results > in one huge mapfile for each parameter at each time... > And then I stumbled over mapserver... :). > I have a test server running on docker on my home server at the moment > and I'm making (I think) pretty good progress understanding how it > works. I don't really know a lot about GIS stuff but I got my first > working WMS maps from grib files out of it after about one and a half > day of tinkering. > On the old version of my website, I published almost all my GrADS > scripts because there wasn't much information on the web on how to do > nice weather maps with GrADS when I started. If I can make mapserver > work for me, I'm planning to do the same with mapserver map files. > > Looking forward to the community! > Have a beautiful Sunday! > Stefan > From lists at home.gofferje.net Tue Mar 16 03:11:29 2021 From: lists at home.gofferje.net (Stefan Gofferje) Date: Tue, 16 Mar 2021 12:11:29 +0200 Subject: [mapserver-users] Hello from Finland In-Reply-To: <1a29c34c-8ad1-29a7-df7e-8c28bf18a1fa@gatewaygeomatics.com> References: <1a29c34c-8ad1-29a7-df7e-8c28bf18a1fa@gatewaygeomatics.com> Message-ID: Hi Jeff, On 3/15/21 2:35 PM, Jeff McKenna wrote: > Thanks Stefan for this wonderful introduction, it's nice to hear of > positive experiences to give us all more motivation to continue to > helping to improve MapServer! Welcome to both you and Iliya to the > community. Keep us posted on your activities! > > Oh, once your MapServer backend is plugged-in be sure to add your > weather site to the MapServer gallery, so we can help new users see > what can be done... Thanks for your nice words! I now have mapserver and mapcache running successfully and talking to each other in my home lab. I have set up a number of test layers from different input sources pretty successfully. The documentation and the configuration required some adaption of my thinking processes because it's pretty different from anything else I have ever done before but I love this kind of challenges! Currently I have only two issues which prevent me from doing the swap right now. Thanks to NOAA using a 0-360? grid instead of (-180)-180?, there's some artefacts when reprojecting to epsg:3857. For raster maps this is already addressed in #6106. I ran into an issue which LINE maps with a CONTOUR source which probably is related and opened #6261. Then I have a question... Although it's not strictly necessary, I would like to paint those Hs and Ls in my atmospheric pressure map. Is there a way to run scripts or something like that in mapserver. I could probably run some GDAL stuff externally over the grid files in my download script and create some point overlays which could be in a GROUP with the isoline layer but doing this inside of mapserver would kinda be cooler. And a second question: How is mapserver handling raster files memory-wise? Does it try to load the whole file to the memory at once? My biggest issue during the whole 14 years I run this website always has been balancing preprocessing and map rendering. The more preprocessing I have to do, the longer the whole process of map generation takes but of course I couldn't afford to buy a $100k top-of-the-line server with terabytes of RAM and plenty of CPU cores for the rendering... I'm thinking about my input file strategy at the moment... 1.) One GRIB file for everything (around 1.9GB) 2.) One GRIB file per parameter for all time steps 3.) One GRIB file per time step including all parameters 4.) One GRIB file per parameter for each time step 4. would be the most complex to configure and also would be fun (irony!) to make local forecasts from but probably take the least memory. 2. and 3. are a fairly good compromise with 2 being easier for calculating local forecasts and 3 easier for calculating additional stuff like my "severe weather warning level" 1. would be easiest for rendering and calculating but even at the moment with GrADS I every now and then run into memory issues (leaks?). This project really turns out to be a fun challenge! Stefan -- (o_ Stefan Gofferje | SCLT, MCP, CCSA //\ Reg'd Linux User #247167 | VCP #2263 V_/_ https://www.gofferje.net | https://www.saakeskus.fi From jukka.rahkonen at maanmittauslaitos.fi Tue Mar 16 03:54:13 2021 From: jukka.rahkonen at maanmittauslaitos.fi (Rahkonen Jukka (MML)) Date: Tue, 16 Mar 2021 10:54:13 +0000 Subject: [mapserver-users] Hello from Finland Message-ID: Hi Stefan, When it comes to memory usage with raster files, Mapserver is reading rasters with GDAL so in this case with https://gdal.org/drivers/raster/grib.html. I have never been playing with GRIB files and I do not know how well the format is optimized for features that are needed for making fast map services: fast access to subregions and overview levels. I have no idea how well GRIB files are indexed for random access, it there is difference between png and JPEG2000 encoding, and how well the GDAL GRIB driver can utilize the possibilities that the format offer. The most common raster format that Mapserver services are using is certainly GeoTIFF and when they are tiled, compressed, and have overviews they work pretty well. It is certainly an advantage if you can use the GRIB files directly without conversions. I see that you are using Mapcache in front of Mapserver and it will help a lot if you pre-seed the tiles. If you do not pre-seed then each time when the user is the first visitor on some site the bottle neck will be Mapserver. The Ls/Hs idea is interesting. I do not believe there is anything ready available but I know that my colleagues at the Finnish Geodetic Institute have made quite nice things with Mapserver and shell scripts running some GDAL commands. Should they be computed on-the-fly for the area that is visible on the map? I wonder if it could be possible to query the CONTOUR layer that you already has https://www.mapserver.org/input/vector/contour.html and search the min/max values from there. Or then to improve the contouring algorithm a bit because it is already processing all the pixels. -Jukka Rahkonen- -----Alkuper?inen viesti----- L?hett?j?: mapserver-users Puolesta Stefan Gofferje L?hetetty: tiistai 16. maaliskuuta 2021 12.11 Vastaanottaja: mapserver-users at lists.osgeo.org Aihe: Re: [mapserver-users] Hello from Finland Hi Jeff, On 3/15/21 2:35 PM, Jeff McKenna wrote: > Thanks Stefan for this wonderful introduction, it's nice to hear of > positive experiences to give us all more motivation to continue to > helping to improve MapServer! Welcome to both you and Iliya to the > community. Keep us posted on your activities! > > Oh, once your MapServer backend is plugged-in be sure to add your > weather site to the MapServer gallery, so we can help new users see > what can be done... Thanks for your nice words! I now have mapserver and mapcache running successfully and talking to each other in my home lab. I have set up a number of test layers from different input sources pretty successfully. The documentation and the configuration required some adaption of my thinking processes because it's pretty different from anything else I have ever done before but I love this kind of challenges! Currently I have only two issues which prevent me from doing the swap right now. Thanks to NOAA using a 0-360? grid instead of (-180)-180?, there's some artefacts when reprojecting to epsg:3857. For raster maps this is already addressed in #6106. I ran into an issue which LINE maps with a CONTOUR source which probably is related and opened #6261. Then I have a question... Although it's not strictly necessary, I would like to paint those Hs and Ls in my atmospheric pressure map. Is there a way to run scripts or something like that in mapserver. I could probably run some GDAL stuff externally over the grid files in my download script and create some point overlays which could be in a GROUP with the isoline layer but doing this inside of mapserver would kinda be cooler. And a second question: How is mapserver handling raster files memory-wise? Does it try to load the whole file to the memory at once? My biggest issue during the whole 14 years I run this website always has been balancing preprocessing and map rendering. The more preprocessing I have to do, the longer the whole process of map generation takes but of course I couldn't afford to buy a $100k top-of-the-line server with terabytes of RAM and plenty of CPU cores for the rendering... I'm thinking about my input file strategy at the moment... 1.) One GRIB file for everything (around 1.9GB) 2.) One GRIB file per parameter for all time steps 3.) One GRIB file per time step including all parameters 4.) One GRIB file per parameter for each time step 4. would be the most complex to configure and also would be fun (irony!) to make local forecasts from but probably take the least memory. 2. and 3. are a fairly good compromise with 2 being easier for calculating local forecasts and 3 easier for calculating additional stuff like my "severe weather warning level" 1. would be easiest for rendering and calculating but even at the moment with GrADS I every now and then run into memory issues (leaks?). This project really turns out to be a fun challenge! Stefan -- (o_ Stefan Gofferje | SCLT, MCP, CCSA //\ Reg'd Linux User #247167 | VCP #2263 V_/_ https://www.gofferje.net | https://www.saakeskus.fi _______________________________________________ mapserver-users mailing list mapserver-users at lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users From sln at fcoo.dk Tue Mar 16 05:52:25 2021 From: sln at fcoo.dk (=?iso-8859-1?Q?S=F8ren_Laursen?=) Date: Tue, 16 Mar 2021 12:52:25 +0000 Subject: [mapserver-users] Hello from Finland In-Reply-To: References: Message-ID: Hi Jukka, You might want to look at: https://www.pydap.org/en/latest/index.html It is the backend of ifm.fcoo.dk using netcdf files. grib can be converted to netcdf. S?ren FORSVARSMINISTERIETS MATERIEL- OG INDK?BSSTYRELSE DANISH MINISTRY OF DEFENCE ACQUISITION AND LOGISTICS ORGANISATION Joint GEOMETOC Support Centre Lautrupbjerg 1-5, DK-2750 Ballerup TLF: +45 7281 5627 MOBIL: +45 2961 9652 E-mail: fmi-ma-opd07 at mil.dk www.fmi.dk ________________________________ Fra: mapserver-users p? vegne af Rahkonen Jukka (MML) Sendt: 16. marts 2021 11:54:13 Til: Stefan Gofferje; mapserver-users at lists.osgeo.org Emne: Re: [mapserver-users] Hello from Finland Hi Stefan, When it comes to memory usage with raster files, Mapserver is reading rasters with GDAL so in this case with https://gdal.org/drivers/raster/grib.html. I have never been playing with GRIB files and I do not know how well the format is optimized for features that are needed for making fast map services: fast access to subregions and overview levels. I have no idea how well GRIB files are indexed for random access, it there is difference between png and JPEG2000 encoding, and how well the GDAL GRIB driver can utilize the possibilities that the format offer. The most common raster format that Mapserver services are using is certainly GeoTIFF and when they are tiled, compressed, and have overviews they work pretty well. It is certainly an advantage if you can use the GRIB files directly without conversions. I see that you are using Mapcache in front of Mapserver and it will help a lot if you pre-seed the tiles. If you do not pre-seed then each time when the user is the first visitor on some site the bottle neck will be Mapserver. The Ls/Hs idea is interesting. I do not believe there is anything ready available but I know that my colleagues at the Finnish Geodetic Institute have made quite nice things with Mapserver and shell scripts running some GDAL commands. Should they be computed on-the-fly for the area that is visible on the map? I wonder if it could be possible to query the CONTOUR layer that you already has https://www.mapserver.org/input/vector/contour.html and search the min/max values from there. Or then to improve the contouring algorithm a bit because it is already processing all the pixels. -Jukka Rahkonen- -----Alkuper?inen viesti----- L?hett?j?: mapserver-users Puolesta Stefan Gofferje L?hetetty: tiistai 16. maaliskuuta 2021 12.11 Vastaanottaja: mapserver-users at lists.osgeo.org Aihe: Re: [mapserver-users] Hello from Finland Hi Jeff, On 3/15/21 2:35 PM, Jeff McKenna wrote: > Thanks Stefan for this wonderful introduction, it's nice to hear of > positive experiences to give us all more motivation to continue to > helping to improve MapServer! Welcome to both you and Iliya to the > community. Keep us posted on your activities! > > Oh, once your MapServer backend is plugged-in be sure to add your > weather site to the MapServer gallery, so we can help new users see > what can be done... Thanks for your nice words! I now have mapserver and mapcache running successfully and talking to each other in my home lab. I have set up a number of test layers from different input sources pretty successfully. The documentation and the configuration required some adaption of my thinking processes because it's pretty different from anything else I have ever done before but I love this kind of challenges! Currently I have only two issues which prevent me from doing the swap right now. Thanks to NOAA using a 0-360? grid instead of (-180)-180?, there's some artefacts when reprojecting to epsg:3857. For raster maps this is already addressed in #6106. I ran into an issue which LINE maps with a CONTOUR source which probably is related and opened #6261. Then I have a question... Although it's not strictly necessary, I would like to paint those Hs and Ls in my atmospheric pressure map. Is there a way to run scripts or something like that in mapserver. I could probably run some GDAL stuff externally over the grid files in my download script and create some point overlays which could be in a GROUP with the isoline layer but doing this inside of mapserver would kinda be cooler. And a second question: How is mapserver handling raster files memory-wise? Does it try to load the whole file to the memory at once? My biggest issue during the whole 14 years I run this website always has been balancing preprocessing and map rendering. The more preprocessing I have to do, the longer the whole process of map generation takes but of course I couldn't afford to buy a $100k top-of-the-line server with terabytes of RAM and plenty of CPU cores for the rendering... I'm thinking about my input file strategy at the moment... 1.) One GRIB file for everything (around 1.9GB) 2.) One GRIB file per parameter for all time steps 3.) One GRIB file per time step including all parameters 4.) One GRIB file per parameter for each time step 4. would be the most complex to configure and also would be fun (irony!) to make local forecasts from but probably take the least memory. 2. and 3. are a fairly good compromise with 2 being easier for calculating local forecasts and 3 easier for calculating additional stuff like my "severe weather warning level" 1. would be easiest for rendering and calculating but even at the moment with GrADS I every now and then run into memory issues (leaks?). This project really turns out to be a fun challenge! Stefan -- (o_ Stefan Gofferje | SCLT, MCP, CCSA //\ Reg'd Linux User #247167 | VCP #2263 V_/_ https://www.gofferje.net | https://www.saakeskus.fi _______________________________________________ mapserver-users mailing list mapserver-users at lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users _______________________________________________ mapserver-users mailing list mapserver-users at lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at home.gofferje.net Tue Mar 16 08:15:35 2021 From: lists at home.gofferje.net (Stefan Gofferje) Date: Tue, 16 Mar 2021 17:15:35 +0200 Subject: [mapserver-users] Hello from Finland In-Reply-To: References: Message-ID: <53d5a61d-5e68-8273-0f29-35dcff29ed88@home.gofferje.net> Terve Jukka, On 3/16/21 12:54 PM, Rahkonen Jukka (MML) wrote: > When it comes to memory usage with raster files, Mapserver is reading rasters with GDAL so in this case with https://gdal.org/drivers/raster/grib.html. I have never been playing with GRIB files and I do not know how well the format is optimized for features that are needed for making fast map services: fast access to subregions and overview levels. I have no idea how well GRIB files are indexed for random access, it there is difference between png and JPEG2000 encoding, and how well the GDAL GRIB driver can utilize the possibilities that the format offer. That sounds good. IIRC GDAL is pretty mature with its memory management. At least I haven't manged to crash it yet :D. > The most common raster format that Mapserver services are using is certainly GeoTIFF and when they are tiled, compressed, and have overviews they work pretty well. It is certainly an advantage if you can use the GRIB files directly without conversions. I see that you are using Mapcache in front of Mapserver and it will help a lot if you pre-seed the tiles. If you do not pre-seed then each time when the user is the first visitor on some site the bottle neck will be Mapserver. That is a good point! Especially as the forecast data is updated every 6 hours. I will have to integrate that into my update script somewhow once I got there. > The Ls/Hs idea is interesting. I do not believe there is anything ready available but I know that my colleagues at the Finnish Geodetic Institute have made quite nice things with Mapserver and shell scripts running some GDAL commands. Should they be computed on-the-fly for the area that is visible on the map? I'm doing some GDAL stuff in shell scripts for beta.saakeskus.fi at the moment. You can indeed do some fun stuff with the GDAL tools. In the long run, I was thinking of changing from bash scripts to Python, though, because then I can make some more calculations than are possible just with the GDAL tools. Yes, I was thinking if it was possible to do that inside of mapserver. Finding _the_ highest and lowest point is fairly trivial with GDAL. IIRC, gdalinfo can output that info by default. A little more challenging is finding all points which are highest or lowest among their neighbors. That's what's needed for a weather map because usually, you have more than one high and one low area on one map. -Stefan -- (o_ Stefan Gofferje | SCLT, MCP, CCSA //\ Reg'd Linux User #247167 | VCP #2263 V_/_ https://www.gofferje.net | https://www.saakeskus.fi From jukka.rahkonen at maanmittauslaitos.fi Tue Mar 16 13:23:08 2021 From: jukka.rahkonen at maanmittauslaitos.fi (Rahkonen Jukka (MML)) Date: Tue, 16 Mar 2021 20:23:08 +0000 Subject: [mapserver-users] Hello from Finland Message-ID: Hi, -----Alkuper?inen viesti----- Stefan Gofferje wrote: > Terve Jukka, > On 3/16/21 12:54 PM, Rahkonen Jukka (MML) wrote: >> When it comes to memory usage with raster files, Mapserver is reading rasters with GDAL so in this case with https://gdal.org/drivers/raster/grib.html. I have never been playing with GRIB files and I do not know how well the format is optimized for features that are needed for making fast map services: fast access to subregions and overview levels. I have no idea how well GRIB files are indexed for random access, it there is difference between png and JPEG2000 encoding, and how well the GDAL GRIB driver can utilize the possibilities that the format offer. > That sounds good. IIRC GDAL is pretty mature with its memory management. >At least I haven't manged to crash it yet :D. >> The most common raster format that Mapserver services are using is certainly GeoTIFF and when they are tiled, compressed, and have overviews they work pretty well. It is certainly an advantage if you can use the GRIB files directly without conversions. I see that you are using Mapcache in front of Mapserver and it will help a lot if you pre-seed the tiles. If you do not pre-seed then each time when the user is the first visitor on some site the bottle neck will be Mapserver. > That is a good point! Especially as the forecast data is updated every 6 hours. I will have to integrate that into my update script somewhow once I got there. >> The Ls/Hs idea is interesting. I do not believe there is anything ready available but I know that my colleagues at the Finnish Geodetic Institute have made quite nice things with Mapserver and shell scripts running some GDAL commands. Should they be computed on-the-fly for the area that is visible on the map? > I'm doing some GDAL stuff in shell scripts for beta.saakeskus.fi at the moment. You can indeed do some fun stuff with the GDAL tools. In the long run, I was thinking of changing from bash scripts to Python, though, because then I can make some more calculations than are possible just with the GDAL tools. > Yes, I was thinking if it was possible to do that inside of mapserver. Finding _the_ highest and lowest point is fairly trivial with GDAL. IIRC, gdalinfo can output that info by default. A little more challenging is finding all points which are highest or lowest among their neighbors. That's what's needed for a weather map because usually, you have more than one high and one low area on one map. Gdalinfo -stats reports the min/max values but not their locations. As far as I know there is no GDAL tool or Python script in the GDAL samples that can do that. Users have been asking for such tool in the mailing lists and in gis.stackexchange so if you find of create such utility please let me know. For finding the local min/max areas you could perhaps try the gdaldem tools https://gdal.org/programs/gdaldem.html. Areas with slope=0 should be what you are searching. -Jukka- > -Stefan -- (o_ Stefan Gofferje | SCLT, MCP, CCSA //\ Reg'd Linux User #247167 | VCP #2263 V_/_ https://www.gofferje.net | https://www.saakeskus.fi _______________________________________________ mapserver-users mailing list mapserver-users at lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users From lists at home.gofferje.net Wed Mar 17 06:59:01 2021 From: lists at home.gofferje.net (Stefan Gofferje) Date: Wed, 17 Mar 2021 15:59:01 +0200 Subject: [mapserver-users] Hello from Finland In-Reply-To: References: Message-ID: On 3/16/21 10:23 PM, Rahkonen Jukka (MML) wrote: > Gdalinfo -stats reports the min/max values but not their locations. As far as I know there is no GDAL tool or Python script in the GDAL samples that can do that. Users have been asking for such tool in the mailing lists and in gis.stackexchange so if you find of create such utility please let me know. I'm fairly sure, I ran across something some time ago when I was googling for a solution for the H and L issue. I'll try to find it. > For finding the local min/max areas you could perhaps try the gdaldem tools https://gdal.org/programs/gdaldem.html. Areas with slope=0 should be what you are searching. I'll look into it, thanks! -Stefan -- (o_ Stefan Gofferje | SCLT, MCP, CCSA //\ Reg'd Linux User #247167 | VCP #2263 V_/_ https://www.gofferje.net | https://www.saakeskus.fi From zmitserk at mail.ru Tue Mar 23 00:54:33 2021 From: zmitserk at mail.ru (=?UTF-8?B?Wm1pdHNlciBLb3podWto?=) Date: Tue, 23 Mar 2021 10:54:33 +0300 Subject: [mapserver-users] =?utf-8?q?data_parameter_in_tileindex?= Message-ID: <1616486073.47122277@f338.i.mail.ru> Dear mapserver users, I need to specify data parameter at the location item of tileindex. So far, I have tried many possibilities but none works: ?host=127.0.0.1 dbname=database?user=admin?password=admin?DATA "geom from table using unique id using srid=4326"? or host=127.0.0.1 dbname=database?user=admin?password=admin data="geom from table using unique id using srid=4326" etc. Still I get an error : msDrawMap(): Image handling error. Failed to draw layer named 'layer'. msPostGISLayerOpen(): Query error. Nothing specified in DATA statement. Could someone please tell in which format data statement should be? In documentation it is written vaguely: Normally the location should contain the path to the tile file relative to the shapepath, not relative to the tileindex itself. If the DATA parameter contains a value then it is added to the end of the location. ? -- Zmitser Kozhukh -------------- next part -------------- An HTML attachment was scrubbed... URL: From boolean10001 at yahoo.com Tue Mar 23 20:53:13 2021 From: boolean10001 at yahoo.com (Carlos Ruiz) Date: Wed, 24 Mar 2021 03:53:13 +0000 (UTC) Subject: [mapserver-users] data parameter in tileindex In-Reply-To: <1616486073.47122277@f338.i.mail.ru> References: <1616486073.47122277@f338.i.mail.ru> Message-ID: <355834207.4624423.1616557993099@mail.yahoo.com> Zmitser, You have to have the following on a MAP file: LAYER ... CONNECTION "host= ... password="CONNECTIONTYPE DATA " FROM USING UNIQUE USING SRID=" ... END On DATA you must specify the name of the geometry column and the table, if you're using a database. If you are using a shape file you just have to include the DATA keyword with the path and the name of the SHP file. On Tuesday, March 23, 2021, 2:34:48 AM CST, Zmitser Kozhukh via mapserver-users wrote: Dear mapserver users, I need to specify data parameter at the location item of tileindex.So far, I have tried many possibilities but none works: ?host=127.0.0.1 dbname=database?user=admin?password=admin?DATA "geom from table using unique id using srid=4326"?orhost=127.0.0.1 dbname=database?user=admin?password=admin data="geom from table using unique id using srid=4326"etc.Still I get an error :msDrawMap(): Image handling error. Failed to draw layer named 'layer'. msPostGISLayerOpen(): Query error. Nothing specified in DATA statement.Could someone please tell in which format data statement should be?In documentation it is written vaguely:Normally the location should contain the path to the tile file relative to the shapepath, not relative to the tileindex itself. If the DATA parameter contains a value then it is added to the end of the location.?-- Zmitser Kozhukh_______________________________________________ mapserver-users mailing list mapserver-users at lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From zmitserk at mail.ru Wed Mar 24 00:02:45 2021 From: zmitserk at mail.ru (=?UTF-8?B?Wm1pdHNlciBLb3podWto?=) Date: Wed, 24 Mar 2021 10:02:45 +0300 Subject: [mapserver-users] =?utf-8?q?data_parameter_in_tileindex?= In-Reply-To: <355834207.4624423.1616557993099@mail.yahoo.com> References: <1616486073.47122277@f338.i.mail.ru> <355834207.4624423.1616557993099@mail.yahoo.com> Message-ID: <1616569365.339414664@f491.i.mail.ru> Dear Carlos, it is not about mapfile, it is in location cell of tileindex. It is one cell in table (database table,?dbf file etc.), and I need to put there both connection and data details. The question is how to make it so that mapserver parses it correctly. Sincerely, Zmitser >?????, 24 ????? 2021, 6:53 +03:00 ?? Carlos Ruiz : >? >Zmitser, >? >You have to have the following on a MAP file: >? >LAYER >? >... >? >CONNECTION "host= ... password=" >CONNECTIONTYPE >DATA " FROM
USING UNIQUE USING SRID=" >? >... >? >END >? >On DATA you must specify the name of the geometry column and the table, if you're using a database. If you are using a shape file you just have to include the DATA keyword with the path and the name of the SHP file. >? >? >On Tuesday, March 23, 2021, 2:34:48 AM CST, Zmitser Kozhukh via mapserver-users < mapserver-users at lists.osgeo.org > wrote: >? >? >Dear mapserver users, I need to specify data parameter at the location item of tileindex. >So far, I have tried many possibilities but none works: >?host=127.0.0.1 dbname=database?user=admin?password=admin?DATA "geom from table using unique id using srid=4326"? >or >host=127.0.0.1 dbname=database?user=admin?password=admin data="geom from table using unique id using srid=4326" >etc. >Still I get an error : >msDrawMap(): Image handling error. Failed to draw layer named 'layer'. msPostGISLayerOpen(): Query error. Nothing specified in DATA statement. >Could someone please tell in which format data statement should be? >In documentation it is written vaguely: >Normally the location should contain the path to the tile file relative to the shapepath, not relative to the tileindex itself. If the DATA parameter contains a value then it is added to the end of the location. >? >-- >Zmitser Kozhukh _______________________________________________ >mapserver-users mailing list >mapserver-users at lists.osgeo.org >https://lists.osgeo.org/mailman/listinfo/mapserver-users ? ? -- Zmitser Kozhukh ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From emacgillavry at hotmail.com Wed Mar 24 00:45:45 2021 From: emacgillavry at hotmail.com (Edward Mac Gillavry) Date: Wed, 24 Mar 2021 07:45:45 +0000 Subject: [mapserver-users] data parameter in tileindex In-Reply-To: <1616569365.339414664@f491.i.mail.ru> References: <1616486073.47122277@f338.i.mail.ru> <355834207.4624423.1616557993099@mail.yahoo.com>, <1616569365.339414664@f491.i.mail.ru> Message-ID: Zmister, Check out the GH repo of the Amsterdam municipality. The MAP-file for 2020 is at https://github.com/Amsterdam/mapserver/blob/master/lufo2020.map where you can see the DATA connection to the Shape file that holds the tile index. The Shape files for the various years are in the folder https://github.com/Amsterdam/mapserver/tree/master/luchtfoto. These were created using gdaltindex as you can see in the bash script at https://github.com/Amsterdam/mapserver/blob/master/lufopyramids.sh. That's where you pass the location of the GeoTIFFs to populate the column in the Shape files so MapServer can open the required GeoTIFFs to generate an image Hopefully these practical examples guide you. Best, Edward Outlook voor Android downloaden ________________________________ From: mapserver-users on behalf of Zmitser Kozhukh via mapserver-users Sent: Wednesday, March 24, 2021 8:02:45 AM To: Carlos Ruiz Cc: mapserver-users at lists.osgeo.org Subject: Re: [mapserver-users] data parameter in tileindex Dear Carlos, it is not about mapfile, it is in location cell of tileindex. It is one cell in table (database table, dbf file etc.), and I need to put there both connection and data details. The question is how to make it so that mapserver parses it correctly. Sincerely, Zmitser ?????, 24 ????? 2021, 6:53 +03:00 ?? Carlos Ruiz : Zmitser, You have to have the following on a MAP file: LAYER ... CONNECTION "host= ... password=" CONNECTIONTYPE DATA " FROM
USING UNIQUE USING SRID=" ... END On DATA you must specify the name of the geometry column and the table, if you're using a database. If you are using a shape file you just have to include the DATA keyword with the path and the name of the SHP file. On Tuesday, March 23, 2021, 2:34:48 AM CST, Zmitser Kozhukh via mapserver-users > wrote: Dear mapserver users, I need to specify data parameter at the location item of tileindex. So far, I have tried many possibilities but none works: ?host=127.0.0.1 dbname=database user=admin password=admin DATA "geom from table using unique id using srid=4326"? or host=127.0.0.1 dbname=database user=admin password=admin data="geom from table using unique id using srid=4326" etc. Still I get an error : msDrawMap(): Image handling error. Failed to draw layer named 'layer'. msPostGISLayerOpen(): Query error. Nothing specified in DATA statement. Could someone please tell in which format data statement should be? In documentation it is written vaguely: Normally the location should contain the path to the tile file relative to the shapepath, not relative to the tileindex itself. If the DATA parameter contains a value then it is added to the end of the location. -- Zmitser Kozhukh _______________________________________________ mapserver-users mailing list mapserver-users at lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users -- Zmitser Kozhukh -------------- next part -------------- An HTML attachment was scrubbed... URL: From jukka.rahkonen at maanmittauslaitos.fi Wed Mar 24 00:47:49 2021 From: jukka.rahkonen at maanmittauslaitos.fi (Rahkonen Jukka (MML)) Date: Wed, 24 Mar 2021 07:47:49 +0000 Subject: [mapserver-users] data parameter in tileindex Message-ID: <78fe3691efc44d9e80b34dda804ab481@maanmittauslaitos.fi> Hi, Obviously you are playing with vector data, not rasters, and you are trying to make a ogrtileindex instead of gdaltindex. I know, technically there is no difference. I have used ogrtileindex with great success with shapefiles. I have never tried to do the same with PostGIS data and I can only wish you luck with your experiments. One thing that came into my mind it that perhaps OGR connection to PostGIS would work better than the native one with tileindex. -Jukka Rahkonen- L?hett?j?: mapserver-users > Puolesta Zmitser Kozhukh via mapserver-users L?hetetty: keskiviikko 24. maaliskuuta 2021 9.03 Vastaanottaja: Carlos Ruiz > Kopio: mapserver-users at lists.osgeo.org Aihe: Re: [mapserver-users] data parameter in tileindex Dear Carlos, it is not about mapfile, it is in location cell of tileindex. It is one cell in table (database table, dbf file etc.), and I need to put there both connection and data details. The question is how to make it so that mapserver parses it correctly. Sincerely, Zmitser ?????, 24 ????? 2021, 6:53 +03:00 ?? Carlos Ruiz >: Zmitser, You have to have the following on a MAP file: LAYER ... CONNECTION "host= ... password=" CONNECTIONTYPE DATA " FROM
USING UNIQUE USING SRID=" ... END On DATA you must specify the name of the geometry column and the table, if you're using a database. If you are using a shape file you just have to include the DATA keyword with the path and the name of the SHP file. On Tuesday, March 23, 2021, 2:34:48 AM CST, Zmitser Kozhukh via mapserver-users > wrote: Dear mapserver users, I need to specify data parameter at the location item of tileindex. So far, I have tried many possibilities but none works: ?host=127.0.0.1 dbname=database user=admin password=admin DATA "geom from table using unique id using srid=4326"? or host=127.0.0.1 dbname=database user=admin password=admin data="geom from table using unique id using srid=4326" etc. Still I get an error : msDrawMap(): Image handling error. Failed to draw layer named 'layer'. msPostGISLayerOpen(): Query error. Nothing specified in DATA statement. Could someone please tell in which format data statement should be? In documentation it is written vaguely: Normally the location should contain the path to the tile file relative to the shapepath, not relative to the tileindex itself. If the DATA parameter contains a value then it is added to the end of the location. -- Zmitser Kozhukh _______________________________________________ mapserver-users mailing list mapserver-users at lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users -- Zmitser Kozhukh -------------- next part -------------- An HTML attachment was scrubbed... URL: From zmitserk at mail.ru Wed Mar 24 01:32:33 2021 From: zmitserk at mail.ru (=?UTF-8?B?Wm1pdHNlciBLb3podWto?=) Date: Wed, 24 Mar 2021 11:32:33 +0300 Subject: [mapserver-users] =?utf-8?q?data_parameter_in_tileindex?= In-Reply-To: <69e4edd83e2e47ecb7020af6ea35b462@maanmittauslaitos.fi> References: <69e4edd83e2e47ecb7020af6ea35b462@maanmittauslaitos.fi> Message-ID: <1616574753.910807289@f104.i.mail.ru> Dear Jukka, you right with OGR it works. If I type something like: "PG:"dbname=database?host=127.0.0.1 port=5432 user=admin?password=admin?tables=objects_in_countryA(geom)"" It works. Because for OGR layer types, providing connection is enough. But for PostGIS , also the data part need to be provided separately. In fact , it is just a matter of interest for me. What will be faster , have it with OGR PostGIS driver or with native mapserver postgis driver (I assume if type of connection PostGIS mapserver uses its own driver for reading features). I will let you know after I find out the way. Sincerely, Zmitser >?????, 24 ????? 2021, 10:43 +03:00 ?? Rahkonen Jukka (MML) : >? >Hi, >? >Obviously you are playing with vector data, not rasters, and you are trying to make a ogrtileindex instead of gdaltindex. I know, technically there is no difference. >? >I have used ogrtileindex with great success with shapefiles. I have never tried to do the same with PostGIS data and I can only wish you luck with your experiments. One thing that came into my mind it that perhaps OGR connection to PostGIS would work better than the native one with tileindex. >? >-Jukka Rahkonen- >? >L?hett?j?: mapserver-users < mapserver-users-bounces at lists.osgeo.org > Puolesta Zmitser Kozhukh via mapserver-users >L?hetetty: keskiviikko 24. maaliskuuta 2021 9.03 >Vastaanottaja: Carlos Ruiz < boolean10001 at yahoo.com > >Kopio: mapserver-users at lists.osgeo.org >Aihe: Re: [mapserver-users] data parameter in tileindex >? >Dear Carlos, it is not about mapfile, it is in location cell of tileindex. It is one cell in table (database table,?dbf file etc.), and I need to put there both connection and data details. The question is how to make it so that mapserver parses it correctly. >Sincerely, Zmitser >? >>?????, 24 ????? 2021, 6:53 +03:00 ?? Carlos Ruiz < boolean10001 at yahoo.com >: >>? >>Zmitser, >>? >>You have to have the following on a MAP file: >>? >>LAYER >>? >>... >>? >>CONNECTION "host= ... password=" >>CONNECTIONTYPE >>DATA " FROM
USING UNIQUE USING SRID=" >>? >>... >>? >>END >>? >>On DATA you must specify the name of the geometry column and the table, if you're using a database. If you are using a shape file you just have to include the DATA keyword with the path and the name of the SHP file. >>? >>? >>On Tuesday, March 23, 2021, 2:34:48 AM CST, Zmitser Kozhukh via mapserver-users < mapserver-users at lists.osgeo.org > wrote: >>? >>? >>Dear mapserver users, I need to specify data parameter at the location item of tileindex. >>So far, I have tried many possibilities but none works: >>?host=127.0.0.1 dbname=database?user=admin?password=admin?DATA "geom from table using unique id using srid=4326"? >>or >>host=127.0.0.1 dbname=database?user=admin?password=admin data="geom from table using unique id using srid=4326" >>etc. >>Still I get an error : >>msDrawMap(): Image handling error. Failed to draw layer named 'layer'. msPostGISLayerOpen(): Query error. Nothing specified in DATA statement. >>Could someone please tell in which format data statement should be? >>In documentation it is written vaguely: >>Normally the location should contain the path to the tile file relative to the shapepath, not relative to the tileindex itself. If the DATA parameter contains a value then it is added to the end of the location. >>? >>-- >>Zmitser Kozhukh >>_______________________________________________ >>mapserver-users mailing list >>mapserver-users at lists.osgeo.org >>https://lists.osgeo.org/mailman/listinfo/mapserver-users >? >? >-- >Zmitser Kozhukh >? ? ? -- Zmitser Kozhukh ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From zmitserk at mail.ru Wed Mar 24 01:58:00 2021 From: zmitserk at mail.ru (=?UTF-8?B?Wm1pdHNlciBLb3podWto?=) Date: Wed, 24 Mar 2021 11:58:00 +0300 Subject: [mapserver-users] =?utf-8?q?data_parameter_in_tileindex?= In-Reply-To: References: <1616486073.47122277@f338.i.mail.ru> <1616569365.339414664@f491.i.mail.ru> Message-ID: <1616576279.909857355@f152.i.mail.ru> Thank you very much, Edward, for the suggestion. But I use tiles of vector data. And currently the problem is how to put postgis connection and data parameter in one (so that mapserver parses it correctly from location column?of tileindex) . With best regards, Zmitser ? >?????, 24 ????? 2021, 10:45 +03:00 ?? Edward Mac Gillavry : >? >Zmister, >? >Check out the GH repo of the Amsterdam municipality. The MAP-file for 2020 is at? https://github.com/Amsterdam/mapserver/blob/master/lufo2020.map ?where you can see the DATA connection to the Shape file that holds the tile index. The Shape files for the various years are in the folder? https://github.com/Amsterdam/mapserver/tree/master/luchtfoto . These were created using? gdaltindex? as you can see in the bash script at? https://github.com/Amsterdam/mapserver/blob/master/lufopyramids.sh . That's where you pass the location of the GeoTIFFs to populate the column in the Shape files so MapServer can open the required GeoTIFFs to generate an image? Hopefully these practical examples guide you. >? >Best, >? >Edward >? >? >? >? >Outlook voor Android downloaden >---------------------------------------------------------------------- >From: mapserver-users < mapserver-users-bounces at lists.osgeo.org > on behalf of Zmitser Kozhukh via mapserver-users < mapserver-users at lists.osgeo.org > >Sent: Wednesday, March 24, 2021 8:02:45 AM >To: Carlos Ruiz < boolean10001 at yahoo.com > >Cc: mapserver-users at lists.osgeo.org < mapserver-users at lists.osgeo.org > >Subject: Re: [mapserver-users] data parameter in tileindex >? >Dear Carlos, it is not about mapfile, it is in location cell of tileindex. It is one cell in table (database table,?dbf file etc.), and I need to put there both connection and data details. The question is how to make it so that mapserver parses it correctly. >Sincerely, Zmitser ? >>?????, 24 ????? 2021, 6:53 +03:00 ?? Carlos Ruiz < boolean10001 at yahoo.com >: >>? >>Zmitser, >>? >>You have to have the following on a MAP file: >>? >>LAYER >>? >>... >>? >>CONNECTION "host= ... password=" >>CONNECTIONTYPE >>DATA " FROM
USING UNIQUE USING SRID=" >>? >>... >>? >>END >>? >>On DATA you must specify the name of the geometry column and the table, if you're using a database. If you are using a shape file you just have to include the DATA keyword with the path and the name of the SHP file. >>? >>? >>On Tuesday, March 23, 2021, 2:34:48 AM CST, Zmitser Kozhukh via mapserver-users < mapserver-users at lists.osgeo.org > wrote: >>? >>? >>Dear mapserver users, I need to specify data parameter at the location item of tileindex. >>So far, I have tried many possibilities but none works: >>?host=127.0.0.1 dbname=database?user=admin?password=admin?DATA "geom from table using unique id using srid=4326"? >>or >>host=127.0.0.1 dbname=database?user=admin?password=admin data="geom from table using unique id using srid=4326" >>etc. >>Still I get an error : >>msDrawMap(): Image handling error. Failed to draw layer named 'layer'. msPostGISLayerOpen(): Query error. Nothing specified in DATA statement. >>Could someone please tell in which format data statement should be? >>In documentation it is written vaguely: >>Normally the location should contain the path to the tile file relative to the shapepath, not relative to the tileindex itself. If the DATA parameter contains a value then it is added to the end of the location. >>? >>-- >>Zmitser Kozhukh _______________________________________________ >>mapserver-users mailing list >>mapserver-users at lists.osgeo.org >>https://lists.osgeo.org/mailman/listinfo/mapserver-users >? >? >-- >Zmitser Kozhukh >? ? ? -- Zmitser Kozhukh ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jmckenna at gatewaygeomatics.com Wed Mar 24 07:39:01 2021 From: jmckenna at gatewaygeomatics.com (Jeff McKenna) Date: Wed, 24 Mar 2021 11:39:01 -0300 Subject: [mapserver-users] data parameter in tileindex In-Reply-To: <1616569365.339414664@f491.i.mail.ru> References: <1616486073.47122277@f338.i.mail.ru> <355834207.4624423.1616557993099@mail.yahoo.com> <1616569365.339414664@f491.i.mail.ru> Message-ID: Spurred on by Zmitser, I've given some much needed love to the tileindex examples (see https://mapserver.org/optimization/tileindex.html#long-ogrtindex-example ). It doesn't answer Zmitser's exact questions but it hopefully helps the next person understand how to implement. Also, pull requests are encouraged if you want to add more, to the documentation repository at https://github.com/MapServer/MapServer-documentation Side note: this latest effort was kind of special to me, as in fact back in year 2004 I did a similar effort to give examples of tile4ms, notice the QGIS version 0.6 screen captures that I had made back in 2004, and now these tileindex examples use a more recent QGIS version of 3.18.1 ha. The circle of life. https://mapserver.org/utilities/tile4ms.html#long-example -jeff -- Jeff McKenna GatewayGeo: Developers of MS4W, MapServer Consulting and Training co-founder of FOSS4G http://gatewaygeo.com/ On 2021-03-24 4:02 a.m., Zmitser Kozhukh via mapserver-users wrote: > Dear Carlos, it is not about mapfile, it is in location cell of > tileindex. It is one cell in table (database table,?dbf file etc.), and > I need to put there both connection and data details. The question is > how to make it so that mapserver parses it correctly. > Sincerely, Zmitser > > ?????, 24 ????? 2021, 6:53 +03:00 ?? Carlos Ruiz > : > Zmitser, > You have to have the following on a MAP file: > LAYER > ... > CONNECTION "host= ... password=" > CONNECTIONTYPE > DATA " FROM
USING UNIQUE USING > SRID=" > ... > END > On DATA you must specify the name of the geometry column and the > table, if you're using a database. If you are using a shape file you > just have to include the DATA keyword with the path and the name of > the SHP file. > On Tuesday, March 23, 2021, 2:34:48 AM CST, Zmitser Kozhukh via > mapserver-users > wrote: > Dear mapserver users, I need to specify data parameter at the > location item of tileindex. > So far, I have tried many possibilities but none works: > ?host=127.0.0.1 dbname=database?user=admin?password=admin?DATA "geom > from table using unique id using srid=4326"? > or > host=127.0.0.1 dbname=database?user=admin?password=admin data="geom > from table using unique id using srid=4326" > etc. > Still I get an error : > msDrawMap(): Image handling error. Failed to draw layer named > 'layer'. msPostGISLayerOpen(): Query error. Nothing specified in > DATA statement. > Could someone please tell in which format data statement should be? > In documentation it is written vaguely: > Normally the location should contain the path to the tile file > relative to the shapepath, not relative to the tileindex itself. If > the DATA parameter contains a value then it is added to the end of > the location. > -- > Zmitser Kozhukh > _______________________________________________ > mapserver-users mailing list > mapserver-users at lists.osgeo.org > > https://lists.osgeo.org/mailman/listinfo/mapserver-users > > > -- > Zmitser Kozhukh > > _______________________________________________ > mapserver-users mailing list > mapserver-users at lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/mapserver-users > From lars.schylberg at blixtmail.se Wed Mar 24 13:22:24 2021 From: lars.schylberg at blixtmail.se (Lars Schylberg) Date: Wed, 24 Mar 2021 21:22:24 +0100 Subject: [mapserver-users] data parameter in tileindex In-Reply-To: References: <1616486073.47122277@f338.i.mail.ru> <355834207.4624423.1616557993099@mail.yahoo.com> <1616569365.339414664@f491.i.mail.ru> Message-ID: I also have this fascination about tile indexes.I have one example how to use shp2tile and create faster setup if You have large data files. It shows how to set up ogrtindex. I still use this script in some projects. I guess I must have shared it on the users-list a couple of years ago, since it is on my GIST: https://gist.github.com/LarsSchy/644b37ab2b1e58b48808bfb2b57d0359 The extra optimization to create an extra index on the main attribute used for filtering (expressions) can also give some additional speed improvements. I also have my own twist on tileindexes for certain rasterdata sets.I talked about that in my talk in Bucharest at FOSS4G 2019.In the second part of the talk, I described how to do ?tight tile indexes?. The video is still available. And to Jeff, I know, I should try and get this up on the Wiki sometime. Have fun / Lars Schylberg ----- Den 2021-03-24 kl. 15:39, skrev Jeff McKenna: .. -------------- next part -------------- An HTML attachment was scrubbed... URL: From efujioka at duke.edu Thu Mar 25 21:00:54 2021 From: efujioka at duke.edu (Fujioka (Duke)) Date: Fri, 26 Mar 2021 00:00:54 -0400 Subject: [mapserver-users] BINVALS is supported by PostGIS 1.5? Message-ID: <1e78dcff-a341-0fce-08cb-f56a3f369b4e@duke.edu> Hello Mapserver experts! I googled BINVALS but couldn't find any useful information. What I want to achieve is to include a placeholder in DATA and substitute it using BINVALS. Simplified example: LAYER CONNECTION "user=xxx password=xxx dbname=xxx host=xxx" CONNECTIONTYPE POSTGIS DEBUG 5 DATA "geom from (select column1, geom, count(*) from table1 where column1 = $1 group by column1, geom) AS foo USING UNIQUE column1 USING SRID=4326" BINDVALS ?????? "1"? "10" END END I expect the final query would be "select ... where column1 = 10 group by ...". But the map image does not show anything. It's a gray image. When I set "DEBUG 5" at the layer level and took a look at the debug info, the placeholder was not substituted (it was still "where column1 = $1..."). So, it seems BINDVALS is not working. Any tips on this? Does the complexity of the query matter (e.g. can't use GROUP BY or JOIN)? Should the query in the debug info be the final one with the placeholder substituted with the value specified in BINDVALS? I use Mapserver 6.4 & PostGIS 1.5 & PostgreSQL 9.1. Is BINDVALS not supported for these versions? Also, how can I change the bind value using PHP Mapscript (say, from "10" to "20")? Thanks, -- ======================== Ei Fujioka Research Scientist Nicholas School Duke University (919) 684-3518 efujioka at duke.edu ======================== From jukka.rahkonen at maanmittauslaitos.fi Fri Mar 26 01:08:59 2021 From: jukka.rahkonen at maanmittauslaitos.fi (Rahkonen Jukka (MML)) Date: Fri, 26 Mar 2021 08:08:59 +0000 Subject: [mapserver-users] BINVALS is supported by PostGIS 1.5? Message-ID: <5b80cb68fa5741cd9b470b0768527812@maanmittauslaitos.fi> Hi, Developers have paid attention to BINDVALS recently https://github.com/MapServer/MapServer/issues/6236 and they should work as in your mapfile. The usage example is about string "1" "Nova Scotia". I wonder how a numeric bindvalue should be expressed in mapfile. Could it be that in the "1" "10" case the value gets converted into SQL string '10'? -Jukka Rahkonen- -----Alkuper?inen viesti----- L?hett?j?: mapserver-users Puolesta Fujioka (Duke) L?hetetty: perjantai 26. maaliskuuta 2021 6.01 Vastaanottaja: mapserver-users at lists.osgeo.org Aihe: [mapserver-users] BINVALS is supported by PostGIS 1.5? Hello Mapserver experts! I googled BINVALS but couldn't find any useful information. What I want to achieve is to include a placeholder in DATA and substitute it using BINVALS. Simplified example: LAYER CONNECTION "user=xxx password=xxx dbname=xxx host=xxx" CONNECTIONTYPE POSTGIS DEBUG 5 DATA "geom from (select column1, geom, count(*) from table1 where column1 = $1 group by column1, geom) AS foo USING UNIQUE column1 USING SRID=4326" BINDVALS ?????? "1"? "10" END END I expect the final query would be "select ... where column1 = 10 group by ...". But the map image does not show anything. It's a gray image. When I set "DEBUG 5" at the layer level and took a look at the debug info, the placeholder was not substituted (it was still "where column1 = $1..."). So, it seems BINDVALS is not working. Any tips on this? Does the complexity of the query matter (e.g. can't use GROUP BY or JOIN)? Should the query in the debug info be the final one with the placeholder substituted with the value specified in BINDVALS? I use Mapserver 6.4 & PostGIS 1.5 & PostgreSQL 9.1. Is BINDVALS not supported for these versions? Also, how can I change the bind value using PHP Mapscript (say, from "10" to "20")? Thanks, -- ======================== Ei Fujioka Research Scientist Nicholas School Duke University (919) 684-3518 efujioka at duke.edu ======================== _______________________________________________ mapserver-users mailing list mapserver-users at lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users From sethg at geographika.co.uk Sat Mar 27 14:11:00 2021 From: sethg at geographika.co.uk (Seth G) Date: Sat, 27 Mar 2021 22:11:00 +0100 Subject: [mapserver-users] Layer Attribute Metadata Message-ID: Hi all, Is anyone aware of any OGC service which can return comments for a layer attribute/field? The WFS DescribeFeatureType [1] seems the most likely candidate but has no "description" or equivalent property at the attribute level. Attributes are returned as follows: The docs for the WFS layer object https://www.mapserver.org/ogc/wfs_server.html#layer-object allow the following to be set for each attribute: gml_[item name]_alias gml_[item name]_precision gml_[item name]_type gml_[item name]_value gml_[item name]_width It would be nice if there was an option to set something like the following, and have it returned as part of the layer metadata: gml_[item name]_description This way the automated parts of the attribute (type, precision etc.) could be included along with a description of what the field contains. Does anyone have a way of handling this with MapServer? Seth [1] http://opengeospatial.github.io/e-learning/wfs/text/operations.html#describefeaturetype -- web:http://geographika.co.uk twitter: @geographika From sdlime at gmail.com Tue Mar 30 11:24:54 2021 From: sdlime at gmail.com (Steve Lime) Date: Tue, 30 Mar 2021 13:24:54 -0500 Subject: [mapserver-users] =?utf-8?q?Security_Advisory_=E2=80=93_Limiting?= =?utf-8?q?_Mapfile_Access?= Message-ID: Hi all: This is an important reminder that, as part of a secure deployment, it is important to limit MapServer CGI access to mapfiles. The MapServer CGI has long supported the use of environment variables as a primary mechanism to do this. If you haven?t implemented these controls then that constitutes undue risk that is easily mitigated and we strongly encourage you to do so as soon as possible. It?s also a great time to review those settings if you already have them in place as we?ve recently updated regex examples related to MS_MAP_PATTERN to limit path traversal. Relevant documentation can be found at: - https://mapserver.org/optimization/limit_mapfile_access.html - https://mapserver.org/environment_variables.html Please don?t hesitate to reach out with questions. --Steve -------------- next part -------------- An HTML attachment was scrubbed... URL: From sdlime at gmail.com Tue Mar 30 11:36:23 2021 From: sdlime at gmail.com (Steve Lime) Date: Tue, 30 Mar 2021 13:36:23 -0500 Subject: [mapserver-users] Layer Attribute Metadata In-Reply-To: References: Message-ID: Hi Seth: I don't think adding it would be difficult. What would the emitted attribute look like w/that property added? --Steve On Sat, Mar 27, 2021 at 4:11 PM Seth G wrote: > Hi all, > > Is anyone aware of any OGC service which can return comments for a layer > attribute/field? > The WFS DescribeFeatureType [1] seems the most likely candidate but has no > "description" or equivalent property at the attribute level. Attributes are > returned as follows: > > > > The docs for the WFS layer object > https://www.mapserver.org/ogc/wfs_server.html#layer-object allow the > following to be set for each attribute: > > gml_[item name]_alias > gml_[item name]_precision > gml_[item name]_type > gml_[item name]_value > gml_[item name]_width > > It would be nice if there was an option to set something like the > following, and have it returned as part of the layer metadata: > > gml_[item name]_description > > This way the automated parts of the attribute (type, precision etc.) could > be included along with a description of what the field contains. > > Does anyone have a way of handling this with MapServer? > > Seth > > [1] > http://opengeospatial.github.io/e-learning/wfs/text/operations.html#describefeaturetype > > > -- > web:http://geographika.co.uk > twitter: @geographika > _______________________________________________ > mapserver-users mailing list > mapserver-users at lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/mapserver-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sethg at geographika.co.uk Tue Mar 30 13:47:56 2021 From: sethg at geographika.co.uk (Seth G) Date: Tue, 30 Mar 2021 22:47:56 +0200 Subject: [mapserver-users] Layer Attribute Metadata In-Reply-To: References: Message-ID: <421ec23a-90ac-4f1c-9e9b-6a7a8b251184@www.fastmail.com> Hi Steve, The output would look something like the output in https://inspire.ec.europa.eu/schemas/au/4.0/AdministrativeUnits.xsd -- Name -- upper level unit -- Definition -- A unit established at a higher level of national administrative hierarchy that this administrative unit administers. -- Description -- NOTE Administrative units at the highest level of national hierarchy (i.e. the country) do not have upper level units. CONSTRAINT Each administrative unit at the level other than '1st order' (i.e. nationalLevel <> '1st order') shall refer their upper level unit. au:AdministrativeUnit au:lowerLevelUnit Maybe this is part of the ISO standard used by GetLayerMetadata - https://mapserver.org/ogc/layer_metadata.html ? WFS GetFeatureType doesn't seem to have a description field - http://schemas.opengis.net/wfs/2.0/examples/DescribeFeatureType/DescribeFeatureType_Example01_Response.xsd It seems a missed opportunity not to be able to document fields alongside autogenerated type/nillable information from the source. Seth -- web:http://geographika.co.uk twitter: @geographika On Tue, Mar 30, 2021, at 8:36 PM, Steve Lime wrote: > Hi Seth: I don't think adding it would be difficult. What would the emitted attribute look like w/that property added? --Steve > > On Sat, Mar 27, 2021 at 4:11 PM Seth G wrote: >> Hi all, >> >> Is anyone aware of any OGC service which can return comments for a layer attribute/field? >> The WFS DescribeFeatureType [1] seems the most likely candidate but has no "description" or equivalent property at the attribute level. Attributes are returned as follows: >> >> >> >> The docs for the WFS layer object https://www.mapserver.org/ogc/wfs_server.html#layer-object allow the following to be set for each attribute: >> >> gml_[item name]_alias >> gml_[item name]_precision >> gml_[item name]_type >> gml_[item name]_value >> gml_[item name]_width >> >> It would be nice if there was an option to set something like the following, and have it returned as part of the layer metadata: >> >> gml_[item name]_description >> >> This way the automated parts of the attribute (type, precision etc.) could be included along with a description of what the field contains. >> >> Does anyone have a way of handling this with MapServer? >> >> Seth >> >> [1] http://opengeospatial.github.io/e-learning/wfs/text/operations.html#describefeaturetype >> >> >> -- >> web:http://geographika.co.uk >> twitter: @geographika >> _______________________________________________ >> mapserver-users mailing list >> mapserver-users at lists.osgeo.org >> https://lists.osgeo.org/mailman/listinfo/mapserver-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From jmckenna at gatewaygeomatics.com Wed Mar 31 05:30:01 2021 From: jmckenna at gatewaygeomatics.com (Jeff McKenna) Date: Wed, 31 Mar 2021 09:30:01 -0300 Subject: [mapserver-users] =?utf-8?q?Security_Advisory_=E2=80=93_Limiting?= =?utf-8?q?_Mapfile_Access?= In-Reply-To: References: Message-ID: <82fe51a8-6222-9ebe-6984-ea45ddd1be32@gatewaygeomatics.com> All: please share the advisory in your networks: https://mapserver.org/announcements/2021-03-30-limit-mapfile-access.html -the MapServer PSC On 2021-03-30 3:24 p.m., Steve Lime wrote: > Hi all: This is an important reminder that, as part of a secure > deployment, it is important to limit MapServer CGI access to mapfiles. > The MapServer CGI has long supported the use of environment variables as > a primary mechanism to do this. If you haven?t implemented these > controls then that constitutes undue risk that is easily mitigated and > we strongly encourage you to do so as soon as?possible. It?s also a > great time to review those settings if you already have them in place as > we?ve recently updated regex examples related to MS_MAP_PATTERN to limit > path traversal. > > Relevant documentation can be found at: > > * https://mapserver.org/optimization/limit_mapfile_access.html > > * https://mapserver.org/environment_variables.html > > > Please don?t hesitate to reach out with questions. > > --Steve >