From sdlime at gmail.com Fri Nov 1 12:45:10 2019 From: sdlime at gmail.com (Steve Lime) Date: Fri, 1 Nov 2019 14:45:10 -0500 Subject: [mapserver-users] two values in runtime-substitution Filter vs. processing native_filter In-Reply-To: References: Message-ID: So you're seeing a different substitution result when using %key% in: FILTER '...%key%...' versus: PROCESSING 'NATIVE_FILTER=...%key%' If so that would be a bug. The value that replaces %key% should be consistent. The value you supply could be different though because how the expression is being further processed (e.g. MapServer vs. layer-specific driver) is different. I'll have to whip up a quick test case to verify. Otherwise I think in general you'd want to avoid lists if possible and send things individually so key1=value1&key2=value2... and then use something like: PROCESSING 'NATIVE_FILTER=...column in ('%key1%','%key2'%) I would think validation would be easier in this case but I understand it might not be practical with an indeterminate number of options. In that case its probably best to send a completely encoded set of values (complete with quotes) depending on the diver, so... key=%27value1%27%2C%27value2%27%2C%27value3%27 and then use: PROCESSING 'NATIVE_FILTER=...column in ('%key%) I can't think of an alternative short of 1) adding the ability to split and format strings before substitution (set a delimiter and a printf pattern) or 2) adding a function like Perl's qw (quote word) that would handle string delimiters for you qw(value1,value2). The latter would only work for FILTERs, --Steve On Thu, Oct 31, 2019 at 3:31 AM J?rg Thomsen (WhereGroup) < joerg.thomsen at wheregroup.com> wrote: > Good morning everyone, > > we built an application which uses a runtime substitution for the filter > string. And we found a different behavior between (the old) Filter and > the native_processing filter if two values are parsed. > > the request http://....?..&key=vaule1,value2 > results to > 'value1','value2' with the old FILTER () and to > 'value1,value2'when using PROCESSING 'NATIVE_FILTER > > We have to use the PROCESSING 'NATIVE_FILTER, but need as result > 'value1','value2', because we want to use it in a sql-query like: > value in ('value1','value2'). > > First question: > has anyone an idea how to handle this? We could request > http://....?..&key=value1','value2 > but that's no nice solution. > > Second question: is it the wanted behavior or perhaps a bug? > > J?rg > > -- > > *************************** > Where2B Konferenz 2019 > 12. Dezember 2019 in Bonn > where2b-conference.com > *************************** > > J?rg Thomsen > WhereGroup GmbH > Gillweg 3 > 14193 Berlin > Germany > > Fon: +49 (0)30 / 5130 278 74 > Fax: +49 (0)30 / 5130 278 11 > > joerg.thomsen at wheregroup.com > www.wheregroup.com > > Gesch?ftsf?hrer: > Olaf Knopp, Peter Stamm > Amtsgericht Bonn, HRB 9885 > > ------------------------------- > Folgen Sie der WhereGroup auf twitter: http://twitter.com/WhereGroup_com > _______________________________________________ > mapserver-users mailing list > mapserver-users at lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/mapserver-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From djrobnett at gmail.com Tue Nov 5 13:49:16 2019 From: djrobnett at gmail.com (djrob) Date: Tue, 5 Nov 2019 14:49:16 -0700 (MST) Subject: [mapserver-users] PHP_Mapscript swig issue on macOS. In-Reply-To: <0325bab2-c004-4df8-a3a5-52a58ed94d8d@www.fastmail.com> References: <311BA2FF-6C7B-4F2A-9155-31557FC5BB5F@offroadsoftware.com> <0325bab2-c004-4df8-a3a5-52a58ed94d8d@www.fastmail.com> Message-ID: <1572990556436-0.post@n6.nabble.com> I am having the same issue on a RHEL7 server as the OP. Using SWIG with PHP mapscript when I attempt to add a label with any font specified it fails to create the image and throws a fatal error. If I leave the font parameter out of the label section of the map file the image is generated but without any text. I did a little digging and found that the actual error being thrown is a FreeType error 0x021 or "invalid library handle" when the FT_New_Face function is called from within msGetFontFace in fontcache.c. It appears that the font cache is not being properly initialized but unfortunately that is as far as I was able to troubleshoot. I have tried with multiple locations of the fontset file and fonts themselves. I have tried both full and relative paths to the fonts within the fonts.list file. Nothing seems to be able to resolve the issue. Hoping someone can point me in the right direction. Here are the pertinent details on my install. I built the mapserv files from source, everything else was installed via yum from RedHat repositories. MapServer version 7.4.2 OUTPUT=PNG OUTPUT=JPEG SUPPORTS=PROJ SUPPORTS=AGG SUPPORTS=FREETYPE SUPPORTS=CAIRO SUPPORTS=ICONV SUPPORTS=FRIBIDI SUPPORTS=WMS_SERVER SUPPORTS=WMS_CLIENT SUPPORTS=WFS_SERVER SUPPORTS=WFS_CLIENT SUPPORTS=WCS_SERVER SUPPORTS=SOS_SERVER SUPPORTS=FASTCGI SUPPORTS=GEOS INPUT=JPEG INPUT=POSTGIS INPUT=OGR INPUT=GDAL INPUT=SHAPEFILE PHP PHP 7.2.24 (cli) (built: Oct 30 2019 04:12:25) ( NTS ) Copyright (c) 1997-2018 The PHP Group Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies SWIG SWIG Version 3.0.12 Compiled with g++ [x86_64-redhat-linux-gnu] Configured options: +pcre -- Sent from: http://osgeo-org.1560.x6.nabble.com/Mapserver-User-f4226646.html From jmckenna at gatewaygeomatics.com Tue Nov 5 14:41:17 2019 From: jmckenna at gatewaygeomatics.com (Jeff McKenna) Date: Tue, 5 Nov 2019 18:41:17 -0400 Subject: [mapserver-users] PHP_Mapscript swig issue on macOS. In-Reply-To: <1572990556436-0.post@n6.nabble.com> References: <311BA2FF-6C7B-4F2A-9155-31557FC5BB5F@offroadsoftware.com> <0325bab2-c004-4df8-a3a5-52a58ed94d8d@www.fastmail.com> <1572990556436-0.post@n6.nabble.com> Message-ID: <17f23406-6d31-e6ab-d6bc-968f3e20f8e1@gatewaygeomatics.com> At least SWIG version 4.0 is recommended for PHP7 support (or git-master). I also compile all mapserv dependencies from source on unix (such as Freetype, libpng, Harfbuzz, etc) as well as for GDAL, to avoid issues such as yours involving packages that you likely did not compile. -jeff -- Jeff McKenna MapServer Consulting and Training Services https://gatewaygeomatics.com/ On 2019-11-05 5:49 PM, djrob wrote: > I am having the same issue on a RHEL7 server as the OP. Using SWIG with PHP > mapscript when I attempt to add a label with any font specified it fails to > create the image and throws a fatal error. If I leave the font parameter > out of the label section of the map file the image is generated but without > any text. > > I did a little digging and found that the actual error being thrown is a > FreeType error 0x021 or "invalid library handle" when the FT_New_Face > function is called from within msGetFontFace in fontcache.c. It appears > that the font cache is not being properly initialized but unfortunately that > is as far as I was able to troubleshoot. > > I have tried with multiple locations of the fontset file and fonts > themselves. I have tried both full and relative paths to the fonts within > the fonts.list file. Nothing seems to be able to resolve the issue. Hoping > someone can point me in the right direction. > > Here are the pertinent details on my install. I built the mapserv files > from source, everything else was installed via yum from RedHat repositories. > > MapServer version 7.4.2 OUTPUT=PNG OUTPUT=JPEG SUPPORTS=PROJ SUPPORTS=AGG > SUPPORTS=FREETYPE SUPPORTS=CAIRO SUPPORTS=ICONV SUPPORTS=FRIBIDI > SUPPORTS=WMS_SERVER SUPPORTS=WMS_CLIENT SUPPORTS=WFS_SERVER > SUPPORTS=WFS_CLIENT SUPPORTS=WCS_SERVER SUPPORTS=SOS_SERVER SUPPORTS=FASTCGI > SUPPORTS=GEOS INPUT=JPEG INPUT=POSTGIS INPUT=OGR INPUT=GDAL INPUT=SHAPEFILE > > PHP > PHP 7.2.24 (cli) (built: Oct 30 2019 04:12:25) ( NTS ) > Copyright (c) 1997-2018 The PHP Group > Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies > > SWIG > SWIG Version 3.0.12 > Compiled with g++ [x86_64-redhat-linux-gnu] > Configured options: +pcre > > From stefan27 at hotmail.com Thu Nov 7 01:38:39 2019 From: stefan27 at hotmail.com (StefanWebViewer) Date: Thu, 7 Nov 2019 02:38:39 -0700 (MST) Subject: [mapserver-users] integrate variable openlayer map extent Message-ID: <1573119519942-0.post@n6.nabble.com> Good day everybody, thanks for the last hints. That was very helpful. Now I have another question: We have created map accesses for several regions (more than 300). Each of these regions has a different map extent. So that the respective map extent does not have to be adapted manually, I created a .js with XMIN-YMAX for each region. This looks like this: var XMIN = 123 var YMIN = 456 etc. In the OpenLayers.js the map extent section looks like this: map = new OpenLayers.Map ('map', { controls:[ new OpenLayers.Control.Navigation(), new OpenLayers.Control.PanZoomBar(), new OpenLayers.Control.LayerSwitcher(), new OpenLayers.Control.Attribution(), new OpenLayers.Control.MousePosition(), new OpenLayers.Control.OverviewMap({'size': refmapextent})], maxExtent: new OpenLayers.Bounds(3279012,5612977,3311313,5642977), projection: new OpenLayers.Projection('EPSG:31467'), maxResolution: 'auto'} ); Now my goal is to make the line like this: "maxExtent: new OpenLayers.Bounds ('XMIN', 'YMIN,' XMAX ',' YMAX ')," At the beginning of the .js I call the var on each. Unfortunately, when I do this, no map will appear. What possibilities exist to successfully integrate such variables? About hints I am happy as always. Thank you so much! -- Sent from: http://osgeo-org.1560.x6.nabble.com/Mapserver-User-f4226646.html From stefan27 at hotmail.com Thu Nov 7 06:29:04 2019 From: stefan27 at hotmail.com (StefanWebViewer) Date: Thu, 7 Nov 2019 07:29:04 -0700 (MST) Subject: [mapserver-users] integrate variable openlayer map extent In-Reply-To: <1573119519942-0.post@n6.nabble.com> References: <1573119519942-0.post@n6.nabble.com> Message-ID: <1573136944362-0.post@n6.nabble.com> I solved it myself: - the variables without ' - call the .js earlier in the php file -- Sent from: http://osgeo-org.1560.x6.nabble.com/Mapserver-User-f4226646.html From jmckenna at gatewaygeomatics.com Thu Nov 7 07:33:07 2019 From: jmckenna at gatewaygeomatics.com (Jeff McKenna) Date: Thu, 7 Nov 2019 11:33:07 -0400 Subject: [mapserver-users] integrate variable openlayer map extent In-Reply-To: <1573136944362-0.post@n6.nabble.com> References: <1573119519942-0.post@n6.nabble.com> <1573136944362-0.post@n6.nabble.com> Message-ID: On 2019-11-07 10:29 AM, StefanWebViewer wrote: > I solved it myself: > - the variables without ' > - call the .js earlier in the php file > For good karma you could add some of these great tips to the MapServer wiki, in the section "Tips, Tricks, Howtos" at https://github.com/mapserver/mapserver/wiki Thanks :) -jeff From yves.jacolin at camptocamp.com Fri Nov 8 05:42:03 2019 From: yves.jacolin at camptocamp.com (Yves Jacolin) Date: Fri, 8 Nov 2019 14:42:03 +0100 Subject: [mapserver-users] mapcache seed speed optimization In-Reply-To: References: Message-ID: Sebastiano, Some though in my mind: * ?n 20000 : do you really have 2'000 CPU on your WMS server? if not, it could be worst than better, flooding your WMS server. * what is the bottleneck? If WMS server is slow you can't make MapCache faster, you should optimise WMS Server. Multi-domain can help, data improvement, etc. also. * Does your WMS is on the same server? same network? Does you network fast enough? * metatile can help definitively! * "we use sqlite as a cache method" => I don't know if sqlite is really fast or not, but this could be another bottleneck :) What do you mean exactly by improve seed speed? :) This is matter of personal point of view some time. Y. Le ven. 8 nov. 2019 ? 13:43, Sebastiano Laini < Sebastiano.Laini at buchanancomputing.co.uk> a ?crit : > Hi all, > > > > I?m evaluating new stacks but seems that I?m stuck with mapserver and > mapcache due to the input source of our maps, so I?m trying to improve the > speed of the mapserver/mapcache stack and what I cannot improve is the seed > speed. > > > > The ?n and ?p parameters seems to be useless. > > *-n | ?nthreads*: number of parallel threads that should be used to > request tiles from the WMS source. The default is 1, but can be set higher > if the WMS server can withstand parallel requests. (As a rule of thumb, the > value chosen here should never be much higher than the number of CPUs on > the WMS server.) > > *-p | ?nprocesses*: number of parallel processes that should be used to > request tiles from the WMS source. > > I?ve tried to use ?n 20000 and ?p 60000 but for some reason the CPU load > is max 100% in 1 core, no way to make it use more power and speed up the > seed process. > > Max memory used is 470MB between Centos 7 (with all the services running) > and the seed process. > > > > I?ve setup apache to use HTT/2 and it make quite a difference when the > cache is already seeded but during the seed process doesn?t seems to be > affected. > > > > My MapServer build is: > > > > ./configure \ > > --with-ogr=/usr/local/bin/gdal-config \ > > --with-gdal=/usr/local/bin/gdal-config \ > > --with-wfsclient \ > > --with-wmsclient \ > > --with-wfs \ > > --enable-debug \ > > --with-curl-config=/usr/bin/curl-config \ > > --with-proj \ > > --with-jpeg \ > > --with-freetype \ > > --with-postgis=/usr/pgsql-9.3/bin/pg_config \ > > --with-geos=/usr/local/bin/geos-config > > > > I know there is mapserver 7 and yet I?m using 6 for the test. > > > > While the MapCache is: > > > > cmake -DCMAKE_PREFIX_PATH="/usr/bin/sqlite3" \ > > -DWITH_SQLITE=1 \ > > -DWITH_BERKELEY_DB=0 \ > > -DWITH_TIFF=0 \ > > -DWITH_GEOTIFF=0 \ > > -DWITH_FCGI=0 \ > > -DWITH_PCRE=0 \ > > -DWITH_PIXMAN=1 \ > > -DWITH_OGR=1 \ > > -DWITH_GEOS=1 \ > > ../ > > > > MapCache is the latest version and this is my mapcache.xml file, we use > sqlite as a cache method > > > > > > > /home/www/html/maps/cache/{tileset}/{grid}/{z}/{x}-{y}.sqlite3 > > 10000 > > 10000 > > 1573741823 > > > > > > > > bcmaps > > cache_sqlite > > bcgrid > > PNG > > 16 16 > > 0 > > 2628000 > > > > > > > > > > BC custom grid > > > > EPSG:27700 > > 256 256 > > 0 0 700000 1250000 > > 70 28 14 7 2.8 1.4 0.7 > > > > > > > > best > > > > > > > > assemble > > bilinear > > mypng > > 4096 > > > > > > I?ve also tried to use ?DWITH_MAPSERVER but seems that the integration is > not working and they doen?t care? > https://github.com/mapserver/mapcache/issues/169 this issue was open 2 > years ago and still open, not even a reply from the developers. > > > > How can I improve the seed process? > > Should I increase the maxsize and metatile to requests more tiles and > cache faster? > > > > Regards, > > > > Sebastiano Laini > > Web Developer > > Buchanan Computing > > > _______________________________________________ > mapserver-users mailing list > mapserver-users at lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/mapserver-users -- Yves Jacolin Training and support manager - Team Manager Camptocamp Tel (France) : +33 4 58 48 20 43 Tel (Switzerland) : +41 21 619 10 43 Mob. : +33 6 18 75 42 21 email : yves.jacolin at camptocamp.com http://www.camptocamp.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From jukka.rahkonen at maanmittauslaitos.fi Fri Nov 8 06:01:55 2019 From: jukka.rahkonen at maanmittauslaitos.fi (Rahkonen Jukka (MML)) Date: Fri, 8 Nov 2019 14:01:55 +0000 Subject: [mapserver-users] mapcache seed speed optimization Message-ID: <46b48dd679e44694879379dd76211e8c@C119S212VM042.msvyvi.vaha.local> Hi, When you seed tiles from WMS the slowest component is usually the WMS server. And what the WMS server is spending its time is usually rendering. And what makes rendering slow is usually that you render too much, and sometimes that WMS server gets the vector data too slow because the backend data source is slow, very often because it has poor indexes which do not for suit the queries that WMS server is sending. If you have a slow WMS you can't get more speed out of it by bombing it with very many parallel threads. I would have a try if using 2-4, perhaps 8 threads makes seeding faster but if there is no difference I would concentrate on the WMS configuration. BTW. why did you select to use 20000 threads? I am not sure why you see just 1 core running on 100% load. I guess that you are now running WMS and MapCache on the same computer. Mapserver program is using only 1 core but http server should start more processes for parallel requests. That CPU load is at 100% is generally a good thing because it tells that CPU is working, instead of waiting data from a slow disk. Another thing is that CPU may be doing something heavy and stupid. Myself I would start by gathering some information about how Mapserver WMS is working be setting "DEBUG 2" on each layer or for the whole mapfile. By looking at the rendering times of each layer the slowest layers can be found and then it is time to think about how to make those layers faster. You wrote that you use SQLite cache so it would be good to know that SQLite database does not support parallel writes https://sqlite.org/lockingv3.html. Seeding with a few parallel threads may increase the speed if the WMS is the slowest part in your chain but when you have saturated the SQLite writing speed adding threads will do no good. If you have optimized your WMS so well that SQLite sets the limits then you should test other cache alternatives. I guess that file cache would be faster to seed from multiple threads. When is comes to ticket https://github.com/mapserver/mapcache/issues/169 the latest comment is from a Mpaserver developer and their last words "I wouldn't mind helping to fix if there were interested parties..." feel rather polite to me. -Jukka Rahkonen- L?hett?j?: mapserver-users Puolesta Sebastiano Laini L?hetetty: perjantai 8. marraskuuta 2019 14.35 Vastaanottaja: 'mapserver-users at lists.osgeo.org' Aihe: [mapserver-users] mapcache seed speed optimization Hi all, I'm evaluating new stacks but seems that I'm stuck with mapserver and mapcache due to the input source of our maps, so I'm trying to improve the speed of the mapserver/mapcache stack and what I cannot improve is the seed speed. The -n and -p parameters seems to be useless. -n | -nthreads: number of parallel threads that should be used to request tiles from the WMS source. The default is 1, but can be set higher if the WMS server can withstand parallel requests. (As a rule of thumb, the value chosen here should never be much higher than the number of CPUs on the WMS server.) -p | -nprocesses: number of parallel processes that should be used to request tiles from the WMS source. I've tried to use -n 20000 and -p 60000 but for some reason the CPU load is max 100% in 1 core, no way to make it use more power and speed up the seed process. Max memory used is 470MB between Centos 7 (with all the services running) and the seed process. I've setup apache to use HTT/2 and it make quite a difference when the cache is already seeded but during the seed process doesn't seems to be affected. My MapServer build is: ./configure \ --with-ogr=/usr/local/bin/gdal-config \ --with-gdal=/usr/local/bin/gdal-config \ --with-wfsclient \ --with-wmsclient \ --with-wfs \ --enable-debug \ --with-curl-config=/usr/bin/curl-config \ --with-proj \ --with-jpeg \ --with-freetype \ --with-postgis=/usr/pgsql-9.3/bin/pg_config \ --with-geos=/usr/local/bin/geos-config I know there is mapserver 7 and yet I'm using 6 for the test. While the MapCache is: cmake -DCMAKE_PREFIX_PATH="/usr/bin/sqlite3" \ -DWITH_SQLITE=1 \ -DWITH_BERKELEY_DB=0 \ -DWITH_TIFF=0 \ -DWITH_GEOTIFF=0 \ -DWITH_FCGI=0 \ -DWITH_PCRE=0 \ -DWITH_PIXMAN=1 \ -DWITH_OGR=1 \ -DWITH_GEOS=1 \ ../ MapCache is the latest version and this is my mapcache.xml file, we use sqlite as a cache method /home/www/html/maps/cache/{tileset}/{grid}/{z}/{x}-{y}.sqlite3 10000 10000 1573741823 bcmaps cache_sqlite bcgrid PNG 16 16 0 2628000 BC custom grid EPSG:27700 256 256 0 0 700000 1250000 70 28 14 7 2.8 1.4 0.7 best assemble bilinear mypng 4096 I've also tried to use -DWITH_MAPSERVER but seems that the integration is not working and they doen't care? https://github.com/mapserver/mapcache/issues/169 this issue was open 2 years ago and still open, not even a reply from the developers. How can I improve the seed process? Should I increase the maxsize and metatile to requests more tiles and cache faster? Regards, Sebastiano Laini Web Developer Buchanan Computing -------------- next part -------------- An HTML attachment was scrubbed... URL: From jukka.rahkonen at maanmittauslaitos.fi Fri Nov 8 07:02:40 2019 From: jukka.rahkonen at maanmittauslaitos.fi (Rahkonen Jukka (MML)) Date: Fri, 8 Nov 2019 15:02:40 +0000 Subject: [mapserver-users] mapcache seed speed optimization In-Reply-To: References: <46b48dd679e44694879379dd76211e8c@C119S212VM042.msvyvi.vaha.local> Message-ID: Hi, You can see examples about how to debug and improve the rendering speed with Mapserver in the presentation of Lars Schylberg https://media.ccc.de/v/bucharest-271-mapserver-optimize-for-performance. Your maps look pretty and I hope you will find some bottleneck layers so you can make them faster. -Jukka- L?hett?j?: Sebastiano Laini L?hetetty: perjantai 8. marraskuuta 2019 16.30 Vastaanottaja: 'mapserver-users at lists.osgeo.org' Kopio: Rahkonen Jukka (MML) ; 'Yves Jacolin' Aihe: RE: mapcache seed speed optimization My MapServer is on the same server as MapCache and my network is pretty fast. I was just trying to experiment with the -n and -p value and checking the CPU load and I didn't seen any difference, but now that I read the responses I'm trying -n 6 and the CPU and the RAM is more in use now and seems to seed faster than before. I reach that numbers because the last time I was seeding with -n 10 was incredibly slow so I started to increase the numbers. Now in HTOP I can actually see 6 mapcache processes and not just 2 Sadly to seed the whole UK in disk cache we run out of inodes, I would like to use SSD. About our data we use a mix of everything, here the final result https://www.buchananmapping.co.uk/ We use raster, vector and tab files; the reason I can't move out from mapserver are mainly the mapinfo tables (tab), possibly those tab files are the slowest layers Even though some of the vector layers are very huge, some of them the data is about 21GB. I'll try to debug layer by layer through MapServer and strip down to the minimum or replace some layers. Lastly, the last time I've tried to seed the whole UK it took me more than a month and I simply stop it because was unacceptable to seed 7 levels of resolutions 70, 28, 14, 7, 2.8, 1.4, 0.7 that correspond to these scales 2500, 5000, 10000, 25000, 50000, 100000, 250000, 500000 Sebastiano Laini Web Developer From: Rahkonen Jukka (MML) [mailto:jukka.rahkonen at maanmittauslaitos.fi] Sent: 08 November 2019 14:02 To: Sebastiano Laini >; 'mapserver-users at lists.osgeo.org' > Subject: Re: mapcache seed speed optimization Hi, When you seed tiles from WMS the slowest component is usually the WMS server. And what the WMS server is spending its time is usually rendering. And what makes rendering slow is usually that you render too much, and sometimes that WMS server gets the vector data too slow because the backend data source is slow, very often because it has poor indexes which do not for suit the queries that WMS server is sending. If you have a slow WMS you can't get more speed out of it by bombing it with very many parallel threads. I would have a try if using 2-4, perhaps 8 threads makes seeding faster but if there is no difference I would concentrate on the WMS configuration. BTW. why did you select to use 20000 threads? I am not sure why you see just 1 core running on 100% load. I guess that you are now running WMS and MapCache on the same computer. Mapserver program is using only 1 core but http server should start more processes for parallel requests. That CPU load is at 100% is generally a good thing because it tells that CPU is working, instead of waiting data from a slow disk. Another thing is that CPU may be doing something heavy and stupid. Myself I would start by gathering some information about how Mapserver WMS is working be setting "DEBUG 2" on each layer or for the whole mapfile. By looking at the rendering times of each layer the slowest layers can be found and then it is time to think about how to make those layers faster. You wrote that you use SQLite cache so it would be good to know that SQLite database does not support parallel writes https://sqlite.org/lockingv3.html. Seeding with a few parallel threads may increase the speed if the WMS is the slowest part in your chain but when you have saturated the SQLite writing speed adding threads will do no good. If you have optimized your WMS so well that SQLite sets the limits then you should test other cache alternatives. I guess that file cache would be faster to seed from multiple threads. When is comes to ticket https://github.com/mapserver/mapcache/issues/169 the latest comment is from a Mpaserver developer and their last words "I wouldn't mind helping to fix if there were interested parties..." feel rather polite to me. -Jukka Rahkonen- L?hett?j?: mapserver-users > Puolesta Sebastiano Laini L?hetetty: perjantai 8. marraskuuta 2019 14.35 Vastaanottaja: 'mapserver-users at lists.osgeo.org' > Aihe: [mapserver-users] mapcache seed speed optimization Hi all, I'm evaluating new stacks but seems that I'm stuck with mapserver and mapcache due to the input source of our maps, so I'm trying to improve the speed of the mapserver/mapcache stack and what I cannot improve is the seed speed. The -n and -p parameters seems to be useless. -n | -nthreads: number of parallel threads that should be used to request tiles from the WMS source. The default is 1, but can be set higher if the WMS server can withstand parallel requests. (As a rule of thumb, the value chosen here should never be much higher than the number of CPUs on the WMS server.) -p | -nprocesses: number of parallel processes that should be used to request tiles from the WMS source. I've tried to use -n 20000 and -p 60000 but for some reason the CPU load is max 100% in 1 core, no way to make it use more power and speed up the seed process. Max memory used is 470MB between Centos 7 (with all the services running) and the seed process. I've setup apache to use HTT/2 and it make quite a difference when the cache is already seeded but during the seed process doesn't seems to be affected. My MapServer build is: ./configure \ --with-ogr=/usr/local/bin/gdal-config \ --with-gdal=/usr/local/bin/gdal-config \ --with-wfsclient \ --with-wmsclient \ --with-wfs \ --enable-debug \ --with-curl-config=/usr/bin/curl-config \ --with-proj \ --with-jpeg \ --with-freetype \ --with-postgis=/usr/pgsql-9.3/bin/pg_config \ --with-geos=/usr/local/bin/geos-config I know there is mapserver 7 and yet I'm using 6 for the test. While the MapCache is: cmake -DCMAKE_PREFIX_PATH="/usr/bin/sqlite3" \ -DWITH_SQLITE=1 \ -DWITH_BERKELEY_DB=0 \ -DWITH_TIFF=0 \ -DWITH_GEOTIFF=0 \ -DWITH_FCGI=0 \ -DWITH_PCRE=0 \ -DWITH_PIXMAN=1 \ -DWITH_OGR=1 \ -DWITH_GEOS=1 \ ../ MapCache is the latest version and this is my mapcache.xml file, we use sqlite as a cache method /home/www/html/maps/cache/{tileset}/{grid}/{z}/{x}-{y}.sqlite3 10000 10000 1573741823 bcmaps cache_sqlite bcgrid PNG 16 16 0 2628000 BC custom grid EPSG:27700 256 256 0 0 700000 1250000 70 28 14 7 2.8 1.4 0.7 best assemble bilinear mypng 4096 I've also tried to use -DWITH_MAPSERVER but seems that the integration is not working and they doen't care? https://github.com/mapserver/mapcache/issues/169 this issue was open 2 years ago and still open, not even a reply from the developers. How can I improve the seed process? Should I increase the maxsize and metatile to requests more tiles and cache faster? Regards, Sebastiano Laini Web Developer Buchanan Computing -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan27 at hotmail.com Fri Nov 8 08:55:49 2019 From: stefan27 at hotmail.com (StefanWebViewer) Date: Fri, 8 Nov 2019 09:55:49 -0700 (MST) Subject: [mapserver-users] OpenLayers Zoom Selected Feature Message-ID: <1573232149582-0.post@n6.nabble.com> In our OpenLayer.js we first call the maximum extent of the map: "maxExtent: new OpenLayers.Bounds (X1, Y1, X2, Y2)," on. After the layer definition follows "map.addLayers ([lyr1, layer3, layer4]);" and "map.zoomToMaxExtent ();". In between, the variable for "OpenLayers.Control.GetFeature" is defined. Our problem now is that after our feature query, the map always returns to "MaxExtent". Does anyone have an idea, how after the feature query the zoom level does not change? After selected feature, the zoom should not change. We only want the "MaxExtent" when loading the map for the first time. -- Sent from: http://osgeo-org.1560.x6.nabble.com/Mapserver-User-f4226646.html From richard.greenwood at gmail.com Fri Nov 8 09:32:39 2019 From: richard.greenwood at gmail.com (Richard Greenwood) Date: Fri, 8 Nov 2019 10:32:39 -0700 Subject: [mapserver-users] mapserv on IIS 10 Message-ID: I used the the excellent instructions at https://mapserver.org/installation/iis.html to setup MapServer on WIndows Server 2019, IIS 10, FastCGI. It works, but it's not seeing my environment variables. I don't usually run MapServer on Windows but in this case it's a requirement. In previous IIS installs I've just set the environment variables globally outside of IIS and everything was fine. I've done that on this server and also set them a couple places in IIS, but it's not working with this version of Windows and/or IIS. (I'm not a Windows guy so I'm pretty much setting the environment variables everywhere I can). I posted my question to stackoverflow [1]. Anybody have any suggestions that I should try? [1] https://stackoverflow.com/questions/58697933/setting-and-reading-environment-variables-in-iis-10 -- Richard W. Greenwood, PLS www.greenwoodmap.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From traviskirstine at gmail.com Mon Nov 11 06:42:07 2019 From: traviskirstine at gmail.com (Travis Kirstine) Date: Mon, 11 Nov 2019 09:42:07 -0500 Subject: [mapserver-users] mapcache seed speed optimization In-Reply-To: <4b13326044f0493d8f23e8c1ed67f2d0@BCExchange.BuchananComputing.co.uk> References: <46b48dd679e44694879379dd76211e8c@C119S212VM042.msvyvi.vaha.local> <3416b08f42ba4954b77f10b1f21c7b30@BCExchange.BuchananComputing.co.uk> <4b13326044f0493d8f23e8c1ed67f2d0@BCExchange.BuchananComputing.co.uk> Message-ID: I may not be unrealistic for it take mapserver to 4-6 seconds to generate 4096px image depending on the amount data being rendered and the complexity of your styles / expressions / labels. If your mapserver is able to generate a 1024x1024 in under a second that is pretty good. Like Jukka and other suggest I would look at translating the files to shp files with spatial indexes or postgis tables with spatial and attribute indexes if necessary. Preprocessing the data (simplification / prefiltering / point thinning cluster) and using SCALETOKENs to reference different source layers at different scales can lead to big performance boosts at the mapserver end. For us the biggest limiting factor has been the write blocking to sqlite cache when seeding tiles, if you increase the number of threads / process in mapcache seed those process will just end up waiting to write to the cache. You can generally figure out what the max number of processes the cache can handle by running a seed on a test area and look at the number of tiles seeded per second, as some point -n will have no effect which your process are just waiting to write the cache. I would look at using this method to test sqlite vs disk or other backends. The GeoTiff cache may be promising as it appears to be non-blocking but experimental... We use a riak cache with a riak leveldb backend which works well for us but is a bit of a pain to manage as it difficult to delete objects so it's not great for caches that need to be refreshed. Finally you may want to test MapProxy as an alternative to MapCache (we use both). MapProxy supports a non-blocking compact cache that could solve your inode problem Regards On Fri, 8 Nov 2019 at 11:48, Sebastiano Laini < Sebastiano.Laini at buchanancomputing.co.uk> wrote: > We don?t create the data, we rely on OS (ordnance survey) to supply us the > maps files and then we publish them in our service but I assume that > probably we will need to create some flow to improve it or download it in > other format that is faster. > > > > Sebastiano Laini > > Web Developer > > Buchanan Computing > > > > *From:* Fawcett, David (MNIT) [mailto:david.fawcett at state.mn.us] > *Sent:* 08 November 2019 16:41 > *To:* Sebastiano Laini ; > 'Rahkonen Jukka (MML)' ; ' > mapserver-users at lists.osgeo.org' > *Subject:* RE: mapcache seed speed optimization > > > > For data formats that are slower to read, would it add fit your workflow > to convert it to different data format before creating the tiles? > _______________________________________________ > mapserver-users mailing list > mapserver-users at lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/mapserver-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From lars.schylberg at blixtmail.se Mon Nov 11 12:07:22 2019 From: lars.schylberg at blixtmail.se (lars.schylberg at blixtmail.se) Date: Mon, 11 Nov 2019 20:07:22 +0000 Subject: [mapserver-users] mapcache seed speed optimization In-Reply-To: References: <46b48dd679e44694879379dd76211e8c@C119S212VM042.msvyvi.vaha.local> <3416b08f42ba4954b77f10b1f21c7b30@BCExchange.BuchananComputing.co.uk> <4b13326044f0493d8f23e8c1ed67f2d0@BCExchange.BuchananComputing.co.uk> Message-ID: <5a37d772b31f0374b78f88a1f5055034@blixtmail.se> Hi I have had a similar situation as You with large national data sets in shape format. One of the biggest improvements for me has been to split large shape files into smaller and then use vector tile indexes in Mapserver. The tool shp2tile has been working well for me. Stephen Woodbridge has written that tool. It can be found at http://imaptools.com/download-software.html (http://imaptools.com/download-software.html) I think that You had to compile yourself on linux, but that was easy. I have an example where I have automated this in a script that runs through all shape file in a directory and splits shape files larger than a certain size and then creates the vector tile index files. The code snippet that is part of larger import script can be found here: https://gist.github.com/LarsSchy/644b37ab2b1e58b48808bfb2b57d0359 (https://gist.github.com/LarsSchy/644b37ab2b1e58b48808bfb2b57d0359) The script snippet also shows the normal optimizations we do. Hope this could give You some ideas how optimize and make Mapserver and Mapcache run faster. I also ran into the problem with to many inodes. That got solved by using sqlite as backend in Mapcache. Best regards / Lars Schylberg 11 november 2019 kl. 15:42, "Travis Kirstine" )> skrev: I may not be unrealistic for it take mapserver to 4-6 seconds to generate 4096px image depending on the amount data being rendered and the complexity of your styles / expressions / labels. If your mapserver is able to generate a 1024x1024 in under a second that is pretty good. Like Jukka and other suggest I would look at translating the files to shp files with spatial indexes or postgis tables with spatial and attribute indexes if necessary. Preprocessing the data (simplification / prefiltering / point thinning cluster) and using SCALETOKENs to reference different source layers at different scales can lead to big performance boosts at the mapserver end. For us the biggest limiting factor has been the write blocking to sqlite cache when seeding tiles, if you increase the number of threads / process in mapcache seed those process will just end up waiting to write to the cache. You can generally figure out what the max number of processes the cache can handle by running a seed on a test area and look at the number of tiles seeded per second, as some point -n will have no effect which your process are just waiting to write the cache. I would look at using this method to test sqlite vs disk or other backends. The GeoTiff cache may be promising as it appears to be non-blocking but experimental... We use a riak cache with a riak leveldb backend which works well for us but is a bit of a pain to manage as it difficult to delete objects so it's not great for caches that need to be refreshed. Finally you may want to test MapProxy as an alternative to MapCache (we use both). MapProxy supports a non-blocking compact cache that could solve your inode problem Regards On Fri, 8 Nov 2019 at 11:48, Sebastiano Laini wrote: We don?t create the data, we rely on OS (ordnance survey) to supply us the maps files and then we publish them in our service but I assume that probably we will need to create some flow to improve it or download it in other format that is faster. Sebastiano Laini Web Developer Buchanan Computing From: Fawcett, David (MNIT) [mailto:david.fawcett at state.mn.us (mailto:david.fawcett at state.mn.us)] Sent: 08 November 2019 16:41 To: Sebastiano Laini ; 'Rahkonen Jukka (MML)' ; 'mapserver-users at lists.osgeo.org (mailto:mapserver-users at lists.osgeo.org)' Subject: RE: mapcache seed speed optimization For data formats that are slower to read, would it add fit your workflow to convert it to different data format before creating the tiles? _______________________________________________ mapserver-users mailing list mapserver-users at lists.osgeo.org (mailto:mapserver-users at lists.osgeo.org) https://lists.osgeo.org/mailman/listinfo/mapserver-users (https://lists.osgeo.org/mailman/listinfo/mapserver-users) -------------- next part -------------- An HTML attachment was scrubbed... URL: From jmckenna at gatewaygeomatics.com Mon Nov 11 13:26:57 2019 From: jmckenna at gatewaygeomatics.com (Jeff McKenna) Date: Mon, 11 Nov 2019 17:26:57 -0400 Subject: [mapserver-users] mapcache seed speed optimization In-Reply-To: <5a37d772b31f0374b78f88a1f5055034@blixtmail.se> References: <46b48dd679e44694879379dd76211e8c@C119S212VM042.msvyvi.vaha.local> <3416b08f42ba4954b77f10b1f21c7b30@BCExchange.BuchananComputing.co.uk> <4b13326044f0493d8f23e8c1ed67f2d0@BCExchange.BuchananComputing.co.uk> <5a37d772b31f0374b78f88a1f5055034@blixtmail.se> Message-ID: For Windows users I have been distributing shp2tile with MS4W (https://ms4w.com/), and in fact the last MS4W contains a rebuilt shp2tile (and an updated Shapelib), which as Lars said is still very useful. -jeff On 2019-11-11 4:07 PM, lars.schylberg at blixtmail.se wrote: > Hi > > I have had a similar situation as You with large national data sets in > shape format. One of the biggest improvements for me has been to split > large shape files into smaller and > then use vector tile indexes in Mapserver. > The tool shp2tile has been working well for me. Stephen Woodbridge has > written that tool. It can be found at > http://imaptools.com/download-software.html > > I think that You had to compile yourself on linux, but that was easy. > > I have an example where I have automated this in a script that runs > through all shape file in a directory and splits shape files larger than > a certain size and then creates the vector tile index files. > The code snippet that is part of larger import script can be found here: > https://gist.github.com/LarsSchy/644b37ab2b1e58b48808bfb2b57d0359 > > The script snippet also shows the normal optimizations we do. > > Hope this could give You some ideas how optimize and make Mapserver and > Mapcache run faster. I also ran into the problem with to many inodes. > That got solved by using sqlite as backend in Mapcache. > > Best regards / Lars Schylberg > > > > 11 november 2019 kl. 15:42, "Travis Kirstine" >> > skrev: > > I may not be unrealistic for it take mapserver to 4-6 seconds to > generate 4096px image depending on the amount data being rendered > and the complexity of your styles / expressions / labels. If your > mapserver is able to generate a 1024x1024 in under a second that is > pretty good. Like Jukka and other suggest I would look at > translating the files to shp files with spatial indexes or postgis > tables with spatial and attribute indexes if necessary. > Preprocessing the data (simplification / prefiltering / point > thinning cluster) and using SCALETOKENs to reference different > source layers at different scales can lead to big performance boosts > at the mapserver end. > For us the biggest limiting factor has been the write blocking to > sqlite cache when seeding tiles, if you increase the number of > threads / process in mapcache seed those process will just end up > waiting to write to the cache. You can generally figure out what the > max number of processes the cache can handle by running a seed on a > test area and look at the number of tiles seeded per second, as some > point -n will have no effect which your process are just waiting to > write the cache. I would look at using this method to test sqlite vs > disk or other backends. The GeoTiff cache may be promising as it > appears to be non-blocking but experimental... > We use a riak cache with a riak leveldb backend which works well for > us but is a bit of a pain to manage as it difficult to delete > objects so it's not great for caches that need to be refreshed. > Finally you may want to test MapProxy as an alternative to MapCache > (we use both). MapProxy supports a non-blocking compact cache that > could solve your inode problem > Regards > On Fri, 8 Nov 2019 at 11:48, Sebastiano Laini > > wrote: > > We don?t create the data, we rely on OS (ordnance survey) to > supply us the maps files and then we publish them in our service > but I assume that probably we will need to create some flow to > improve it or download it in other format that is faster. > > Sebastiano Laini > > Web Developer > > Buchanan Computing > > *From:*Fawcett, David (MNIT) [mailto:david.fawcett at state.mn.us > ] > *Sent:* 08 November 2019 16:41 > *To:* Sebastiano Laini >; 'Rahkonen > Jukka (MML)' >; > 'mapserver-users at lists.osgeo.org > ' > > > *Subject:* RE: mapcache seed speed optimization > > For data formats that are slower to read, would it add fit your > workflow to convert it to different data format before creating > the tiles? > > From stephenwoodbridge37 at gmail.com Mon Nov 11 13:50:41 2019 From: stephenwoodbridge37 at gmail.com (Stephen Woodbridge) Date: Mon, 11 Nov 2019 16:50:41 -0500 Subject: [mapserver-users] mapcache seed speed optimization In-Reply-To: <5a37d772b31f0374b78f88a1f5055034@blixtmail.se> References: <46b48dd679e44694879379dd76211e8c@C119S212VM042.msvyvi.vaha.local> <3416b08f42ba4954b77f10b1f21c7b30@BCExchange.BuchananComputing.co.uk> <4b13326044f0493d8f23e8c1ed67f2d0@BCExchange.BuchananComputing.co.uk> <5a37d772b31f0374b78f88a1f5055034@blixtmail.se> Message-ID: FYI, imaptools.com is not online any more (except the we are moving pages). Lars if you email me the files, I'll push them to my github repository so others can find them. Thanks, ? -Steve W On 11/11/2019 3:07 PM, lars.schylberg at blixtmail.se wrote: > Hi > > I have had a similar situation as You with large national data sets in > shape format. One of the biggest improvements for me has been to split > large shape files into smaller and > then use vector tile indexes in Mapserver. > The tool shp2tile has been working well for me. Stephen Woodbridge has > written that tool. It can be found at > http://imaptools.com/download-software.html > I think that You had to compile yourself on linux, but that was easy. > > I have an example where I have automated this in a script that runs > through all shape file in a directory and splits shape files larger > than a certain size and then creates the vector tile index files. > The code snippet that is part of larger import script can be found here: > https://gist.github.com/LarsSchy/644b37ab2b1e58b48808bfb2b57d0359 > > The script snippet also shows the normal optimizations we do. > > Hope this could give You some ideas how optimize and make Mapserver > and Mapcache run faster. I also ran into the problem with to many inodes. > That got solved by using sqlite as backend in Mapcache. > > Best regards / Lars Schylberg > > > > 11 november 2019 kl. 15:42, "Travis Kirstine" > >> > skrev: > > I may not be unrealistic for it take mapserver to 4-6 seconds to > generate 4096px image depending on the amount data being rendered > and the complexity of your styles / expressions / labels. If your > mapserver is able to generate a 1024x1024 in under a second that > is pretty good. Like Jukka and other suggest I would look at > translating the files to shp files with spatial indexes or postgis > tables with spatial and attribute indexes if necessary. > Preprocessing the data (simplification / prefiltering / point > thinning cluster) and using SCALETOKENs to reference different > source layers at different scales can lead to big performance > boosts at the mapserver end. > For us the biggest limiting factor has been the write blocking to > sqlite cache when seeding tiles, if you increase the number of > threads / process in mapcache seed those process will just end up > waiting to write to the cache. You can generally figure out what > the max number of processes the cache can handle by running a seed > on a test area and look at the number of tiles seeded per second, > as some point -n will have no effect which your process are just > waiting to write the cache. I would look at using this method to > test sqlite vs disk or other backends. The GeoTiff cache may be > promising as it appears to be non-blocking but experimental... > We use a riak cache with a riak leveldb backend which works well > for us but is a bit of a pain to manage as it difficult to delete > objects so it's not great for caches that need to be refreshed. > Finally you may want to test MapProxy as an alternative to > MapCache (we use both). MapProxy supports a non-blocking compact > cache that could solve your inode problem > Regards > On Fri, 8 Nov 2019 at 11:48, Sebastiano Laini > > wrote: > > We don?t create the data, we rely on OS (ordnance survey) to > supply us the maps files and then we publish them in our > service but I assume that probably we will need to create some > flow to improve it or download it in other format that is faster. > > Sebastiano Laini > > Web Developer > > Buchanan Computing > > *From:*Fawcett, David (MNIT) [mailto:david.fawcett at state.mn.us > ] > *Sent:* 08 November 2019 16:41 > *To:* Sebastiano Laini > >; 'Rahkonen > Jukka (MML)' >; > 'mapserver-users at lists.osgeo.org > ' > > > *Subject:* RE: mapcache seed speed optimization > > For data formats that are slower to read, would it add fit > your workflow to convert it to different data format before > creating the tiles? > > _______________________________________________ > mapserver-users mailing list > mapserver-users at lists.osgeo.org > > https://lists.osgeo.org/mailman/listinfo/mapserver-users > > > > > _______________________________________________ > mapserver-users mailing list > mapserver-users at lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/mapserver-users From stephenwoodbridge37 at gmail.com Tue Nov 12 08:51:41 2019 From: stephenwoodbridge37 at gmail.com (Stephen Woodbridge) Date: Tue, 12 Nov 2019 11:51:41 -0500 Subject: [mapserver-users] mapcache seed speed optimization In-Reply-To: <5a37d772b31f0374b78f88a1f5055034@blixtmail.se> References: <46b48dd679e44694879379dd76211e8c@C119S212VM042.msvyvi.vaha.local> <3416b08f42ba4954b77f10b1f21c7b30@BCExchange.BuchananComputing.co.uk> <4b13326044f0493d8f23e8c1ed67f2d0@BCExchange.BuchananComputing.co.uk> <5a37d772b31f0374b78f88a1f5055034@blixtmail.se> Message-ID: <19b17054-f652-50fd-a121-72832a607099@gmail.com> Thank Lars. You can now fetch it from https://github.com/woodbri/imaptools.com Enjoy, ? -Steve W On 11/11/2019 3:07 PM, lars.schylberg at blixtmail.se wrote: > Hi > > I have had a similar situation as You with large national data sets in > shape format. One of the biggest improvements for me has been to split > large shape files into smaller and > then use vector tile indexes in Mapserver. > The tool shp2tile has been working well for me. Stephen Woodbridge has > written that tool. It can be found at > http://imaptools.com/download-software.html > I think that You had to compile yourself on linux, but that was easy. > > I have an example where I have automated this in a script that runs > through all shape file in a directory and splits shape files larger > than a certain size and then creates the vector tile index files. > The code snippet that is part of larger import script can be found here: > https://gist.github.com/LarsSchy/644b37ab2b1e58b48808bfb2b57d0359 > > The script snippet also shows the normal optimizations we do. > > Hope this could give You some ideas how optimize and make Mapserver > and Mapcache run faster. I also ran into the problem with to many inodes. > That got solved by using sqlite as backend in Mapcache. > > Best regards / Lars Schylberg > > > > 11 november 2019 kl. 15:42, "Travis Kirstine" > >> > skrev: > > I may not be unrealistic for it take mapserver to 4-6 seconds to > generate 4096px image depending on the amount data being rendered > and the complexity of your styles / expressions / labels. If your > mapserver is able to generate a 1024x1024 in under a second that > is pretty good. Like Jukka and other suggest I would look at > translating the files to shp files with spatial indexes or postgis > tables with spatial and attribute indexes if necessary. > Preprocessing the data (simplification / prefiltering / point > thinning cluster) and using SCALETOKENs to reference different > source layers at different scales can lead to big performance > boosts at the mapserver end. > For us the biggest limiting factor has been the write blocking to > sqlite cache when seeding tiles, if you increase the number of > threads / process in mapcache seed those process will just end up > waiting to write to the cache. You can generally figure out what > the max number of processes the cache can handle by running a seed > on a test area and look at the number of tiles seeded per second, > as some point -n will have no effect which your process are just > waiting to write the cache. I would look at using this method to > test sqlite vs disk or other backends. The GeoTiff cache may be > promising as it appears to be non-blocking but experimental... > We use a riak cache with a riak leveldb backend which works well > for us but is a bit of a pain to manage as it difficult to delete > objects so it's not great for caches that need to be refreshed. > Finally you may want to test MapProxy as an alternative to > MapCache (we use both). MapProxy supports a non-blocking compact > cache that could solve your inode problem > Regards > On Fri, 8 Nov 2019 at 11:48, Sebastiano Laini > > wrote: > > We don?t create the data, we rely on OS (ordnance survey) to > supply us the maps files and then we publish them in our > service but I assume that probably we will need to create some > flow to improve it or download it in other format that is faster. > > Sebastiano Laini > > Web Developer > > Buchanan Computing > > *From:*Fawcett, David (MNIT) [mailto:david.fawcett at state.mn.us > ] > *Sent:* 08 November 2019 16:41 > *To:* Sebastiano Laini > >; 'Rahkonen > Jukka (MML)' >; > 'mapserver-users at lists.osgeo.org > ' > > > *Subject:* RE: mapcache seed speed optimization > > For data formats that are slower to read, would it add fit > your workflow to convert it to different data format before > creating the tiles? > > _______________________________________________ > mapserver-users mailing list > mapserver-users at lists.osgeo.org > > https://lists.osgeo.org/mailman/listinfo/mapserver-users > > > > > _______________________________________________ > mapserver-users mailing list > mapserver-users at lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/mapserver-users From bob.basques at ci.stpaul.mn.us Wed Nov 13 06:33:30 2019 From: bob.basques at ci.stpaul.mn.us (Basques, Bob (CI-StPaul)) Date: Wed, 13 Nov 2019 14:33:30 +0000 Subject: [mapserver-users] ** REMINDER ** : OSGeo Local Chapter (aka TCMUG) Meeting Message-ID: <9700FC1B-E3AA-4247-885B-48BBC445F32F@ci.stpaul.mn.us> All, Had a good time at Holmans Table last meeting. pic1 pic2 Lot?s of aircraft stuff to watch while there. We had a good (but loud) spot on the outside patio. The next meeting will be at Dual Citizen Brewery Co., in the Midway area. Should be easier for all you Minneapolis folks to attend. :c) bobb OSGeo Local Chapter (aka TCMUG) Meeting Scheduled: Nov 13, 2019 at 4:30 PM to 6:00 PM Location: Dual Citizen Brewing Co. 725 Raymond Ave, Saint Paul, MN 55114, United States http://www.flatearthbrewing.com/ https://goo.gl/maps/4sXKnj1nn9J2 _______________________________________________ Geomoose-users mailing list Geomoose-users at lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/geomoose-users -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: iCal-20191014-105540.ics Type: text/calendar Size: 1645 bytes Desc: iCal-20191014-105540.ics URL: From bob.basques at ci.stpaul.mn.us Wed Nov 13 12:40:49 2019 From: bob.basques at ci.stpaul.mn.us (Basques, Bob (CI-StPaul)) Date: Wed, 13 Nov 2019 20:40:49 +0000 Subject: [mapserver-users] [TCMUG] ** REMINDER ** : OSGeo Local Chapter (aka TCMUG) Meeting In-Reply-To: <9700FC1B-E3AA-4247-885B-48BBC445F32F@ci.stpaul.mn.us> References: <9700FC1B-E3AA-4247-885B-48BBC445F32F@ci.stpaul.mn.us> Message-ID: <20FB0F2A-B3BE-4099-90F0-AC633E506B01@ci.stpaul.mn.us> All, Just noticed that I had the wrong place inside of the CAL event I sent earlier. sorry about that. We are really meeting at the Dual Citizen Brewery in the Midway on Raymaond as the Mail listed. https://dualcitizenbrewing.com/ Sorry about the confusion. bobb On Nov 13, 2019, at 8:33 AM, Basques, Bob (CI-StPaul) > wrote: Think Before You Click: This email originated outside our organization. All, Had a good time at Holmans Table last meeting. pic1 pic2 Lot?s of aircraft stuff to watch while there. We had a good (but loud) spot on the outside patio. The next meeting will be at Dual Citizen Brewery Co., in the Midway area. Should be easier for all you Minneapolis folks to attend. :c) bobb OSGeo Local Chapter (aka TCMUG) Meeting Scheduled: Nov 13, 2019 at 4:30 PM to 6:00 PM Location: Dual Citizen Brewing Co. 725 Raymond Ave, Saint Paul, MN 55114, United States http://www.flatearthbrewing.com/ https://goo.gl/maps/4sXKnj1nn9J2 _______________________________________________ Geomoose-users mailing list Geomoose-users at lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/geomoose-users _______________________________________________ Tcmug mailing list Tcmug at lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/tcmug -------------- next part -------------- An HTML attachment was scrubbed... URL: From bob.basques at ci.stpaul.mn.us Thu Nov 14 07:12:49 2019 From: bob.basques at ci.stpaul.mn.us (Basques, Bob (CI-StPaul)) Date: Thu, 14 Nov 2019 15:12:49 +0000 Subject: [mapserver-users] OSGeo Local Chapter (aka TCMUG) Meeting Message-ID: <66143773-7BDF-4D62-AFDA-C3D941ED07E4@ci.stpaul.mn.us> Hello, Had a good time at Dual Citizen Brewery last evening. New faces (and stories) about GIS. All present had a good time. The next meeting will be at Black Stack Brewery. See you there. NOTE: Don?t worry about feeling left out if you aren?t a big drinker. I'm not a big drinker either. It?s all good, and everyone's welcome. bobb OSGeo Local Chapter (aka TCMUG) Meeting Scheduled: Dec 11, 2019 at 4:30 PM to 6:00 PM Location: BlackStack Brewing 755 Prior Ave N, Saint Paul, MN 55104, United States 755 Prior Ave N, St Paul, MN 55104 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: iCal-20191114-090030.ics Type: text/calendar Size: 1729 bytes Desc: iCal-20191114-090030.ics URL: From jaap at solarmonkey.nl Thu Nov 14 07:50:38 2019 From: jaap at solarmonkey.nl (Jaap Donker) Date: Thu, 14 Nov 2019 16:50:38 +0100 Subject: [mapserver-users] Mapserver bumped from 7.0.7 to 7.4.1 leads to wrong projection of features Message-ID: Hi, Upon updating mapserver, projections for the features we use in our webapp were off. We updated from version 7.0.7 to version 7.4.1. I'd like to stress that nothing else changed in setup during testing. In the images below you can see the situation before (left) and after (right). The blue lines are polygons representing seamlines for the orthomosaic of aerials we are using. This change in projection is troubling. Original situation: https://pasteboard.co/IGFtgdY.png New situation: https://pasteboard.co/IGFtxfs.png Going through the backlog I see that there's some changes in PROJ4, but I'm a bit at a loss as to how best to investigate this issue. Any help or guidance would be very much appreciated. Regards, Jaap -------------- next part -------------- An HTML attachment was scrubbed... URL: From jukka.rahkonen at maanmittauslaitos.fi Thu Nov 14 09:53:46 2019 From: jukka.rahkonen at maanmittauslaitos.fi (Rahkonen Jukka (MML)) Date: Thu, 14 Nov 2019 17:53:46 +0000 Subject: [mapserver-users] Mapserver bumped from 7.0.7 to 7.4.1 leads to wrong projection of features Message-ID: Hi, Please add relevant parts of your mapfile. Obviously all that relates to projections and an example of the client side requests as well if you are changing the coordinate system on the fly. -Jukka Rahkonen- L?hett?j?: mapserver-users Puolesta Jaap Donker L?hetetty: torstai 14. marraskuuta 2019 17.51 Vastaanottaja: mapserver-users at lists.osgeo.org Aihe: [mapserver-users] Mapserver bumped from 7.0.7 to 7.4.1 leads to wrong projection of features Hi, Upon updating mapserver, projections for the features we use in our webapp were off. We updated from version 7.0.7 to version 7.4.1. I'd like to stress that nothing else changed in setup during testing. In the images below you can see the situation before (left) and after (right). The blue lines are polygons representing seamlines for the orthomosaic of aerials we are using. This change in projection is troubling. Original situation: https://pasteboard.co/IGFtgdY.png New situation: https://pasteboard.co/IGFtxfs.png Going through the backlog I see that there's some changes in PROJ4, but I'm a bit at a loss as to how best to investigate this issue. Any help or guidance would be very much appreciated. Regards, Jaap -------------- next part -------------- An HTML attachment was scrubbed... URL: From jpass at bgs.ac.uk Fri Nov 15 02:09:45 2019 From: jpass at bgs.ac.uk (Passmore, James H.) Date: Fri, 15 Nov 2019 10:09:45 +0000 Subject: [mapserver-users] Mapserver bumped from 7.0.7 to 7.4.1 leads to wrong projection of features Message-ID: Hi Jaap, What is/are the CRS used, and what is the shift in position between the features? James This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UK Research and Innovation does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UK Research and Innovation business are solely those of the author and do not represent the views of UK Research and Innovation. From bobby.biran at uk.ibm.com Fri Nov 15 06:29:19 2019 From: bobby.biran at uk.ibm.com (Bobby Biran) Date: Fri, 15 Nov 2019 14:29:19 +0000 Subject: [mapserver-users] Query Message-ID: An HTML attachment was scrubbed... URL: From fl_v at inbox.ru Fri Nov 15 09:12:34 2019 From: fl_v at inbox.ru (=?UTF-8?B?VmxhZGltaXI=?=) Date: Fri, 15 Nov 2019 20:12:34 +0300 Subject: [mapserver-users] =?utf-8?q?Question_about_python_mapscript_and_p?= =?utf-8?q?oint_style=2E?= Message-ID: <1573837954.714622239@f115.i.mail.ru> Hi all! ? I unsuccessfully try to style ?Point? layer with following code: (first) ? ? style = mapscript.styleObj(classObj) ? ? style.symbolname = 'circle' ? ? style.size = 10 ? ? style.color = mapscript.colorObj(255, 0, 0) ? No error, no picture as result of using this method. But similar approach works for ?Line? and ?Polygon? layers. ? Also I figured out working method for the ?Point? layer: (second) classObj.updateFromString("CLASS STYLE COLOR 255 0 0 SIZE 10? SYMBOL 'circle' END END") ? O utput of ?layer.convertToString()? is the same for both methods : LAYER ??? CONNECTION \"dbname='db' host='db' user='user' password='1'\" ??? CONNECTIONTYPE POSTGIS ? ??? DATA \"value FROM (SELECT value FROM points WHERE ST_Intersects(value, !BOX!)) AS subquery USING UNIQUE id USING srid = 3857\" ??? LABELITEM \"name\" ??? NAME \"points\" ??? STATUS ON ??? TYPE POINT ??? UNITS METERS ??? CLASS ??????? NAME \"points\" ??????? STYLE??? ? ??????????? COLOR 255 0 0 ??????????? SIZE 10??? ? ??????????? SYMBOL \"circle\" ??????? END # STYLE ??? END # CLASS END # LAYER ? How to get working ?Point? style using first method? MapServer version: 7.2.1 ? Thanks! -- Regards, Vladimir. ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From pschmitt at gmail.com Fri Nov 15 11:02:18 2019 From: pschmitt at gmail.com (Peter Schmitt) Date: Fri, 15 Nov 2019 12:02:18 -0700 Subject: [mapserver-users] Configuring a layer with one output with multiple input projections Message-ID: Hi, Is it possible to configure one raster layer with a single output projection (epsg:4326) where input rasters are in different UTM zones? I'd like to avoid reprojecting the source data if possible. The rasters will be determined from a Postgis-based tile index where raster extents are stored in epsg:4326. The database will also track what UTM zone the source raster is in. Is it possible to pass this projection info from the database to the PROJECTION block? I tried an experiment which does not quite work. I built a tile index with two tifs in different UTM zones: gdaltindex -t_srs EPSG:4326 test_tindex.gpkg zone45.tif zone46.tif Then my layers are: LAYER NAME "test_tindex" TYPE POLYGON CONNECTIONTYPE OGR CONNECTION "test_tindex.gpkg" STATUS OFF PROJECTION # proj4 config for +init=epsg:4326 http://spatialreference.org/ref/epsg/wgs-84/ "+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs" END CLASS STYLE WIDTH 2 OUTLINECOLOR 0 255 0 END END END LAYER NAME "test" TYPE RASTER METADATA "ows_title" "test" "ows_srs" "EPSG:4326 EPSG:3857" END TILEINDEX "test_tindex" STATUS OFF PROJECTION AUTO END END When I make a WMS request for test in an area where the images overlap, it seems MapServer stops rendering rasters after the first image in the tile index is found... so in this case since zone45.tif was the first image in the tile index, that's what I see for a request bbox in the overlap region. If I swap the order of files in the command to build the tindex, the overlapped area only shows me zone46.tif. $ mapserv -v MapServer version 7.2.2 OUTPUT=PNG OUTPUT=JPEG SUPPORTS=PROJ SUPPORTS=AGG SUPPORTS=FREETYPE SUPPORTS=ICONV SUPPORTS=FRIBIDI SUPPORTS=WMS_SERVER SUPPORTS=WFS_SERVER SUPPORTS=WCS_SERVER SUPPORTS=FASTCGI SUPPORTS=GEOS SUPPORTS=PBF INPUT=JPEG INPUT=POSTGIS INPUT=OGR INPUT=GDAL INPUT=SHAPEFILE Thanks, Pete -------------- next part -------------- An HTML attachment was scrubbed... URL: From jukka.rahkonen at maanmittauslaitos.fi Fri Nov 15 11:19:20 2019 From: jukka.rahkonen at maanmittauslaitos.fi (Rahkonen Jukka (MML)) Date: Fri, 15 Nov 2019 19:19:20 +0000 Subject: [mapserver-users] Configuring a layer with one output with multiple input projections Message-ID: <0013edb2062446d8974db54eae49d495@C119S212VM042.msvyvi.vaha.local> Hi, Please read https://mapserver.org/optimization/tileindex.html#tileindex-mixed-srs and https://mapserver.org/development/rfc/ms-rfc-100.html? -Jukka Rahkonen- L?hett?j?: mapserver-users Puolesta Peter Schmitt L?hetetty: perjantai 15. marraskuuta 2019 21.02 Vastaanottaja: mapserver-users at lists.osgeo.org Aihe: [mapserver-users] Configuring a layer with one output with multiple input projections Hi, Is it possible to configure one raster layer with a single output projection (epsg:4326) where input rasters are in different UTM zones? I'd like to avoid reprojecting the source data if possible. The rasters will be determined from a Postgis-based tile index where raster extents are stored in epsg:4326. The database will also track what UTM zone the source raster is in. Is it possible to pass this projection info from the database to the PROJECTION block? I tried an experiment which does not quite work. I built a tile index with two tifs in different UTM zones: gdaltindex -t_srs EPSG:4326 test_tindex.gpkg zone45.tif zone46.tif Then my layers are: LAYER NAME "test_tindex" TYPE POLYGON CONNECTIONTYPE OGR CONNECTION "test_tindex.gpkg" STATUS OFF PROJECTION # proj4 config for +init=epsg:4326 http://spatialreference.org/ref/epsg/wgs-84/ "+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs" END CLASS STYLE WIDTH 2 OUTLINECOLOR 0 255 0 END END END LAYER NAME "test" TYPE RASTER METADATA "ows_title" "test" "ows_srs" "EPSG:4326 EPSG:3857" END TILEINDEX "test_tindex" STATUS OFF PROJECTION AUTO END END When I make a WMS request for test in an area where the images overlap, it seems MapServer stops rendering rasters after the first image in the tile index is found... so in this case since zone45.tif was the first image in the tile index, that's what I see for a request bbox in the overlap region. If I swap the order of files in the command to build the tindex, the overlapped area only shows me zone46.tif. $ mapserv -v MapServer version 7.2.2 OUTPUT=PNG OUTPUT=JPEG SUPPORTS=PROJ SUPPORTS=AGG SUPPORTS=FREETYPE SUPPORTS=ICONV SUPPORTS=FRIBIDI SUPPORTS=WMS_SERVER SUPPORTS=WFS_SERVER SUPPORTS=WCS_SERVER SUPPORTS=FASTCGI SUPPORTS=GEOS SUPPORTS=PBF INPUT=JPEG INPUT=POSTGIS INPUT=OGR INPUT=GDAL INPUT=SHAPEFILE Thanks, Pete -------------- next part -------------- An HTML attachment was scrubbed... URL: From sdlime at gmail.com Fri Nov 15 14:49:02 2019 From: sdlime at gmail.com (Steve Lime) Date: Fri, 15 Nov 2019 16:49:02 -0600 Subject: [mapserver-users] Query In-Reply-To: References: Message-ID: That was setup by one of the MapServer PSC members and can be considered official - it's a little out of date however (MapServer 7.2 I believe). On Fri, Nov 15, 2019 at 8:54 AM Bobby Biran wrote: > Hi MapServer > > > > We can across https://hub.docker.com/u/mapserver in DockerHub and wanted > to confirm if this is an official account from yourselves ? > > > > Regards > > Bobby > > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > _______________________________________________ > mapserver-users mailing list > mapserver-users at lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/mapserver-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.smith.erdc at gmail.com Fri Nov 15 14:52:59 2019 From: michael.smith.erdc at gmail.com (Michael Smith) Date: Fri, 15 Nov 2019 17:52:59 -0500 Subject: [mapserver-users] Query In-Reply-To: References: Message-ID: <7DAE10B2-4625-4956-B3CC-734ECB0C65E5@gmail.com> I have it as a task to push updates to it this weekend. I?ll bring it up to date writhe the current MapServer release. Michael Smith > On Nov 15, 2019, at 5:49 PM, Steve Lime wrote: > > ? > That was setup by one of the MapServer PSC members and can be considered official - it's a little out of date however (MapServer 7.2 I believe). > >> On Fri, Nov 15, 2019 at 8:54 AM Bobby Biran wrote: >> Hi MapServer >> >> We can across https://hub.docker.com/u/mapserver in DockerHub and wanted to confirm if this is an official account from yourselves ? >> >> Regards >> Bobby >> >> >> Unless stated otherwise above: >> IBM United Kingdom Limited - Registered in England and Wales with number 741598. >> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU >> >> _______________________________________________ >> mapserver-users mailing list >> mapserver-users at lists.osgeo.org >> https://lists.osgeo.org/mailman/listinfo/mapserver-users > _______________________________________________ > mapserver-users mailing list > mapserver-users at lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/mapserver-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From pschmitt at gmail.com Fri Nov 15 15:54:30 2019 From: pschmitt at gmail.com (Peter Schmitt) Date: Fri, 15 Nov 2019 16:54:30 -0700 Subject: [mapserver-users] Configuring a layer with one output with multiple input projections In-Reply-To: <0013edb2062446d8974db54eae49d495@C119S212VM042.msvyvi.vaha.local> References: <0013edb2062446d8974db54eae49d495@C119S212VM042.msvyvi.vaha.local> Message-ID: Thanks Jukka! I was looking at https://www.mapserver.org/mapfile/projection.html, which doesn't mention TILESRS. I just submitted a PR to address that: https://github.com/mapserver/docs/pull/293 Cheers, Pete On Fri, Nov 15, 2019 at 12:19 PM Rahkonen Jukka (MML) < jukka.rahkonen at maanmittauslaitos.fi> wrote: > Hi, > > > > Please read > https://mapserver.org/optimization/tileindex.html#tileindex-mixed-srs and > https://mapserver.org/development/rfc/ms-rfc-100.html? > > > > -Jukka Rahkonen- > > *L?hett?j?:* mapserver-users *Puolesta > *Peter Schmitt > *L?hetetty:* perjantai 15. marraskuuta 2019 21.02 > *Vastaanottaja:* mapserver-users at lists.osgeo.org > *Aihe:* [mapserver-users] Configuring a layer with one output with > multiple input projections > > > > Hi, > > > > Is it possible to configure one raster layer with a single output > projection (epsg:4326) where input rasters are in different UTM zones? I'd > like to avoid reprojecting the source data if possible. > > > > The rasters will be determined from a Postgis-based tile index where > raster extents are stored in epsg:4326. The database will also track what > UTM zone the source raster is in. Is it possible to pass this > projection info from the database to the PROJECTION block? > > > > I tried an experiment which does not quite work. I built a tile index > with two tifs in different UTM zones: > > > > gdaltindex -t_srs EPSG:4326 test_tindex.gpkg zone45.tif zone46.tif > > > > Then my layers are: > > > > LAYER > NAME "test_tindex" > TYPE POLYGON > > CONNECTIONTYPE OGR > CONNECTION "test_tindex.gpkg" > STATUS OFF > PROJECTION > # proj4 config for +init=epsg:4326 > http://spatialreference.org/ref/epsg/wgs-84/ > "+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs" > END > CLASS > STYLE > WIDTH 2 > OUTLINECOLOR 0 255 0 > END > END > END > > LAYER > NAME "test" > TYPE RASTER > METADATA > "ows_title" "test" > "ows_srs" "EPSG:4326 EPSG:3857" > END > TILEINDEX "test_tindex" > STATUS OFF > > PROJECTION > AUTO > END > END > > > > When I make a WMS request for test in an area where the images overlap, it > seems MapServer stops rendering rasters after the first image in the tile > index is found... so in this case since zone45.tif was the first image in > the tile index, that's what I see for a request bbox in the overlap > region. If I swap the order of files in the command to build the tindex, > the overlapped area only shows me zone46.tif. > > > > $ mapserv -v > > MapServer version 7.2.2 OUTPUT=PNG OUTPUT=JPEG SUPPORTS=PROJ SUPPORTS=AGG > SUPPORTS=FREETYPE SUPPORTS=ICONV SUPPORTS=FRIBIDI SUPPORTS=WMS_SERVER > SUPPORTS=WFS_SERVER SUPPORTS=WCS_SERVER SUPPORTS=FASTCGI SUPPORTS=GEOS > SUPPORTS=PBF INPUT=JPEG INPUT=POSTGIS INPUT=OGR INPUT=GDAL INPUT=SHAPEFILE > > > > Thanks, > > Pete > -- Pete -------------- next part -------------- An HTML attachment was scrubbed... URL: From jaap at solarmonkey.nl Tue Nov 19 05:02:45 2019 From: jaap at solarmonkey.nl (Jaap Donker) Date: Tue, 19 Nov 2019 14:02:45 +0100 Subject: [mapserver-users] Mapserver bumped from 7.0.7 to 7.4.1 leads to wrong projection of features In-Reply-To: References: Message-ID: Thanks for your response. In our frontend we use the following projection string for EPSG:28992: +proj=sterea +lat_0=52.15616055555555 +lon_0=5.38763888888889 +k=0.9999079 +x_0=155000 +y_0=463000 +ellps=bessel +towgs84=565.417,50.3319,465.552,-0.398957,0.343988,-1.8774,4.0725 +units=m +no_defs I hope these parts of the map file are informative enough: MAP PROJECTION 'init=epsg:28992' END OUTPUTFORMAT NAME "GTiff" DRIVER GDAL/GTiff MIMETYPE "image/tiff" IMAGEMODE INT16 EXTENSION "tif" END OUTPUTFORMAT NAME "GTiff-FLOAT" DRIVER GDAL/GTiff MIMETYPE "image/tiff" IMAGEMODE FLOAT32 EXTENSION "tif" END OUTPUTFORMAT NAME "geojson" DRIVER "OGR/GEOJSON" MIMETYPE "application/json; subtype=geojson; charset=utf-8" FORMATOPTION "STORAGE=memory" FORMATOPTION "FORM=SIMPLE" END ... LAYER METADATA "wfs_srs" "EPSG:4326" "wfs_enable_request" "*" "wfs_getfeature_formatlist" "geojson" "gml_include_items" "all" "wfs_maxfeatures" "100" # Max features to load END TYPE POLYGON STATUS ON PROJECTION "init=epsg:4326" END END END Kind regards, Jaap On Thu, 14 Nov 2019 at 18:53, Rahkonen Jukka (MML) < jukka.rahkonen at maanmittauslaitos.fi> wrote: > Hi, > > > > Please add relevant parts of your mapfile. Obviously all that relates to > projections and an example of the client side requests as well if you are > changing the coordinate system on the fly. > > > > -Jukka Rahkonen- > > > > *L?hett?j?:* mapserver-users *Puolesta > *Jaap Donker > *L?hetetty:* torstai 14. marraskuuta 2019 17.51 > *Vastaanottaja:* mapserver-users at lists.osgeo.org > *Aihe:* [mapserver-users] Mapserver bumped from 7.0.7 to 7.4.1 leads to > wrong projection of features > > > > Hi, > > > > Upon updating mapserver, projections for the features we use in our webapp > were off. > > We updated from version 7.0.7 to version 7.4.1. I'd like to stress that > nothing else changed in setup during testing. > > In the images below you can see the situation before (left) and after > (right). The blue lines are polygons representing seamlines for the > orthomosaic of aerials we are using. > > This change in projection is troubling. > > > > Original situation: > > https://pasteboard.co/IGFtgdY.png > > > > New situation: > > https://pasteboard.co/IGFtxfs.png > > > > Going through the backlog I see that there's some changes in PROJ4, but > I'm a bit at a loss as to how best to investigate this issue. > > > > Any help or guidance would be very much appreciated. > > > > Regards, > > > > Jaap > -------------- next part -------------- An HTML attachment was scrubbed... URL: From traviskirstine at gmail.com Tue Nov 19 08:11:11 2019 From: traviskirstine at gmail.com (Travis Kirstine) Date: Tue, 19 Nov 2019 11:11:11 -0500 Subject: [mapserver-users] MapCache clusters In-Reply-To: <832135f0016547ca8542dd294f82b574@BCExchange.BuchananComputing.co.uk> References: <832135f0016547ca8542dd294f82b574@BCExchange.BuchananComputing.co.uk> Message-ID: I don't think multiple instances of MapCache will help for seeding, in general most of the seeding time depends on the zoom level, with each higher zoom level taking 4 x the time as the proceeding zoom level (e.g. zoom 20 take 4 x the time to seed as zoom 19) . This can vary depending on your source service as there may be certain map scales / zoom levels that take longer per tile to render due to expression, labels, # of features etc... If you were to seed two levels for each mapcache instance you still would be waiting for the higher zoom levels to complete. Using multiple load balanced mapserver nodes across multiple VM may improve things a bit however if all the VMs are hosted on the same physical server this may be slower overall. For seeding I have heard you can use 2x the number of cores so you could set up a single mapcache server using -n 16 pointing at the load balanced source. Regards On Mon, 18 Nov 2019 at 10:47, Sebastiano Laini < Sebastiano.Laini at buchanancomputing.co.uk> wrote: > Hi All, > > > > I was wondering if is possible to dockerise my server and then use > multiple cluster to seed the cache? I know I can use multiple processes > with the ?n and ?p flag though I?m wondering there is a limit on how much > you can really increase those number. > > > > E.g. I dockerise the server and I spawn a 8core VM with 16GB ram, I can > use sure ?p 8 or maybe 10 but still I think that will require like 60 days > to seed the whole UK with our data, can I spawn another machine and speed > up the process? For example another 8core VM. > > > > We have 4 caches with different style that are updated twice a year so if > it takes me 60 days for each by the time I finish it?s already time to > start again. > > > > I was wondering maybe I can run multiple VM and seed one or maybe 2 levels > on each VM or if I spawn a new VM the original process will take advantage > of the new VM? > > In a VM I?m going to have MapServer and MapCache and I will use a > subdomain under a load balancer just to seed while the domain will be use > just for the seeded cache and in the same way will be under a load balancer. > > > > Regards, > > > > Sebastiano Laini > > Web Developer > > Buchanan Computing > _______________________________________________ > mapserver-users mailing list > mapserver-users at lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/mapserver-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From geodaten at wipflerplan.de Wed Nov 20 04:52:50 2019 From: geodaten at wipflerplan.de (=?iso-8859-1?Q?Markus_L=FCtke_Brintrup?=) Date: Wed, 20 Nov 2019 13:52:50 +0100 Subject: [mapserver-users] Info Template Message-ID: <002701d59fa1$6a821a40$3f864ec0$@wipflerplan.de> Hallo Liste, besteht die M?glichkeit in der Template.html eines Map-Projektes eine Datenbankanbindung an zu steuern? Beispiel: Projektname: [Werte aus MYSQL-DB] Freundliche Gr??e Markus L?tke Brintrup WipflerPLAN Niederlassung Donauries Telefon: 09081 27509-44 Telefax: 09081 27509-50 mlb at wipflerplan.de www.wipflerplan.de WipflerPLAN Planungsgesellschaft mbH Standort N?rdlingen An der Lach 11a 86720 N?rdlingen Amtsgericht Ingolstadt HRB 190595 Gesch?ftsf?hrer: Wilhelm Wipfler Ursula Burkart Adelheid Buchard -------------- next part -------------- An HTML attachment was scrubbed... URL: From jukka.rahkonen at maanmittauslaitos.fi Wed Nov 20 05:29:15 2019 From: jukka.rahkonen at maanmittauslaitos.fi (Rahkonen Jukka (MML)) Date: Wed, 20 Nov 2019 13:29:15 +0000 Subject: [mapserver-users] Info Template Message-ID: Hi Markus, I am not sure if you'd prefer me to answer in Finnish but English is easier for the majority of readers. Certainly it is possible to use data from a database with templates yes but you did not tell the details about for what purpose you are trying to use the templates. The DATA (perhaps with CONNECTION) that is used in LAYER https://mapserver.org/mapfile/layer.html can also be used as a data source for templates. An example from an old mail http://osgeo-org.1560.x6.nabble.com/html-tags-in-GetFeatureInfo-td5224781.html contains a template that defines the html format of GetFeatureInfo. Each "item name" is a name of the attribute in the source data. If vector data for the LAYER comes from MYSQL then the attributes are read from MYSQL as well.

NIMI

[item name=NIMI format="$value" escape=none]

LIS?TIEDOT

[item name=KIELLONLISATIETO format="$value" escape=none]

VOIMASSAOLO

[item name=VOIMASSAOLO format="$value" escape=none]

YHTEYSTIEDOT

[item name=YHTEYSTIEDOT format="$value" escape=none] -Jukka Rahkonen- L?hett?j?: mapserver-users Puolesta Markus L?tke Brintrup L?hetetty: keskiviikko 20. marraskuuta 2019 14.53 Vastaanottaja: mapserver-users at lists.osgeo.org Aihe: [mapserver-users] Info Template Hallo Liste, besteht die M?glichkeit in der Template.html eines Map-Projektes eine Datenbankanbindung an zu steuern? Beispiel: Projektname: [Werte aus MYSQL-DB] Freundliche Gr??e Markus L?tke Brintrup WipflerPLAN Niederlassung Donauries Telefon: 09081 27509-44 Telefax: 09081 27509-50 mlb at wipflerplan.de www.wipflerplan.de WipflerPLAN Planungsgesellschaft mbH Standort N?rdlingen An der Lach 11a 86720 N?rdlingen Amtsgericht Ingolstadt HRB 190595 Gesch?ftsf?hrer: Wilhelm Wipfler Ursula Burkart Adelheid Buchard -------------- next part -------------- An HTML attachment was scrubbed... URL: From sdlime at gmail.com Wed Nov 20 08:59:17 2019 From: sdlime at gmail.com (Steve Lime) Date: Wed, 20 Nov 2019 10:59:17 -0600 Subject: [mapserver-users] Info Template In-Reply-To: References: Message-ID: For the sake of completeness, "$value" is the default format and doesn't need to be explicitly set. --Steve On Wed, Nov 20, 2019 at 7:29 AM Rahkonen Jukka (MML) < jukka.rahkonen at maanmittauslaitos.fi> wrote: > Hi Markus, > > > > I am not sure if you?d prefer me to answer in Finnish but English is > easier for the majority of readers. > > > > Certainly it is possible to use data from a database with templates yes > but you did not tell the details about for what purpose you are trying to > use the templates. The DATA (perhaps with CONNECTION) that is used in LAYER > https://mapserver.org/mapfile/layer.html can also be used as a data > source for templates. > > > > An example from an old mail > http://osgeo-org.1560.x6.nabble.com/html-tags-in-GetFeatureInfo-td5224781.html > contains a template that defines the html format of GetFeatureInfo. Each > ?item name? is a name of the attribute in the source data. If vector data > for the LAYER comes from MYSQL then the attributes are read from MYSQL as > well. > > > > >

NIMI

> [item name=NIMI format="$value" escape=none] >

LIS?TIEDOT

> [item name=KIELLONLISATIETO format="$value" escape=none] >

VOIMASSAOLO

> [item name=VOIMASSAOLO format="$value" escape=none] >

YHTEYSTIEDOT

> [item name=YHTEYSTIEDOT format="$value" escape=none] > > > > > -Jukka Rahkonen- > > > > *L?hett?j?:* mapserver-users *Puolesta > *Markus L?tke Brintrup > *L?hetetty:* keskiviikko 20. marraskuuta 2019 14.53 > *Vastaanottaja:* mapserver-users at lists.osgeo.org > *Aihe:* [mapserver-users] Info Template > > > > Hallo Liste, > > > > besteht die M?glichkeit in der Template.html eines Map-Projektes eine > Datenbankanbindung an zu steuern? > > Beispiel: > > > > Projektname: > > [Werte aus MYSQL-DB] > > > > > > Freundliche Gr??e > > > > Markus L?tke Brintrup > > > > WipflerPLAN > > Niederlassung Donauries > > > > Telefon: 09081 27509-44 > > Telefax: 09081 27509-50 > > mlb at wipflerplan.de > > *www.wipflerplan.de * > > > > WipflerPLAN Planungsgesellschaft mbH > > Standort N?rdlingen > > An der Lach 11a > > 86720 N?rdlingen > > Amtsgericht Ingolstadt HRB 190595 > > Gesch?ftsf?hrer: > > Wilhelm Wipfler > > Ursula Burkart > > Adelheid Buchard > > > > > _______________________________________________ > mapserver-users mailing list > mapserver-users at lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/mapserver-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From sethg at geographika.co.uk Fri Nov 22 12:34:48 2019 From: sethg at geographika.co.uk (Seth G) Date: Fri, 22 Nov 2019 21:34:48 +0100 Subject: [mapserver-users] =?utf-8?q?Question_about_python_mapscript_and_p?= =?utf-8?q?oint_style=2E?= In-Reply-To: <1573837954.714622239@f115.i.mail.ru> References: <1573837954.714622239@f115.i.mail.ru> Message-ID: Hi Vladimir, Could you try the following syntax? new_style = new mapscript.styleObj() new_style.symbolname = 'circle' new_style.size = 10 the_class.insertStyle(new_style) Also make sure your variables aren't overriding the class names - e.g. maybe use different variable names than classObj. Regards, Seth -- web:http://geographika.co.uk twitter: @geographika On Fri, Nov 15, 2019, at 6:12 PM, Vladimir wrote: > Hi all! > > I unsuccessfully try to style ?Point? layer with following code: > (first) > style = mapscript.styleObj(classObj) > style.symbolname = 'circle' > style.size = 10 > style.color = mapscript.colorObj(255, 0, 0) > > No error, no picture as result of using this method. > But similar approach works for ?Line? and ?Polygon? layers. > > Also I figured out working method for the ?Point? layer: > (second) > classObj.updateFromString("CLASS STYLE COLOR 255 0 0 SIZE 10 SYMBOL 'circle' END END") > > Output of ?layer.convertToString()? is the same for both methods: > LAYER > CONNECTION \"dbname='db' host='db' user='user' password='1'\" > CONNECTIONTYPE POSTGIS > DATA \"value FROM (SELECT value FROM points WHERE ST_Intersects(value, !BOX!)) AS subquery USING UNIQUE id USING srid = 3857\" > LABELITEM \"name\" > NAME \"points\" > STATUS ON > TYPE POINT > UNITS METERS > CLASS > NAME \"points\" > STYLE > COLOR 255 0 0 > SIZE 10 > SYMBOL \"circle\" > END # STYLE > END # CLASS > END # LAYER > > How to get working ?Point? style using first method? > MapServer version: 7.2.1 > > Thanks! > > -- > Regards, Vladimir. > > > _______________________________________________ > mapserver-users mailing list > mapserver-users at lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/mapserver-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From sethg at geographika.co.uk Sat Nov 23 07:58:25 2019 From: sethg at geographika.co.uk (Seth G) Date: Sat, 23 Nov 2019 16:58:25 +0100 Subject: [mapserver-users] =?utf-8?q?Question_about_python_mapscript_and_p?= =?utf-8?q?oint_style=2E?= In-Reply-To: <1574516646.606499529@f734.i.mail.ru> References: <1573837954.714622239@f115.i.mail.ru> <1574516646.606499529@f734.i.mail.ru> Message-ID: Hi, The following example works for me - 2 styles are in the output Mapfile. Maybe there is an issue elsewhere? Is there a Python error, or is the output not what you are expecting? import mapscript m = mapscript.fromstring(""" MAP LAYER NAME "test" TYPE POINT CLASS NAME "0" STYLE COLOR 0 0 0 SYMBOL 1 SIZE 13 END END END END """) lyr = m.getLayerByName('test') cls = lyr.getClass(0) style = mapscript.styleObj() style.color.setRGB(0, 0, 0) style.symbol = 1 style.size = 3 cls.insertStyle(style) print(m.convertToString()) For Mapfile manipulation in Python you could also try https://pypi.org/project/mappyfile/ Regards, Seth -- web:http://geographika.co.uk twitter: @geographika On Sat, Nov 23, 2019, at 2:44 PM, Vladimir wrote: > Hi all again! > > Seth, thank you for response. > I tried this way with a single Point layer, unfortunately this also does not work. > > -- > Regards, Vladimir. >> 23 2019, 2:41 +06:00 ?? Seth G : >> >> Hi Vladimir, >> >> Could you try the following syntax? >> >> new_style = new mapscript.styleObj() >> new_style.symbolname = 'circle' >> new_style.size = 10 >> the_class.insertStyle(new_style) >> >> >> Also make sure your variables aren't overriding the class names - e.g. maybe use different variable names than classObj. >> >> Regards, >> >> Seth >> >> -- >> web:http://geographika.co.uk >> twitter: @geographika >> >> >> On Fri, Nov 15, 2019, at 6:12 PM, Vladimir wrote: >>> Hi all! >>> >>> I unsuccessfully try to style ?Point? layer with following code: >>> (first) >>> style = mapscript.styleObj(classObj) >>> style.symbolname = 'circle' >>> style.size = 10 >>> style.color = mapscript.colorObj(255, 0, 0) >>> >>> No error, no picture as result of using this method. >>> But similar approach works for ?Line? and ?Polygon? layers. >>> >>> Also I figured out working method for the ?Point? layer: >>> (second) >>> classObj.updateFromString("CLASS STYLE COLOR 255 0 0 SIZE 10 SYMBOL 'circle' END END") >>> >>> Output of ?layer.convertToString()? is the same for both methods: >>> LAYER >>> CONNECTION \"dbname='db' host='db' user='user' password='1'\" >>> CONNECTIONTYPE POSTGIS >>> DATA \"value FROM (SELECT value FROM points WHERE ST_Intersects(value, !BOX!)) AS subquery USING UNIQUE id USING srid = 3857\" >>> LABELITEM \"name\" >>> NAME \"points\" >>> STATUS ON >>> TYPE POINT >>> UNITS METERS >>> CLASS >>> NAME \"points\" >>> STYLE >>> COLOR 255 0 0 >>> SIZE 10 >>> SYMBOL \"circle\" >>> END # STYLE >>> END # CLASS >>> END # LAYER >>> >>> How to get working ?Point? style using first method? >>> MapServer version: 7.2.1 >>> >>> Thanks! >>> >>> -- >>> Regards, Vladimir. >>> >>> >>> _______________________________________________ >>> mapserver-users mailing list >>> mapserver-users at lists.osgeo.org >>> https://lists.osgeo.org/mailman/listinfo/mapserver-users >> >> _______________________________________________ >> mapserver-users mailing list >> mapserver-users at lists.osgeo.org >> https://lists.osgeo.org/mailman/listinfo/mapserver-users > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From klassen.js at gmail.com Fri Nov 29 11:51:24 2019 From: klassen.js at gmail.com (Jim Klassen) Date: Fri, 29 Nov 2019 13:51:24 -0600 Subject: [mapserver-users] mapserver cgi-bin security In-Reply-To: <487fc53f105145b890c5dfbf0a0ddeef@BCExchange.BuchananComputing.co.uk> References: <487fc53f105145b890c5dfbf0a0ddeef@BCExchange.BuchananComputing.co.uk> Message-ID: <1860d052-079e-022b-06ea-fc70000aa3a6@gmail.com> You didn't provide enough information to give specific recommendations.? However, generally there are some things you can do. Look at the environment variables MapServer supports for selecting the mapfile.? If you let that pass into the QUERYSTRING, someone can specify the map= paramater multiple times and I'm not sure which one takes precedence. Look for: ??? MS_MAPFILE ??? MS_MAPFILE_PATTERN ??? MS_MAP_NO_PATH Also, make sure you set the VALIDATION blocks in your mapfiles for any substitution parameters you allow in from the request. You should avoid allowing substitutions into the DATA field (assuming here that SQL injection means your mapfile is hitting a database without a fixed SQL statement). As as a defense in depth measure, you should make sure that the account MapServer uses to connect to your database has the minimum privileges necessary (generally only SELECT on a handful of tables). I'm not sure what running through that script is getting you except for slowing down the response time and opening you up to potential bugs in your script and bash.? You can set environment variables and filter based on method directly in Apache (and I presume other common web servers as well). On 11/25/19 4:19 AM, Sebastiano Laini wrote: > > Hi All, > > We submitted our new system to a pen-test and apparently is pretty > easy to be infected by SQL-injection, anyone came across this before? > > We run the feature layer through a cgi-bin request, see the script below > > #! /bin/sh > > MAPSERV="/var/?/cgi-bin/mapserv" > > MAPFILE="/var/?/twfeat.map" > > if [ "${REQUEST_METHOD}" = "GET" ]; then > > ?? if [ -z "${QUERY_STRING}" ]; then > > ????? QUERY_STRING="map=${MAPFILE}" > > ?? else > > QUERY_STRING="map=${MAPFILE}&${QUERY_STRING}" > > ?? fi > > ?? exec ${MAPSERV} > > else > > ?? echo "Sorry, I only understand GET requests." > > fi > > exit 1 > > how can you sanitize or implement security layers? > > Kind Regards, > > Sebastiano Laini > > Web Developer > > Buchanan Computing > > > _______________________________________________ > mapserver-users mailing list > mapserver-users at lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/mapserver-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From even.rouault at spatialys.com Fri Nov 29 13:05:25 2019 From: even.rouault at spatialys.com (Even Rouault) Date: Fri, 29 Nov 2019 22:05:25 +0100 Subject: [mapserver-users] mapserver cgi-bin security In-Reply-To: <1860d052-079e-022b-06ea-fc70000aa3a6@gmail.com> References: <487fc53f105145b890c5dfbf0a0ddeef@BCExchange.BuchananComputing.co.uk> <1860d052-079e-022b-06ea-fc70000aa3a6@gmail.com> Message-ID: <2156568.BDFclUNcYG@even-i700> > You didn't provide enough information to give specific recommendations. > However, generally there are some things you can do. There was a bit of private follow-up. It appears to be a mis-use of validation patterns, lacking a starting ^ and a trailing $ The examples at https://mapserver.org/mapfile/validation.html do it properly, but this would probably require a more prominent warning that they are necessary to get strict matching of the regexp. Even -- Spatialys - Geospatial professional services http://www.spatialys.com