[mapserver-users] mapcache seed speed optimization

[Jeff McKenna]

For Windows users I have been distributing shp2tile with MS4W 
(https://ms4w.com/), and in fact the last MS4W contains a rebuilt 
shp2tile (and an updated Shapelib), which as Lars said is still very useful.

-jeff



On 2019-11-11 4:07 PM, lars.schylberg at blixtmail.se wrote:
> Hi
> 
> I have had a similar situation as You with large national data sets in 
> shape format. One of the biggest improvements for me has been to split 
> large shape files into smaller and
> then use vector tile indexes in Mapserver.
> The tool shp2tile has been working well for me. Stephen Woodbridge has 
> written that tool. It can be found at 
> http://imaptools.com/download-software.html 
> <http://imaptools.com/download-software.html>
> I think that You had to compile yourself on linux, but that was easy.
> 
> I have an example where I have automated this in a script that runs 
> through all shape file in a directory and splits shape files larger than 
> a certain size and then creates the vector tile index files.
> The code snippet that is part of larger import script can be found here:
> https://gist.github.com/LarsSchy/644b37ab2b1e58b48808bfb2b57d0359
> 
> The script snippet also shows the normal optimizations we do.
> 
> Hope this could give You some ideas how optimize and make Mapserver and 
> Mapcache run faster. I also ran into the problem with to many inodes.
> That got solved by using sqlite as backend in Mapcache.
> 
> Best regards / Lars Schylberg
> 
> 
> 
> 11 november 2019 kl. 15:42, "Travis Kirstine" <traviskirstine at gmail.com 
> <mailto:traviskirstine at gmail.com?to=%22Travis%20Kirstine%22%20<traviskirstine at gmail.com>>> 
> skrev:
> 
>     I may not be unrealistic for it take mapserver to 4-6 seconds to
>     generate 4096px image depending on the amount data being rendered
>     and the complexity of your styles / expressions / labels. If your
>     mapserver is able to generate a 1024x1024 in under a second that is
>     pretty good. Like Jukka and other suggest I would look at
>     translating the files to shp files with spatial indexes or postgis
>     tables with spatial and attribute indexes if necessary.
>     Preprocessing the data (simplification / prefiltering / point
>     thinning cluster) and using SCALETOKENs to reference different
>     source layers at different scales can lead to big performance boosts
>     at the mapserver end.
>     For us the biggest limiting factor has been the write blocking to
>     sqlite cache when seeding tiles, if you increase the number of
>     threads / process in mapcache seed those process will just end up
>     waiting to write to the cache. You can generally figure out what the
>     max number of processes the cache can handle by running a seed on a
>     test area and look at the number of tiles seeded per second, as some
>     point -n will have no effect which your process are just waiting to
>     write the cache. I would look at using this method to test sqlite vs
>     disk or other backends. The GeoTiff cache may be promising as it
>     appears to be non-blocking but experimental...
>     We use a riak cache with a riak leveldb backend which works well for
>     us but is a bit of a pain to manage as it difficult to delete
>     objects so it's not great for caches that need to be refreshed.
>     Finally you may want to test MapProxy as an alternative to MapCache
>     (we use both). MapProxy supports a non-blocking compact cache that
>     could solve your inode problem
>     Regards
>     On Fri, 8 Nov 2019 at 11:48, Sebastiano Laini
>     <Sebastiano.Laini at buchanancomputing.co.uk
>     <mailto:Sebastiano.Laini at buchanancomputing.co.uk>> wrote:
> 
>         We don’t create the data, we rely on OS (ordnance survey) to
>         supply us the maps files and then we publish them in our service
>         but I assume that probably we will need to create some flow to
>         improve it or download it in other format that is faster.
> 
>         Sebastiano Laini
> 
>         Web Developer
> 
>         Buchanan Computing
> 
>         *From:*Fawcett, David (MNIT) [mailto:david.fawcett at state.mn.us
>         <mailto:david.fawcett at state.mn.us>]
>         *Sent:* 08 November 2019 16:41
>         *To:* Sebastiano Laini <Sebastiano.Laini at BuchananComputing.co.uk
>         <mailto:Sebastiano.Laini at BuchananComputing.co.uk>>; 'Rahkonen
>         Jukka (MML)' <jukka.rahkonen at maanmittauslaitos.fi
>         <mailto:jukka.rahkonen at maanmittauslaitos.fi>>;
>         'mapserver-users at lists.osgeo.org
>         <mailto:mapserver-users at lists.osgeo.org>'
>         <mapserver-users at lists.osgeo.org
>         <mailto:mapserver-users at lists.osgeo.org>>
>         *Subject:* RE: mapcache seed speed optimization
> 
>         For data formats that are slower to read, would it add fit your
>         workflow to convert it to different data format before creating
>         the tiles?
> 
>