[mapserver-users] mapcache seed speed optimization

[Stephen Woodbridge]

Thank Lars.

You can now fetch it from https://github.com/woodbri/imaptools.com

Enjoy,
   -Steve W

On 11/11/2019 3:07 PM, lars.schylberg at blixtmail.se wrote:
> Hi
>
> I have had a similar situation as You with large national data sets in 
> shape format. One of the biggest improvements for me has been to split 
> large shape files into smaller and
> then use vector tile indexes in Mapserver.
> The tool shp2tile has been working well for me. Stephen Woodbridge has 
> written that tool. It can be found at 
> http://imaptools.com/download-software.html
> I think that You had to compile yourself on linux, but that was easy.
>
> I have an example where I have automated this in a script that runs 
> through all shape file in a directory and splits shape files larger 
> than a certain size and then creates the vector tile index files.
> The code snippet that is part of larger import script can be found here:
> https://gist.github.com/LarsSchy/644b37ab2b1e58b48808bfb2b57d0359
>
> The script snippet also shows the normal optimizations we do.
>
> Hope this could give You some ideas how optimize and make Mapserver 
> and Mapcache run faster. I also ran into the problem with to many inodes.
> That got solved by using sqlite as backend in Mapcache.
>
> Best regards / Lars Schylberg
>
>
>
> 11 november 2019 kl. 15:42, "Travis Kirstine" 
> <traviskirstine at gmail.com 
> <mailto:traviskirstine at gmail.com?to=%22Travis%20Kirstine%22%20<traviskirstine at gmail.com>>> 
> skrev:
>
>     I may not be unrealistic for it take mapserver to 4-6 seconds to
>     generate 4096px image depending on the amount data being rendered
>     and the complexity of your styles / expressions / labels. If your
>     mapserver is able to generate a 1024x1024 in under a second that
>     is pretty good. Like Jukka and other suggest I would look at
>     translating the files to shp files with spatial indexes or postgis
>     tables with spatial and attribute indexes if necessary.
>     Preprocessing the data (simplification / prefiltering / point
>     thinning cluster) and using SCALETOKENs to reference different
>     source layers at different scales can lead to big performance
>     boosts at the mapserver end.
>     For us the biggest limiting factor has been the write blocking to
>     sqlite cache when seeding tiles, if you increase the number of
>     threads / process in mapcache seed those process will just end up
>     waiting to write to the cache. You can generally figure out what
>     the max number of processes the cache can handle by running a seed
>     on a test area and look at the number of tiles seeded per second,
>     as some point -n will have no effect which your process are just
>     waiting to write the cache. I would look at using this method to
>     test sqlite vs disk or other backends. The GeoTiff cache may be
>     promising as it appears to be non-blocking but experimental...
>     We use a riak cache with a riak leveldb backend which works well
>     for us but is a bit of a pain to manage as it difficult to delete
>     objects so it's not great for caches that need to be refreshed.
>     Finally you may want to test MapProxy as an alternative to
>     MapCache (we use both). MapProxy supports a non-blocking compact
>     cache that could solve your inode problem
>     Regards
>     On Fri, 8 Nov 2019 at 11:48, Sebastiano Laini
>     <Sebastiano.Laini at buchanancomputing.co.uk
>     <mailto:Sebastiano.Laini at buchanancomputing.co.uk>> wrote:
>
>         We don’t create the data, we rely on OS (ordnance survey) to
>         supply us the maps files and then we publish them in our
>         service but I assume that probably we will need to create some
>         flow to improve it or download it in other format that is faster.
>
>         Sebastiano Laini
>
>         Web Developer
>
>         Buchanan Computing
>
>         *From:*Fawcett, David (MNIT) [mailto:david.fawcett at state.mn.us
>         <mailto:david.fawcett at state.mn.us>]
>         *Sent:* 08 November 2019 16:41
>         *To:* Sebastiano Laini
>         <Sebastiano.Laini at BuchananComputing.co.uk
>         <mailto:Sebastiano.Laini at BuchananComputing.co.uk>>; 'Rahkonen
>         Jukka (MML)' <jukka.rahkonen at maanmittauslaitos.fi
>         <mailto:jukka.rahkonen at maanmittauslaitos.fi>>;
>         'mapserver-users at lists.osgeo.org
>         <mailto:mapserver-users at lists.osgeo.org>'
>         <mapserver-users at lists.osgeo.org
>         <mailto:mapserver-users at lists.osgeo.org>>
>         *Subject:* RE: mapcache seed speed optimization
>
>         For data formats that are slower to read, would it add fit
>         your workflow to convert it to different data format before
>         creating the tiles?
>
>         _______________________________________________
>         mapserver-users mailing list
>         mapserver-users at lists.osgeo.org
>         <mailto:mapserver-users at lists.osgeo.org>
>         https://lists.osgeo.org/mailman/listinfo/mapserver-users 
>
>
>
>
> _______________________________________________
> mapserver-users mailing list
> mapserver-users at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/mapserver-users