[mapserver-users] mapcache seed speed optimization

[Stephen Woodbridge]

FYI, imaptools.com is not online any more (except the we are moving pages).

Lars if you email me the files, I'll push them to my github repository 
so others can find them.

Thanks,
   -Steve W

On 11/11/2019 3:07 PM, lars.schylberg at blixtmail.se wrote:
> Hi
>
> I have had a similar situation as You with large national data sets in 
> shape format. One of the biggest improvements for me has been to split 
> large shape files into smaller and
> then use vector tile indexes in Mapserver.
> The tool shp2tile has been working well for me. Stephen Woodbridge has 
> written that tool. It can be found at 
> http://imaptools.com/download-software.html
> I think that You had to compile yourself on linux, but that was easy.
>
> I have an example where I have automated this in a script that runs 
> through all shape file in a directory and splits shape files larger 
> than a certain size and then creates the vector tile index files.
> The code snippet that is part of larger import script can be found here:
> https://gist.github.com/LarsSchy/644b37ab2b1e58b48808bfb2b57d0359
>
> The script snippet also shows the normal optimizations we do.
>
> Hope this could give You some ideas how optimize and make Mapserver 
> and Mapcache run faster. I also ran into the problem with to many inodes.
> That got solved by using sqlite as backend in Mapcache.
>
> Best regards / Lars Schylberg
>
>
>
> 11 november 2019 kl. 15:42, "Travis Kirstine" 
> <traviskirstine at gmail.com 
> <mailto:traviskirstine at gmail.com?to=%22Travis%20Kirstine%22%20<traviskirstine at gmail.com>>> 
> skrev:
>
>     I may not be unrealistic for it take mapserver to 4-6 seconds to
>     generate 4096px image depending on the amount data being rendered
>     and the complexity of your styles / expressions / labels. If your
>     mapserver is able to generate a 1024x1024 in under a second that
>     is pretty good. Like Jukka and other suggest I would look at
>     translating the files to shp files with spatial indexes or postgis
>     tables with spatial and attribute indexes if necessary.
>     Preprocessing the data (simplification / prefiltering / point
>     thinning cluster) and using SCALETOKENs to reference different
>     source layers at different scales can lead to big performance
>     boosts at the mapserver end.
>     For us the biggest limiting factor has been the write blocking to
>     sqlite cache when seeding tiles, if you increase the number of
>     threads / process in mapcache seed those process will just end up
>     waiting to write to the cache. You can generally figure out what
>     the max number of processes the cache can handle by running a seed
>     on a test area and look at the number of tiles seeded per second,
>     as some point -n will have no effect which your process are just
>     waiting to write the cache. I would look at using this method to
>     test sqlite vs disk or other backends. The GeoTiff cache may be
>     promising as it appears to be non-blocking but experimental...
>     We use a riak cache with a riak leveldb backend which works well
>     for us but is a bit of a pain to manage as it difficult to delete
>     objects so it's not great for caches that need to be refreshed.
>     Finally you may want to test MapProxy as an alternative to
>     MapCache (we use both). MapProxy supports a non-blocking compact
>     cache that could solve your inode problem
>     Regards
>     On Fri, 8 Nov 2019 at 11:48, Sebastiano Laini
>     <Sebastiano.Laini at buchanancomputing.co.uk
>     <mailto:Sebastiano.Laini at buchanancomputing.co.uk>> wrote:
>
>         We don’t create the data, we rely on OS (ordnance survey) to
>         supply us the maps files and then we publish them in our
>         service but I assume that probably we will need to create some
>         flow to improve it or download it in other format that is faster.
>
>         Sebastiano Laini
>
>         Web Developer
>
>         Buchanan Computing
>
>         *From:*Fawcett, David (MNIT) [mailto:david.fawcett at state.mn.us
>         <mailto:david.fawcett at state.mn.us>]
>         *Sent:* 08 November 2019 16:41
>         *To:* Sebastiano Laini
>         <Sebastiano.Laini at BuchananComputing.co.uk
>         <mailto:Sebastiano.Laini at BuchananComputing.co.uk>>; 'Rahkonen
>         Jukka (MML)' <jukka.rahkonen at maanmittauslaitos.fi
>         <mailto:jukka.rahkonen at maanmittauslaitos.fi>>;
>         'mapserver-users at lists.osgeo.org
>         <mailto:mapserver-users at lists.osgeo.org>'
>         <mapserver-users at lists.osgeo.org
>         <mailto:mapserver-users at lists.osgeo.org>>
>         *Subject:* RE: mapcache seed speed optimization
>
>         For data formats that are slower to read, would it add fit
>         your workflow to convert it to different data format before
>         creating the tiles?
>
>         _______________________________________________
>         mapserver-users mailing list
>         mapserver-users at lists.osgeo.org
>         <mailto:mapserver-users at lists.osgeo.org>
>         https://lists.osgeo.org/mailman/listinfo/mapserver-users 
>
>
>
>
> _______________________________________________
> mapserver-users mailing list
> mapserver-users at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/mapserver-users