[MapServer-users] Speeding up msDrawMap() for shapefiles

[Steve Lime]

Did you have a look at https://blog.cleverelephant.ca/2022/04/coshp.html?

On Mon, Dec 18, 2023 at 2:39 PM Matthew Graber via MapServer-users <
mapserver-users at lists.osgeo.org> wrote:

> Hi Jukka,
>
> Thanks for the advice. I was able to confirm through local testing that
> using "DATA" is consistently faster than using the OGR connection (was
> getting over 500 ms for OGR vs just over 300 ms for DATA). My results are
> also suggesting that performing the shapefile optimizations may make my
> request several hundredths of a second faster, I'm a little less sure on
> that one. But regardless, it appears that the bulk of the problem with the
> slow S3 connection is likely S3's overhead, so I don't think there's
> anything more I can do with MapServer to speed it up.
>
> Thanks again,
> Matt
>
> On Fri, Dec 8, 2023 at 11:58 AM Rahkonen Jukka <
> jukka.rahkonen at maanmittauslaitos.fi> wrote:
>
>> Hi,
>>
>>
>>
>> You should test the same shapefiles locally for understanding where the
>> time gets spent. Normal time with simple styles is perhaps up to 300 ms.
>> The optimization hints were written long before there were S3 or Amazon
>> (the company).
>>
>> Unfortunately I don’t know how to optimize access to S3.
>>
>>
>>
>> -Jukka Rahkonen-
>>
>>
>>
>> *Lähettäjä:* MapServer-users <mapserver-users-bounces at lists.osgeo.org> *Puolesta
>> *Matthew Graber via MapServer-users
>> *Lähetetty:* perjantai 8. joulukuuta 2023 21.47
>> *Vastaanottaja:* mapserver-users at lists.osgeo.org
>> *Aihe:* [MapServer-users] Speeding up msDrawMap() for shapefiles
>>
>>
>>
>> Hi,
>>
>>
>>
>> I'm trying to speed up the time required for msDrawMap() to execute for
>> rendering shapefiles. Currently, according to the mapserver logs, it's
>> taking 2.5-3.5 seconds, sometimes up to 5 seconds for it to execute:
>>
>>
>>
>> [Wed Dec  6 18:43:47 2023].699791 CGI Request 53 on process 2641
>> [Wed Dec  6 18:43:47 2023].717216 msDrawMap(): rendering using
>> outputformat named png (AGG/PNG).
>> [Wed Dec  6 18:43:52 2023].299580 msDrawMap(): Layer 1010
>> (Shapefile_Layer), 4.582s
>> [Wed Dec  6 18:43:52 2023].299792 msDrawMap(): Drawing Label Cache, 0.000s
>> [Wed Dec  6 18:43:52 2023].299803 msDrawMap() total time: 4.584s
>> [Wed Dec  6 18:43:52 2023].310853 msSaveImage(stdout) total time: 0.011s
>>
>> [Wed Dec  6 18:43:52 2023].310883 mapserv request processing time
>> (msLoadMap not incl.): 4.611s
>>
>>
>>
>> The mapfile is currently set up using "CONNECTIONTYPE OGR" and
>> "CONNECTION" to access the shapefiles from an S3 bucket as follows:
>>
>>
>>
>> CONNECTIONTYPE OGR
>> CONNECTION    '/vsis3/s3_bucket_path/my_shapefile%.shp'
>>
>>
>>
>> I noticed at https://mapserver.org/input/vector/shapefiles.html that it
>> says "Shapefile access is built directly into MapServer. It is also
>> available through OGR, but direct access without OGR is recommended and
>> discussed here."
>>
>> So I tried removing "CONNECTIONTYPE OGR" and changing "CONNECTION" to
>> "DATA" as follows:
>>
>>
>>
>>  DATA    '/vsis3/s3_bucket_path/my_shapefile%.shp'
>>
>>
>>
>> I haven't been able to observe any definitive speed improvements by doing
>> this yet, although I can't say for sure that there aren't any.
>>
>>
>>
>> Based on https://mapserver.org/optimization/vector.html, I also tried
>> using the shptree command to create a spatial index and the coshp command
>> to sort the shapefile and index. Even after doing that, I still see load
>> msDrawMap() times of up to 5 seconds.
>>
>>
>>
>> Does anyone know of any other optimizations that could be made here? Or
>> if not, would using shptree and coshp on the shapefile and then accessing
>> it in the mapfile using "DATA" to take it from S3 be the optimal way to set
>> things up?
>>
>>
>>
>> Thanks!
>>
>> Matt
>>
> _______________________________________________
> MapServer-users mailing list
> MapServer-users at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/mapserver-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-users/attachments/20231218/44a3d67c/attachment.htm>