[MapServer-users] Speed of WMS vs Speed of WCS; How to speed up WCS service?

Thu Jul 27 14:20:38 PDT 2023

Hi,

Mapserver is not clever at all when it finds many overlapping images. It renders them all one by one. If the list of images is sorted correctly then the end result is what was desired, but with all too much work. See https://github.com/MapServer/MapServer/issues/5899.
We use two tricks:
1) We have computed the time range of validity for each image and we save the start data and end date into two attributes in the tileindex table in a database. Start date is the acquisition date of the image, and end date is the date when the last pixel of the image gets covered by some newer image. If the requested time is not within the range of validity then the image is skipped. Skipping is based on the SQL query that is used as DATA for the tileindex layer.
2) Because time is not supported as a dimension we have written a small utility that is manipulating the incoming http GetCoverage requests before they are sent to Mapserver. The utility converts the time subset, for example SUBSET=time("2012-12-31T02:00:00Z") into something like "&start_date<2012-12-31T02:00:00Z&end_date>2012-12-31T02:00:00Z. The start_date and end_date variables are defined and validated in the mapfile.

As a result Mapserver usually does not get overlapping images from the tileindex so it does not need to spend time for multiple rendering either. We have typically about 10 aerial photos from different times at each place and without the above mentioned tweaks the service was all too slow.

-Jukka Rahkonen-

Lähettäjä: MapServer-users <mapserver-users-bounces at lists.osgeo.org> Puolesta Marcin Niemyjski via MapServer-users
Lähetetty: torstai 27. heinäkuuta 2023 12.26
Vastaanottaja: MapServer Users <mapserver-users at lists.osgeo.org>
Aihe: Re: [MapServer-users] Speed of WMS vs Speed of WCS; How to speed up WCS service?

Hello Jukka,

Yup I guess that's the case, I assumed that WCS has the same capabilities as WMS when it comes to handling time dimension and it would use the default time from mapfile while requesting the data.
This is my mapserv log file from one getCoverage request (very small, that's why It took only 30 sec to process it). It would be great if you check it 🙂:
https://s3.waw3-1.cloudferro.com/swift/v1/Julien/log2.txt

as you can see mapserver lists all of tile item objects (or at least I guess it does haha) which intersect with requested BBOX. So, this is the problem, that's why it takes so long to generate file. I've got to pass somehow time parameter to request, or at least to query(?) to reduce number of returning shapes.

You mentioned your own supporting time WCS service, is it open source? If not, can you give me advice on how to start writing my own? Just some basics ^^

Thanks for your complex answer it helped a lot,
Marcin

________________________________
From: Rahkonen Jukka <jukka.rahkonen at maanmittauslaitos.fi<mailto:jukka.rahkonen at maanmittauslaitos.fi>>
Sent: Thursday, July 27, 2023 12:05 AM
To: Marcin Niemyjski <mniemyjski at cloudferro.com<mailto:mniemyjski at cloudferro.com>>; MapServer Users <mapserver-users at lists.osgeo.org<mailto:mapserver-users at lists.osgeo.org>>
Subject: Re: [MapServer-users] Speed of WMS vs Speed of WCS; How to speed up WCS service?

Hi,

Does the tileindex query ” DATA 'geometry from (select * from plua order by "maxCC" desc) “ return several images from the requested BBOX?

I know that MapServer WCS can be fast and we have also a demo at https://beta-karttakuva.maanmittauslaitos.fi/demo/ortokuvat_ja_korkeusmallit/#15.05/67.01553/27.16873

but unfortunately the user interface is only in Finnish. The source images (maybe 10 TB or so) are GeoTIFFs and they are not in the cloud. The time for downloading a small GetCoverage response as GeoTIFF seems to take around 400 milliseconds. We had to write a small service that makes our WCS service to support time as a dimension because MapServer does not support it with WCS https://www.mail-archive.com/mapserver-users@lists.osgeo.org/msg26265.html.

-Jukka Rahkonen-

Lähettäjä: MapServer-users <mapserver-users-bounces at lists.osgeo.org<mailto:mapserver-users-bounces at lists.osgeo.org>> Puolesta Marcin Niemyjski via MapServer-users
Lähetetty: keskiviikko 26. heinäkuuta 2023 15.06
Vastaanottaja: MapServer Users <mapserver-users at lists.osgeo.org<mailto:mapserver-users at lists.osgeo.org>>
Aihe: Re: [MapServer-users] Speed of WMS vs Speed of WCS; How to speed up WCS service?

Hi Seth,

as you suggested, I've added the Extent parameter to all of my layers, and it resulted in decreasing time to 5m33.534s. But still, it's quite a lot of time for generating 1.5 MiB file.

Any other suggestions?
Maybe there is something wrong with projection so beyond just serving data mapserver have to reproject it multiple times? (more than one time by using tilesrs option)?

Thanks,

Marcin

________________________________

From: Seth G <sethg at geographika.co.uk<mailto:sethg at geographika.co.uk>>
Sent: Wednesday, July 26, 2023 12:58 PM
To: Marcin Niemyjski <mniemyjski at cloudferro.com<mailto:mniemyjski at cloudferro.com>>; MapServer Users <mapserver-users at lists.osgeo.org<mailto:mapserver-users at lists.osgeo.org>>
Subject: Re: [MapServer-users] Speed of WMS vs Speed of WCS; How to speed up WCS service?

Hi Marcin,

As a first test I'd recommend adding in EXTENTs to all your layers (and ensuring they are correct):

EXTENT 1334552.857598 5342648.587634 4715044.821842 7547620.486088

If these are missing MapServer will often try and calculate these from the datasource automatically. Depending on the request type this calculation could be made multiple times.

Seth

--

web:https://geographika.net<https://geographika.net/> & https://mapserverstudio.net<https://mapserverstudio.net/>

twitter: @geographika

On Wed, Jul 26, 2023, at 11:53 AM, Marcin Niemyjski via MapServer-users wrote:

Hello,

I have a problem, Namely, I have created a WCS and here is its mapfile:

https://pastecode.io/s/6v3qfssb

For a better understanding, here is a view of the tileindex through which the data is served:

https://pasteboard.co/ydgvDiqxz4lf.png

I am connecting to files located in a private bucket, so I am using vsis3.

The WMS loads quite fast, but the issue arises with the time it takes to generate a requested WCS tile:

 https://pasteboard.co/8G8uF41jqgHc.png

It takes almost 5 minutes to generate the tile, and the received data is 1.5 MiB (which is it's actual size).

Additionally, I am sending you URLs to sample data:

https://s3.waw3-1.cloudferro.com/swift/v1/Julien/Data/T34UFD_20230708T093549_TCI_60m.jp2

https://s3.waw3-1.cloudferro.com/swift/v1/Julien/Data/T37TCN_20220605T083559_TCI_60m.jp2

The case is - how to speed it up? Am I making any goofy mistakes?

I would be grateful for any recommendations or if you notice any errors, I might be making :)

Best,

Marcin

_______________________________________________

MapServer-users mailing list

MapServer-users at lists.osgeo.org<mailto:MapServer-users at lists.osgeo.org>

https://lists.osgeo.org/mailman/listinfo/mapserver-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-users/attachments/20230727/cbaadffc/attachment-0001.htm>