[PyWPS-dev] wps raster processing pipeline (chaining)

[Jachym Cepicky]

Hi Idan,

first off all, thank you for sharing your nice implementation - is
that published somewhere online? Are you working for some organisation
? Would you like to share it in our gallery ?
https://pywps.org/gallery/

About your questions:

Output format should be part of the request, see WPS 1.0.0
specification, table 51 . In all our examples, we just use the
"asReference" attribute, but setting mimeType would lead the output
format definition

Chaining processes should be possible between multiple instances. But
the level of implementation will probably be different among them -
you will have to test against some servers. I'm not sure, how deep we
implemented this in PyWPS to be fair.

Anyway, chaining is again described in the WPS 1.0.0 standard -but
only for KVP (HTTP GET) encoding (page 40).

Having a look into your source code - I'm afraid, WPS client for other
services was not implemented yet. [1][2]


[1] https://github.com/geopython/pywps/blob/master/pywps/app/WPSRequest.py#L491
[2] https://github.com/geopython/pywps/blob/master/pywps/app/WPSRequest.py#L381

Jachym

čt 27. 2. 2020 v 11:38 odesílatel Idan Miara <idan at miara.com> napsal:
>
> Hi all,
>
> I'm publishing as open source our first pywps service:
> https://github.com/talos-gis/talos_wps
> https://hub.docker.com/repository/docker/talosgis/talos_wps
>
> Some inquiries about the following process:
> https://github.com/talos-gis/talos_wps/blob/master/processes/wps_crop_color.py
> It supports tif and czml outputs. Currently the user selects the output format via an input parameter.
> Is there a good way to select the desired output format for the result?
>
> Where can I find examples about how to chain this process to another one?
> Can I chain processes from different WPS services?
>
> Kind regards,
> Idan
>
> On Mon, 17 Feb 2020 at 21:09, Idan Miara <idan at miara.com> wrote:
>>
>> Hi Jachym,
>>
>> Thanks for your reply!
>> I was able to make a working first single pywps service for a different task - cropping a DEM raster to a given cutline, make a color relief out of it and then return the result as a CZML. I think it would be good to use these individually, thus I though about chaining.
>> I wasn't able to figure out how the chaining works, if not for this particular task I think it would be useful for other tasks I need to develop. I will publish it as open source very soon and ask some followup questions regarding improving it.
>>
>> Kind regards,
>> Idan
>>
>> On Mon, 17 Feb 2020, 20:54 Jachym Cepicky, <jachym.cepicky at gmail.com> wrote:
>>>
>>> Hi Idan,
>>>
>>> sorry for late response,
>>>
>>> do you really need to chain processes? Wouldn't it be enough, just to
>>> write single process, maybe consisting out of several python modules,
>>> which would apply the three steps you described?
>>>
>>> I would say, storing raster data to harddrive is not bad idea, if the
>>> harddrive is SSD, then it's usually significantly faster
>>>
>>> and saving the data on the server can be used as "cache".
>>>
>>> Anyway, I'm not completely sure, where the biggest issue of yours
>>> currently is: do you know, how to implement the tasks in python, but
>>> you are not sure, how to wrap it as an PyWPS Process? Are you
>>> completely unaware, how to implement this in Python at all? Everything
>>> is clear, you have your 3 processes, you just need to figure out, how
>>> to chain them?
>>>
>>> Jachym
>>>
>>> ne 2. 2. 2020 v 11:50 odesílatel Idan Miara <idan at miara.com> napsal:
>>> >
>>> > Hi All,
>>> >
>>> > I'm new to pywps :)
>>> > I'm trying to find a way to make an efficient raster processing pipeline.
>>> > For instance, with Geoserver wps I was able to chain wps requests to make the following chain:
>>> > 1. clipping a DEM raster from a geoserver layer.
>>> > 2. apply an SLD style
>>> > 3. publish the result on geoserver.
>>> >
>>> > I wanted to know how can I do something similar with pywps and possibly reuse gdal tools (such as gdal_translate, gdal_transform or gdaldem via their python interfaces)
>>> > Some points to consider:
>>> > 1. I couldn't find a way to chain requests on pywps.
>>> > 2. As raster data can be heavy, ideally intermediate rasters won't be saved/loaded to disk, but kept in memory and deleted once the result is ready (maybe use gdal memory raster?)
>>> > 3. I want to make use of cache in a way that multiple requests for the same input raster(s) blocks will make use of cache (maybe gdal internal cache could be good).
>>> > 4. Is there a best practice for serving the result raster(s) and possibly use them as cache in case the same request is submitted again to the service?
>>> >
>>> > Any pointers would be appreciated!
>>> > Kind regards,
>>> > Idan
>>> >
>>> > _______________________________________________
>>> > pywps-dev mailing list
>>> > pywps-dev at lists.osgeo.org
>>> > https://lists.osgeo.org/mailman/listinfo/pywps-dev
>>>
>>>
>>>
>>> --
>>> Jachym Cepicky
>>> e-mail: jachym.cepicky gmail com
>>> URL: http://les-ejk.cz
>>> GPG: http://les-ejk.cz/pgp/JachymCepicky.pgp



-- 
Jachym Cepicky
e-mail: jachym.cepicky gmail com
URL: http://les-ejk.cz
GPG: http://les-ejk.cz/pgp/JachymCepicky.pgp