[PyWPS-dev] wps raster processing pipeline (chaining)

[Idan Miara]

Hi all,

I'm publishing as open source our first pywps service:
https://github.com/talos-gis/talos_wps
https://hub.docker.com/repository/docker/talosgis/talos_wps

Some inquiries about the following process:
https://github.com/talos-gis/talos_wps/blob/master/processes/wps_crop_color.py

It supports tif and czml outputs. Currently the user selects the output
format via an input parameter.
Is there a good way to select the desired output format for the result?

Where can I find examples about how to chain this process to another one?
Can I chain processes from different WPS services?

Kind regards,
Idan

On Mon, 17 Feb 2020 at 21:09, Idan Miara <idan at miara.com> wrote:

> Hi Jachym,
>
> Thanks for your reply!
> I was able to make a working first single pywps service for a different
> task - cropping a DEM raster to a given cutline, make a color relief out of
> it and then return the result as a CZML. I think it would be good to use
> these individually, thus I though about chaining.
> I wasn't able to figure out how the chaining works, if not for this
> particular task I think it would be useful for other tasks I need to
> develop. I will publish it as open source very soon and ask some followup
> questions regarding improving it.
>
> Kind regards,
> Idan
>
> On Mon, 17 Feb 2020, 20:54 Jachym Cepicky, <jachym.cepicky at gmail.com>
> wrote:
>
>> Hi Idan,
>>
>> sorry for late response,
>>
>> do you really need to chain processes? Wouldn't it be enough, just to
>> write single process, maybe consisting out of several python modules,
>> which would apply the three steps you described?
>>
>> I would say, storing raster data to harddrive is not bad idea, if the
>> harddrive is SSD, then it's usually significantly faster
>>
>> and saving the data on the server can be used as "cache".
>>
>> Anyway, I'm not completely sure, where the biggest issue of yours
>> currently is: do you know, how to implement the tasks in python, but
>> you are not sure, how to wrap it as an PyWPS Process? Are you
>> completely unaware, how to implement this in Python at all? Everything
>> is clear, you have your 3 processes, you just need to figure out, how
>> to chain them?
>>
>> Jachym
>>
>> ne 2. 2. 2020 v 11:50 odesílatel Idan Miara <idan at miara.com> napsal:
>> >
>> > Hi All,
>> >
>> > I'm new to pywps :)
>> > I'm trying to find a way to make an efficient raster processing
>> pipeline.
>> > For instance, with Geoserver wps I was able to chain wps requests to
>> make the following chain:
>> > 1. clipping a DEM raster from a geoserver layer.
>> > 2. apply an SLD style
>> > 3. publish the result on geoserver.
>> >
>> > I wanted to know how can I do something similar with pywps and possibly
>> reuse gdal tools (such as gdal_translate, gdal_transform or gdaldem via
>> their python interfaces)
>> > Some points to consider:
>> > 1. I couldn't find a way to chain requests on pywps.
>> > 2. As raster data can be heavy, ideally intermediate rasters won't be
>> saved/loaded to disk, but kept in memory and deleted once the result is
>> ready (maybe use gdal memory raster?)
>> > 3. I want to make use of cache in a way that multiple requests for the
>> same input raster(s) blocks will make use of cache (maybe gdal internal
>> cache could be good).
>> > 4. Is there a best practice for serving the result raster(s) and
>> possibly use them as cache in case the same request is submitted again to
>> the service?
>> >
>> > Any pointers would be appreciated!
>> > Kind regards,
>> > Idan
>> >
>> > _______________________________________________
>> > pywps-dev mailing list
>> > pywps-dev at lists.osgeo.org
>> > https://lists.osgeo.org/mailman/listinfo/pywps-dev
>>
>>
>>
>> --
>> Jachym Cepicky
>> e-mail: jachym.cepicky gmail com
>> URL: http://les-ejk.cz
>> GPG: http://les-ejk.cz/pgp/JachymCepicky.pgp
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pywps-dev/attachments/20200227/30cf241c/attachment.html>