[Pywps-dev] Guidelines for netCDF file and opendap accesss within pywps

[David Huard]

Thanks !

I'll look at it and come back with a PR.

On Tue, Jun 26, 2018 at 9:16 AM Jachym Cepicky <jachym.cepicky at gmail.com>
wrote:

> I belive,
>
> here is the place, where data get downloaded
> https://github.com/geopython/pywps/blob/master/pywps/app/Service.py#L191
>
> út 26. 6. 2018 v 14:57 odesílatel David Huard <huard.david at ouranos.ca>
> napsal:
>
>> Hi Jachym,
>>
>> Thanks for the pointers, I've started writing validators for netCDF. I'm
>> still wondering where the decision to download a file is made? Can I
>> shortcut that decision and avoid a file download if the href is a valid
>> opendap link, ie it passes the validatenetcdf checks?
>>
>>
>> On Fri, Jun 22, 2018 at 4:53 AM Jachym Cepicky <jachym.cepicky at gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> yes ComplexInput should work for you - you can pass the url with the
>>> data using "<Reference ... />" element.. see [1] for example
>>>
>>> Any Format can have (and has by default) `validator` function, which
>>> return's, whether the input data are valid or no [3]. You can also use
>>> `get_format` function [4] and set the validator there.
>>>
>>> Example, how validating function can look can be shapefile or gml
>>> validators [5]
>>>
>>> You should probably extend foramts [2] with NetCDF mimetype
>>>
>>> But, this will check the file only after it was downloaded to PyWPS -
>>> not the URL. Still. is that sufficient?
>>>
>>> Jachym
>>>
>>> [1]
>>> https://github.com/geopython/pywps/blob/master/tests/requests/wps_execute_request-responsedocument-1.xml#L24
>>> [2]
>>> https://github.com/geopython/pywps/blob/master/pywps/inout/formats/__init__.py
>>> [3]
>>> https://github.com/geopython/pywps/blob/master/pywps/inout/formats/__init__.py#L42
>>> [4]
>>> https://github.com/geopython/pywps/blob/master/pywps/inout/formats/__init__.py#L215
>>> [5]
>>> https://github.com/geopython/pywps/blob/master/pywps/validator/complexvalidator.py
>>>
>>>
>>>
>>> čt 21. 6. 2018 v 17:15 odesílatel David Huard <huard.david at ouranos.ca>
>>> napsal:
>>>
>>>> Hi all,
>>>>
>>>> I'd like to contribute a pull request to better handle netCDF files in
>>>> pywps but I don't know where to start.
>>>>
>>>> We have a number of processes taking netCDF
>>>> <https://www.unidata.ucar.edu/software/netcdf/> files as inputs. For
>>>> those less familiar with the format, netCDF is based on HDF5 and a set of
>>>> conventions <http://cfconventions.org/>. It is the standard data
>>>> format in oceanography and climatology. netCDF files are usually stored on
>>>> servers with support for opendap <https://www.opendap.org/>. This
>>>> means that users can either download the netCDF file and then open it
>>>> locally, or use the opendap protocol to open it remotely. What that means
>>>> is that you can do
>>>>
>>>> from netCDF4 import nc
>>>> ds1 = nc.Dataset("<path to local file>")
>>>> ds2 = nc.Dataset("<link to opendap address>")
>>>>
>>>> and both ds1 and ds2 will behave identically. However ds2 is not
>>>> downloaded locally, but rather read remotely on demand. If a file contains
>>>> a 3D matrix (time, lat, lon), you can read one slice of the matrix without
>>>> downloading it all.
>>>>
>>>> Some of our pywps.Process support both netCDF file and opendap access.
>>>> We define a ComplexInput for the address to an actual netCDF file, and a
>>>> LiteralInput for the opendap address.
>>>>
>>>> My question is whether there would be a clean way for pywps to support
>>>> both modes with one ComplexInput? Internally, pywps would check if the
>>>> address supports opendap (just check if nc.Dataset(url) works), and if not,
>>>> would download the file locally to the server.
>>>>
>>>> In both cases, we could do
>>>>
>>>> ds = nc.Dataset(requests.inputs['resource'][0].file)
>>>>
>>>> I'm willing to put the time to do it, I just don't know where to start.
>>>>
>>>> Thanks,
>>>>
>>>> David
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> pywps-dev mailing list
>>>> pywps-dev at lists.osgeo.org
>>>> https://lists.osgeo.org/mailman/listinfo/pywps-dev
>>>
>>>
>>>
>>> --
>>> Jachym Cepicky
>>> e-mail: jachym.cepicky gmail com
>>> URL: http://les-ejk.cz
>>> GPG: http://les-ejk.cz/pgp/JachymCepicky.pgp
>>> _______________________________________________
>>> pywps-dev mailing list
>>> pywps-dev at lists.osgeo.org
>>> https://lists.osgeo.org/mailman/listinfo/pywps-dev
>>
>>
>
> --
> Jachym Cepicky
> e-mail: jachym.cepicky gmail com
> URL: http://les-ejk.cz
> GPG: http://les-ejk.cz/pgp/JachymCepicky.pgp
> _______________________________________________
> pywps-dev mailing list
> pywps-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/pywps-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pywps-dev/attachments/20180626/cbc7da14/attachment-0001.html>