[Pywps-dev] Guidelines for netCDF file and opendap accesss within pywps

[David Huard]

Hi Jachym,

Thanks for the pointers, I've started writing validators for netCDF. I'm
still wondering where the decision to download a file is made? Can I
shortcut that decision and avoid a file download if the href is a valid
opendap link, ie it passes the validatenetcdf checks?


On Fri, Jun 22, 2018 at 4:53 AM Jachym Cepicky <jachym.cepicky at gmail.com>
wrote:

> Hi,
>
> yes ComplexInput should work for you - you can pass the url with the data
> using "<Reference ... />" element.. see [1] for example
>
> Any Format can have (and has by default) `validator` function, which
> return's, whether the input data are valid or no [3]. You can also use
> `get_format` function [4] and set the validator there.
>
> Example, how validating function can look can be shapefile or gml
> validators [5]
>
> You should probably extend foramts [2] with NetCDF mimetype
>
> But, this will check the file only after it was downloaded to PyWPS - not
> the URL. Still. is that sufficient?
>
> Jachym
>
> [1]
> https://github.com/geopython/pywps/blob/master/tests/requests/wps_execute_request-responsedocument-1.xml#L24
> [2]
> https://github.com/geopython/pywps/blob/master/pywps/inout/formats/__init__.py
> [3]
> https://github.com/geopython/pywps/blob/master/pywps/inout/formats/__init__.py#L42
> [4]
> https://github.com/geopython/pywps/blob/master/pywps/inout/formats/__init__.py#L215
> [5]
> https://github.com/geopython/pywps/blob/master/pywps/validator/complexvalidator.py
>
>
>
> čt 21. 6. 2018 v 17:15 odesílatel David Huard <huard.david at ouranos.ca>
> napsal:
>
>> Hi all,
>>
>> I'd like to contribute a pull request to better handle netCDF files in
>> pywps but I don't know where to start.
>>
>> We have a number of processes taking netCDF
>> <https://www.unidata.ucar.edu/software/netcdf/> files as inputs. For
>> those less familiar with the format, netCDF is based on HDF5 and a set of
>> conventions <http://cfconventions.org/>. It is the standard data format
>> in oceanography and climatology. netCDF files are usually stored on servers
>> with support for opendap <https://www.opendap.org/>. This means that
>> users can either download the netCDF file and then open it locally, or use
>> the opendap protocol to open it remotely. What that means is that you can
>> do
>>
>> from netCDF4 import nc
>> ds1 = nc.Dataset("<path to local file>")
>> ds2 = nc.Dataset("<link to opendap address>")
>>
>> and both ds1 and ds2 will behave identically. However ds2 is not
>> downloaded locally, but rather read remotely on demand. If a file contains
>> a 3D matrix (time, lat, lon), you can read one slice of the matrix without
>> downloading it all.
>>
>> Some of our pywps.Process support both netCDF file and opendap access. We
>> define a ComplexInput for the address to an actual netCDF file, and a
>> LiteralInput for the opendap address.
>>
>> My question is whether there would be a clean way for pywps to support
>> both modes with one ComplexInput? Internally, pywps would check if the
>> address supports opendap (just check if nc.Dataset(url) works), and if not,
>> would download the file locally to the server.
>>
>> In both cases, we could do
>>
>> ds = nc.Dataset(requests.inputs['resource'][0].file)
>>
>> I'm willing to put the time to do it, I just don't know where to start.
>>
>> Thanks,
>>
>> David
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> pywps-dev mailing list
>> pywps-dev at lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/pywps-dev
>
>
>
> --
> Jachym Cepicky
> e-mail: jachym.cepicky gmail com
> URL: http://les-ejk.cz
> GPG: http://les-ejk.cz/pgp/JachymCepicky.pgp
> _______________________________________________
> pywps-dev mailing list
> pywps-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/pywps-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pywps-dev/attachments/20180626/b9613d76/attachment-0001.html>