[Pywps-dev] [ GSOC 2014 ] Database output storage idea.

Jonas Eberle jonas.eberle at gmx.de
Wed Mar 12 15:50:35 PDT 2014 

Jachym, submitting ourselves as a mentor is needed if we want that people work on the provided GSoC ideas. See following link: 
http://wiki.osgeo.org/wiki/Google_Summer_of_Code_2014_Administrative#How_to_register_as_a_mentor 

Furthermore, Anna Ghisla posted a mail in the OSGeo Summer of Code mailinglist: 
http://lists.osgeo.org/pipermail/soc/2014-March/002508.html 

Jonas 

----- Ursprüngliche Mail -----

> Ok, go ahead.

> Is there anything you need from us right now? Shall we submit ourselves as
> mentors to the task as well? Just say, what you need, we will try to help

> Jachym

> P.S. Sorry, this is first GSoC for PyWPS people, the things are still new to
> us.

> On Wed, Mar 12, 2014 at 10:06:07PM +0530, Naveen Panwar wrote:
> > Respected Sir,
> >
> > Here is the summary of our last conversations
> >
> > 1. We have to enable users to configure their own database connectivity,
> > by providing appropriate information in the config file.
> > 2. They will be using PostgreSQL, and SQLite according to their need.
> > 3. In later phase of project we can provide support for NoSql(pycass).
> > 4. Schema for the data we have to generate on the fly based on user
> > demand.
> > Also Google has started application for the Students and Mentors for
> > GSOC-2014. I think, we should put a draft of our proposal on melange, So
> > that other people/mentors in OSGeo can get to know about idea.
> >
> > On Tue, Mar 11, 2014 at 1:01 PM, Jachym Cepicky
> > <[1]jachym.cepicky at gmail.com> wrote:
> >
> > OK
> >
> > On Tue, Mar 11, 2014 at 08:18:19AM +0100, Jorge de Jesus wrote:
> > > Hi Jachym lets discuss the topic on the hangout then I can take some
> > > notes and prepare things for Naveen
> > >
> > > I'm not so certain how the schema in DB will be done, I was thinking a
> > > sort of pre-defined schema/relations like the one in the istSOS
> > > ([2]http://istgeo.ist.supsi.ch/software/istsos/)
> > >
> > > Cheers
> > >
> > >
> > >
> > > On 03/11/2014 12:31 AM, Jachym Cepicky wrote:
> > > > Hi,
> > > >
> > > >
> > > > On Tue, Mar 11, 2014 at 01:02:08AM +0530, Naveen Panwar wrote:
> > > >> Respected Sir,
> > > >>
> > > >> As you suggested in previous mail I have gone through Pywps
> > [1]Cookbook
> > > >> Documentation[1]. As there are three types of inputs and
> > outputs are
> > > >> defined in the OGC standard: LiteralData, ComplexData,
> > BoundingBox data.
> > > >> But as you mentioned we have to take care of ComplexData.
> > > >
> > > > Not true: also literal and bbox data can be stored in database (but
> > they are
> > > > much simplier task to be done thought).
> > > >
> > > >>
> > > >> Currently ComplexData data store in two manner, One with xml as
> > GML
> > > >> format, Second with binary raster data encoded in base64.
> > > >> these data stored as a file in system. Now we need database for
> > the better
> > > >> access and availability.
> > > >>
> > > >> I following questions in my mind.
> > > >>
> > > >> 1. Do we have write database module so that user can configure
> > there own
> > > >> database server by providing db URL and user credential, like
> > standard db
> > > >> connectivity ?
> > > >
> > > > This case
> > > >
> > > >>
> > > >> 2. Or we have to design our own database and provide db
> > connection object
> > > >> to user, so that they can write to the DB ?
> > > >
> > > > User should be able to setup everything using configuration file.
> > > >
> > > >>
> > > >> 3. How are we supposed to store the data in DB as both vector
> > and raster
> > > >> data present in different format ?
> > > >
> > > > PostGIS should do this. SQLite has support for some raster formats
> > as well
> > > >
> > > >> 4. What kind of database support do we need ? Relational
> > DB(PostgreSQL,
> > > >> MySQL) or NOSql DB (as we have Cassandra by pycassa ) ?
> > > >
> > > > PostgreSQL/PostGIS, SQLite for the first steps. NOSql you nemntioned
> > at the
> > > > second.
> > > >
> > > >>
> > > >> 5. What should be the schema of the database ? is it predefine
> > or define
> > > >> by the user ?
> > > >
> > > > User should define, which schema schould be used.
> > > >
> > > >>
> > > >> and last but not the least when will be next hangout 13 march or
> > 26 march
> > > >> ? sorry bit confuse.
> > > >
> > > > sorry for the mess
> > > >
> > > >
> > [3]http://www.timeanddate.com/worldclock/fixedtime.html?iso=20140313T20&ah=1
> > > >
> > > > we are still aranging, how to define timeslots
> > > >
> > > > J
> > > >
> > > >> [1] -
> > > >>
> > [2][4]http://netmar.nersc.no/sites/netmar.nersc.no/files/D7.7_WPS_Cookbook_r1_20111229.pdf
> > > >>
> > > >> On Mon, Mar 10, 2014 at 2:33 AM, Jachym Cepicky
> > > >> <[3][5]jachym.cepicky at gmail.com> wrote:
> > > >>
> > > >> Hi,
> > > >>
> > > >> On Sun, Mar 09, 2014 at 05:46:09PM +0000, Mendes de Jesus,
> > Jorge wrote:
> > > >> > Hi to all
> > > >> >
> > > >> > @Naveen, first give a look at the WPS standard 1.0.0[1] and
> > we have a
> > > >> wiki [2]and doc online [3] (all document refer to the old 3.2
> > version
> > > >> but still a good source of information and technical know-how
> > > >> > @Jachym, correct me when wrong
> > > >> >
> > > >> > WPS defines one output as ComplexData, meaning a raster
> > images,
> > > >> vetorial data etc etc can be contained between the element
> > tags
> > > >> <ComplexData /> The output data can be stored in the server
> > when the
> > > >> user requests an async result, the data stored is then fetch
> > by an URL
> > > >> location indicates by a status response.
> > > >> >
> > > >> > So,
> > > >> > 1) The data stored in the server is currently saved on the
> > filesystem
> > > >> and we need it in a DB
> > > >>
> > > >> better say: we would like to offer option, that all generated
> > output
> > > >> data could
> > > >> be stored in some database, instead of file system
> > > >> > 2) URLs will have to point to some
> > function/service/DBrequest that
> > > >> will fetch the results to the user
> > > >>
> > > >> so service will have to be written/configured (using
> > > >> mapserver/geoserver/custom
> > > >> something), which will do the serving
> > > >> > 3) DB structure should contain metadata on the request,
> > datatype etc
> > > >> etc etc
> > > >>
> > > >> > 4) We are currently working on new code and things are not
> > 100%
> > > >> stable, nor I have a clear knowledge of the code therefore I
> > can't give
> > > >> you a direct answer to were in the code this has to be
> > hookedup (Maybe
> > > >> during the next hangout 13-March I/we will check it
> > out).Please give a
> > > >> look at the pywps4.0 documentation [4]
> > > >>
> > > >> currently, first version of file storage is implemented - just
> > storing,
> > > >> no data
> > > >> exporting
> > > >>
> > [4][6]https://github.com/jachym/pywps-4/blob/master/pywps/inout.py#L227
> > > >> (will be moved to separate file soon, or maybe package)
> > > >> J
> > > >>
> > > >> >
> > > >> >
> > > >> > [1] [5][7]http://www.opengeospatial.org/standards/wps
> > > >> > [2] [6][8]http://wiki.rsg.pml.ac.uk/pywps/Main_Page
> > > >> > [3] [7][9]http://pywps.wald.intevation.org/documentation/
> > > >> > [4] [8][10]http://pywps.readthedocs.or
> > > >> >
> > > >> > I will try to give you more information on the following
> > days
> > > >> >
> > > >> > Jorge
> > > >> >
> > > >> >
> > > >> >
> > > >> > ________________________________________
> > > >> > From: Naveen Panwar [[9][11]panwarn09 at gmail.com]
> > > >> > Sent: 07 March 2014 10:00
> > > >> > To: Mendes de Jesus, Jorge; Jáchym Čepický
> > > >> > Subject: [ GSOC 2014 ] Database output storage idea.
> > > >> >
> > > >> > Hello Sir,
> > > >> >
> > > >> > As we know students application starts on March 10th.
> > > >> > For that I have to write proposal for the idea "Database
> > output
> > > >> storage - PostGIS and SQLite".
> > > >> > So could you suggest anything in this direction. What should
> > be in the
> > > >> IDEA related to Pywps.
> > > >> >
> > > >> > Also could you suggest on which modules we have to add
> > database
> > > >> interaction
> > > >> > in Pywps code base.
> > > >> >
> > > >> > --
> > > >> >
> > > >> > Regards,
> > > >> > Naveen Panwar
> > > >> > Senior Undergraduate Student
> > > >> > IIIT-Hyderabad, India.
> > > >> >
> > > >> > [10][12]http://lsi.iiit.ac.in/naveen.panwar
> > > >> >
> > > >>
> > > >> --
> > > >> Jachym Cepicky
> > > >> URL: [11][13]http://les-ejk.cz
> > > >> e-mail: jachym.cepicky at gmail com
> > > >> PGP: [12][14]http://les-ejk.cz/pgp/JachymCepicky.pgp
> > > >> @jachymc
> > > >>
> > > >> --
> > > >> Regards,
> > > >> Naveen Panwar
> > > >> Senior Undergraduate Student
> > > >> IIIT-Hyderabad, India.
> > > >> [13][15]http://lsi.iiit.ac.in/naveen.panwar
> > > >>
> > > >> References
> > > >>
> > > >> Visible links
> > > >> 1.
> > [16]http://netmar.nersc.no/sites/netmar.nersc.no/files/D7.7_WPS_Cookbook_r1_20111229.pdf
> > > >> 2.
> > [17]http://netmar.nersc.no/sites/netmar.nersc.no/files/D7.7_WPS_Cookbook_r1_20111229.pdf
> > > >> 3. mailto:[18]jachym.cepicky at gmail.com
> > > >> 4.
> > [19]https://github.com/jachym/pywps-4/blob/master/pywps/inout.py#L227
> > > >> 5. [20]http://www.opengeospatial.org/standards/wps
> > > >> 6. [21]http://wiki.rsg.pml.ac.uk/pywps/Main_Page
> > > >> 7. [22]http://pywps.wald.intevation.org/documentation/
> > > >> 8. [23]http://pywps.readthedocs.or/
> > > >> 9. mailto:[24]panwarn09 at gmail.com
> > > >> 10. [25]http://lsi.iiit.ac.in/naveen.panwar
> > > >> 11. [26]http://les-ejk.cz/
> > > >> 12. [27]http://les-ejk.cz/pgp/JachymCepicky.pgp
> > > >> 13. [28]http://lsi.iiit.ac.in/naveen.panwar
> > > >
> > >
> > >
> > > --
> > > ISRIC - World Soil Information Post: PO box 353, 6700 AJ, Wageningen,
> > > The Netherlands Visiting Address: Droevendaalsesteeg 3, 6708 PB
> > > Wageningen (Bdg. 101), Office: C.013 Office Phone: +31 (0) 317 4 83715
> > > Mobile Phone: +31 (0) 613 9 06950 OpenPGPKey: 0xA3D0065A
> > >
> >
> > --
> > Jachym Cepicky
> > URL: [29]http://les-ejk.cz
> > e-mail: jachym.cepicky at gmail com
> > PGP: [30]http://les-ejk.cz/pgp/JachymCepicky.pgp
> > @jachymc
> > _______________________________________________
> > pywps-dev mailing list
> > [31]pywps-dev at lists.osgeo.org
> > [32]http://lists.osgeo.org/cgi-bin/mailman/listinfo/pywps-dev
> >
> > --
> > Regards,
> > Naveen Panwar
> > Senior Undergraduate Student
> > IIIT-Hyderabad, India.
> > [33]http://lsi.iiit.ac.in/naveen.panwar
> >
> > References
> >
> > Visible links
> > 1. mailto:jachym.cepicky at gmail.com
> > 2. http://istgeo.ist.supsi.ch/software/istsos/
> > 3.
> > http://www.timeanddate.com/worldclock/fixedtime.html?iso=20140313T20&ah=1
> > 4.
> > http://netmar.nersc.no/sites/netmar.nersc.no/files/D7.7_WPS_Cookbook_r1_20111229.pdf
> > 5. mailto:jachym.cepicky at gmail.com
> > 6. https://github.com/jachym/pywps-4/blob/master/pywps/inout.py#L227
> > 7. http://www.opengeospatial.org/standards/wps
> > 8. http://wiki.rsg.pml.ac.uk/pywps/Main_Page
> > 9. http://pywps.wald.intevation.org/documentation/
> > 10. http://pywps.readthedocs.or/
> > 11. mailto:panwarn09 at gmail.com
> > 12. http://lsi.iiit.ac.in/naveen.panwar
> > 13. http://les-ejk.cz/
> > 14. http://les-ejk.cz/pgp/JachymCepicky.pgp
> > 15. http://lsi.iiit.ac.in/naveen.panwar
> > 16.
> > http://netmar.nersc.no/sites/netmar.nersc.no/files/D7.7_WPS_Cookbook_r1_20111229.pdf
> > 17.
> > http://netmar.nersc.no/sites/netmar.nersc.no/files/D7.7_WPS_Cookbook_r1_20111229.pdf
> > 18. mailto:jachym.cepicky at gmail.com
> > 19. https://github.com/jachym/pywps-4/blob/master/pywps/inout.py#L227
> > 20. http://www.opengeospatial.org/standards/wps
> > 21. http://wiki.rsg.pml.ac.uk/pywps/Main_Page
> > 22. http://pywps.wald.intevation.org/documentation/
> > 23. http://pywps.readthedocs.or/
> > 24. mailto:panwarn09 at gmail.com
> > 25. http://lsi.iiit.ac.in/naveen.panwar
> > 26. http://les-ejk.cz/
> > 27. http://les-ejk.cz/pgp/JachymCepicky.pgp
> > 28. http://lsi.iiit.ac.in/naveen.panwar
> > 29. http://les-ejk.cz/
> > 30. http://les-ejk.cz/pgp/JachymCepicky.pgp
> > 31. mailto:pywps-dev at lists.osgeo.org
> > 32. http://lists.osgeo.org/cgi-bin/mailman/listinfo/pywps-dev
> > 33. http://lsi.iiit.ac.in/naveen.panwar

> --
> Jachym Cepicky
> URL: http://les-ejk.cz
> e-mail: jachym.cepicky at gmail com
> PGP: http://les-ejk.cz/pgp/JachymCepicky.pgp
> @jachymc
> _______________________________________________
> pywps-dev mailing list
> pywps-dev at lists.osgeo.org
> http://lists.osgeo.org/cgi-bin/mailman/listinfo/pywps-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pywps-dev/attachments/20140312/5342db37/attachment-0001.html>

More information about the pywps-dev mailing list