[ZOO-Discuss] Python script to call grass 7 modules as service

[Soeren Gebbert]

Hello Gérald,
i am really sorry for the delay. I have had other things to do .... . :/
But here i am back again. :)

I am absolutely agree with you that grass is only a service provider
for the zoo project,
like many others. And the zoo kernel should not care if grass or any
other service is called.
The zoo config files and the load mechanism should be the same.
The GrassModuleStarter python script which i have written, will only
take care of all the
import and export things which are needed to get a grass module run. It is not
designed to be directly a service for the zoo project. But it shoudl
be used as is to
execute grass modules. I would like to suggest the following
approach in zoo to use the GrassModuleStarter python script:

1.) Implementation of a general grass module loader class for
zoo-services, which will
  import the GrassModuleStarter python script to start the processing
(grass_loader.py):

import sys
from GrassModuleStarter import *

class grassLoader():
	def __init__(self, conf, inputs, outputs):

               # Read grass specific settings from a config file
               # GRASS base dir, addon path, working directory, data directory
               # This config file can be put in the zoo-service grass directory?
               read_grass_config()
		
		# Write all the inputs into files to the file system
               do_export(inputs)

               # create the input file for the GrassModuleStarter
python script from the conf, input and output kvp's
               do_create_input_file(conf, inputs, outputs)
		filename = "new_input_file"
	
		# Start the GrassModuleStarter class, all the import/export/location
generation/module execution
               # is done here
		starter = GrassModuleStarter(filename)

               # read all the generated results and write it into
output[identifier]
               do_fill_output(outputs)
               return 3

2.) Create for each grass module which should be provide a service a
simple python script and a zoo config file:

r.los script may look like:

from grass_loader import *

def r_los(conf,inputs,outputs):
	loader = grassLoader(conf,inputs,outputs)

r.neighbors may look like:

from grass_loader import *

def r_neighbors(conf,inputs,outputs):
	loader = grassLoader(conf,inputs,outputs)

and so on. A zoo config file may look like:

[r_los]
 processVersion = 2
 storeSupported = true
 statusSupported = true
 serviceProvider = r_los
 serviceType = Python
wpsProcessDescription = r.los.xml


I would suggest to load directly WPS process description xml files, because
these files are generated by grass automatically for each module and
such xml files
are available for several services which are implemented in i.e.
52north (i.e. sextante, ArcGIS, ...).
This can be implemented in addition to the default zoo-service config
syntax. The user can
than choose if he want to code the process description in the
zoo-service config file,
or if he  provides an existing process description xml file.
Otherwise you need to code very complex zoo-service config files for each grass
module. And thats a lot of work.

I would suggest to use python as script language to execute grass
modules as services,
because i do not see any realistic chance in the near future to link
grass modules as shared
objects.

Best regards
Soeren

2010/1/7 Gérald Fenoy <gerald.fenoy at geolabs.fr>:
> Hi Soeren,
> always nice to hear from you.
>
> Le 3 janv. 2010 à 15:17, Soeren Gebbert a écrit :
>
>> Dear developers,
>> JFYI:
>> I have implemented a Python script which allows to call grass 7
>> modules as a service (e.g.: from a WPS server).
>> This script works currently only with grass raster modules. Vector
>> modules are work in progress. The processing is based
>> on grass version 7.
>
> Ok sounds promising.
>
>> The clue is that the user provides the name of the module, some
>> parameters and the input data as png or geotiff files and receives
>> geotiff or gml files as output.
>
> Great, in fact the ZOO Kernel use the same things in his heart. GRASS would only become a module of ZOO Server, in other words a Service Provider. The difference is that the name of the module is the Identifier in Execute Request, it won't point directly on GRASS module but on a function defined in a ZSP GRASS Based, DataInputs wasn't extracted from a file but also from the request, as for output, including mimeType and so on (this way user can dinamicaly change parameter without writing any file). By the way, I'm pretty sure you know that well.
>
> Nevertheless, we can easily imagine a generic way to load and run module, a similar way you used in your script, for the ZSP GRASS Based pacesses. This can became a function which permit to load and run a specific grass module.  Currently, ZOO Kernel only support to call function defined in such an ZSP, but this ZSP can then call others functions defined in linked shared libraries (against the ZSP).
>
> At the begining, I thought that if we suppose that we have to change the behavior of the ZOO Kernel internal mechanisms only for GRASS then we missed something somewhere. Indeed, from my point of view, ZOO Kernel use a generic way to dynamicaly load then run functions from a ZSP (coded in many languages), dealing with error on loading process and/or running process.
>
> Nevetheless, for each supported languages, we handle loading/running in a different way, only input parsing and output restitution was shared betwin languages. I think that, in some sens, we can view GRASS support in the ZOO Kernel as a language support.
>
> If we view GRASS as a new languages we can then easily implement stuff like you did in your script, doing specific stuff before running (automatic environment variables setting is available), run then doing specific stuff after running.
>
> It's similar to what we do in various language support.
>
>> All the location/mapset creation as well as the import and export is
>> done automatically by the script. The coordinate system of the
>> first input file is used to create the grass work location.
>
> Great, this can be used as a generic starting point for each GRASS processes which would be included in the work on  ZOO SP GRASS based. This can became a function in the ZOO API GRASS specific to help development of new processes.
>
>> Next features is, that all the input files are linked via r.external into
>> the grass work location. No explicit import is needed. Currently the output is
>> exported via r.out.gdal and v.out.ogr,
>> but r.external.out will be used in the future, so that raster data
>> will be processed via gdla access without import or export (thanks to
>> Glynn and Frank!).
>
> Cool, we will need this special gdal trick to remove import/export mechnism.
> Frank, if needed, can you help us on this specific requirement when it will be time ?
>
>> The Python script (some parts are based on pyWPS from Jachym Cepicky):
>> http://code.google.com/p/vtk-grass-bridge/source/browse/trunk/WPS/GrassModuleStarter.py
>>
>
> Thanks for this link.
>
>> Example input files:
>> http://code.google.com/p/vtk-grass-bridge/source/browse/trunk/WPS/Testing/Python/r.contour_input.txt
>> http://code.google.com/p/vtk-grass-bridge/source/browse/trunk/WPS/Testing/Python/r.watershed_input.txt
>>
>> I hope this script will simplify the use of grass as WPS server back-end.
>>
>
> Sure it will :)
>
>> The script is still under development and provides only basic features.
>> Better error and message handling is needed, as well as vector support
>> and testing.
>>
>
> ZOO Kernel handles errors in ZSP Python based to return an ExceptionReport as expected.
>
>> Any comments, suggestions, criticism are welcome. :)
>
> We really apriciate your support to the ZOO Project.
>
>> Have fun
>
> Hope we will have :)
>
> Best regards,
>
> Djay
> Just a ZOO Monkey
> gerald.fenoy at geolabs.fr
>
>
>
>