[Zoo-discuss] Fwd: About ZOO Project

[Gérald Fenoy]

Sorry for this mistake,
we've sent this mail 30 minutes before the time I have to leave the  
conference center ....

I will try to be more precise and to give my new thoughts about this  
issue of recode partially the main function. In fact I think about  
creating a new environment specific to ZOO Services redirecting std  
output, input and error and handling the exit function using an  
internal ZOO Kernel one ... let me think again to this idea I get into  
the plane about this remarks,

thank you again Soeren for sharing your thoughts !

Best regards,

Début du message réexpédié :

> De : Gérald Fenoy <gerald.fenoy at geolabs.fr>
> Date : 7 novembre 2009 04:13:19 HNEC
> À : Soeren Gebbert <soerengebbert at googlemail.com>
> Objet : Rép : [Zoo-discuss] About ZOO Project
>
> Hi Soeren,
>
> Sorry for the delay in my response.
>
>
> Le 4 nov. 2009 à 12:35, Soeren Gebbert a écrit :
>
>> Hello Venkatesh,
>>
>> 2009/11/3 Venkatesh Raghavan <raghavan at media.osaka-cu.ac.jp>
> .....
>>
>>> have all the GRASS functions in the command repository.
>>> This way we will be able to run any GRASS command and
>>> also code new command using GRASS libraries.
>>
>> Ok, having the grass libraries in the ZOO kernel, allows developers  
>> to
>> develop the business logic (in case of an WPS server the geo
>> processing) of the WPS server using the ZOO api?
>
> Yes.
>
>> I assume that the ZOO kernel uses threads to handle requests and to
>> execute the business logic. So we have a multi-threaded environment?
>
> We use fork to manage the request. We do not have multi thread  
> environment.
> if the process fails, ZOO kernel still exists in memory.
>
>> Linking and using the GRASS libs in a multi-threaded environment will
>> bring two major issues:
>> 1.) GRASS libraries are not thread safe
>
> Yes, as many other API's like MapServer etc.
>
>> * Most of the library functions using static variables for  
>> computation
>> * parallel access to the vector database is AFAIK not allowed
>> * to enable multi-threading support for GRASS libraries, a major
>> update to GRASS must be done
>
> Yes, maybe but ZOO does not plan to do this, presently.
> It would be great if someone can address these issues.
>
>> 2.) Many GRASS libraries call exit() on error
>
> So only child process will exit since ZOO Kernel still exists in  
> memory.
>
>> * In case an error occurred exit() is called so the WPS server will
>> shutdown. I assume that is not practical ;)
>
> Agreed.
>
>> * the error function call which invokes the exit() call can be
>> replaced using G_set_error_routine()
>
> OK, maybe better in any case. G_set_error_routine have only to  
> return SERVICE_FAILED and the error message.
>
>> * To save the stack and assure a valid library state the combination
>> of setjmp and longjmp must be used. This is implemented in
>> vtk-grass-bridge. Have a look at this code:
>> http://code.google.com/p/vtk-grass-bridge/source/browse/trunk/Common/vtkGRASSInit.cxx
>> http://code.google.com/p/vtk-grass-bridge/source/browse/trunk/Common/vtkGRASSDefines.h
>
> Nice, we will have a look.
>
>> * The GRASS libs need to be patched to assure a valid library state
>> in case the error function using setjmp/longjmp was called (close  
>> open
>> file handler, release unused memory, reinitialize static variables  
>> and
>> so on ...)
>
> Should be tricky. Wonder if we should consider SEXTANTE. Wonder if
> SEXTANTE is thread safe?
>
>> Wow, 20 GRASS examples is a lot ... . I have read the trac  
>> documentation
>> and lurked a bit in the source code. I think i need more  
>> investigation
>> to fully understand
>> the functionality of the ZOO kernel. :)
>
> Aim for 20, that we could be sure to acheive 10.
>
>>
>> Ah, ok gdal read and write directly raster and vector data from  
>> grass mapsets?
>> Thats cool. I think additionally the creation of a temporal mapset  
>> for every
>> response is still needed (see PyWPS). Otherwise grass modules will  
>> interfere
>> each other while computation.
>
> Have to check about the vector. GRASS raster write is supported by  
> GDAL.
> Maybe temporal Mapset is necessary.
>
>>
>>> ...
>>>> Is there a way to connect the grass modules as shared objects to  
>>>> zoo to support
>>>> i.e. the suspending of long running wps processes (without the
>>>> modification of the modules)?
>>>
>>> Sure, you only have to create a shared library which use GRASS  
>>> module
>>> which you want to use as a service. Suspend is already implemented
>>> if you use the storeExecuteResponse already in WPS specification.
>>> If you use this parameter this will imply that the ZOO kernel will
>>> fork to spawn new process which can run your function without
>>> having to wait for response which was already provided by the
>>> parent process.
>>
>> Currently all GRASS modules are stand alone C programs, shell- or  
>> python
>> scripts. In case of shell scripts, i have no clue how to create  
>> shared
>> libraries from.
>> In case of the GRASS modules implemented in C, the main functions  
>> need
>> to be replaced
>> with a function which can be called from ZOO to invoke the service.
>> What have to be done in GRASS to support this:
>> 1.) In every GRASS module which should be used in ZOO the program
>> structure needs to be changed:
>> * Tuning the Makefiles to create an executable and a shared object
>> * #ifdef switches to support the default main() function and the
>> replacement function called from ZOO
>> * Creation of a header file declaring the function which should be
>> called from ZOO to invoke the module as a service
>> ** i.e: r.los -> r_los_main(int argc, char **argv);
>
> sounds like a lot of work.can we find a developer to do this ?
>
>> * ZOO need to provide the command line options for every module as  
>> char arrays
>
> ZOO Kernel use special data structure called maps.
>
>> 2.) All modules call exit() in case of an error and are not thread  
>> safe
>> * The error routine can be replaced using G_set_error_routine(), but
>> each module must be reviewed and patched to clean up before the the
>> error function is called and to return properly to the ZOO kernel.
>> Thats a lot of work!
>
> I think you're right, it seems to be a lot of work. Is there any  
> generic way to achieve this goals ?
>
>> * Because the GRASS libraries are not thread safe, the module are
>> neither. And in case the GRASS libs will be thread safe some times,
>> the modules still need to be reviewed for multi threading support.
>>
>
> Yes.
>
>> In case grass modules are called from forked processes and not from
>> threads of a single process, there should be no problems.
>> My suggestion is to run GRASS modules as separate processes in
>> temporal created mapsets.
>
> In fact this is a good sugestion.
>
> I will answer in more detail about the securre connexion later.
>
> Gérald & Venka
>
> ZOO Tribe Monkeys.

Gérald Fenoy
gerald.fenoy at geolabs.fr


GEOLABS
Siège social :
Futur Building I
1280, avenue des Platanes
34970 Lattes
Tél. fixe : 04 67 53 67 37
Tél. portable : 06 70 08 25 39

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/zoo-discuss/attachments/20091109/28242602/attachment.html>