[Zoo-discuss] Fwd: About ZOO Project

[Soeren Gebbert]

Hello Gérald,
thanks a lot for this constructive discussion, i really appreciate it. :)

Running GRASS as high performance WPS server back end is indeed not trivial.
I would like to discuss some of my ideas with the ZOO community (i
will repeat parts of my former mail).
The easiest way is the following:

GRASS modules need there data in a grass mapset or to be linked with
r.external (raster)
or v.external (vector). So the first thing to do is to is to write the
data into the mapset with
r.in.gdal or v.in.ogr, or create links with r/v.external using fork() or
to write directly into the grass mapset using the gdal libraries.

Now we can call the grass module using fork()

The computation result may be ascii data from stdout or raster/vector
data generated in the
grass mapset.

Now the data must be exported to a format which the WPS server can deliver.
So r.out.gdal or v.out.ogr may be called using fork() or, if possible,
the gdal library reads directly from the grass mapset.

In case single raster or vector data is generated we need at least 3
fork() calls to start the correct
grass module for import/linking, computation and export. And more in
case of multiple inputs and outputs.
All this can easily be managed using shell scripts or python scripts.

Error handling:
For each called grass module (import, computation, export) the return
value must be checked and the stderr output interpreted.
These error data/messages must be converted in a meaningful WPS error response.

Suspend and resume:
To support the suspend of long running processes or to give high
priority process more CPU power
the grass modules must be suspendable and resumable. Well this is IMHO
quite difficult using a
generated shell or python script. The scripts must be listen to stdin
or communicate over a socked with the WPS server to
suspend the currently running process.

In case of this simple solution, only the suspend issue is hard to
solve. But we have a lot of fork() calls, which may result in slow
computation time for small datasets. In case of large datasets this
may be irrelevant.


Now lets talk about a solution using grass modules as libraries linked
to a forked process from the ZOO kernel.
I assume the following:
The grass module are modified so the main function has been replaced
with a function callable from within a process.
i.e: the main() function from r.los will be renamed to r_los_main().
The module will be provided as shared object named
grasslib_r_los.so together with the header file r_los.h.
This is only practical for grass C-modules, shell or python will not
work this way.

The process must provide the command line arguments as char **arrays:
{"input=inputmap", "output=outputmap", "coordinate=2344,4555",
"obs_elev=2", "-c", "--v", "--o"}
to the r_los_main() function. There is no way around.

In case the forked ZOO Process do not need to deliver a log file or do
not need to
communicate in any way with the ZOO server, the grass error function
must not be replaced.
In case an error in grass occurs, the forked process will terminate
with a defined error state and the error message is written to stderr.

If the forked process need to communicate with the ZOO server, than
the error function must be replaced in a way that
segfaults in the module object are avoided. IMHO this can only be
solved with the setjmp/longjmp construct. Un-freed memory and unclosed
file descriptors are not a problem, because the OS will hopefully
close the descriptors and frees the memory of this process after
termination.
Because the mapset for this process was generated temporary, it can be
deleted without caring about corrupted computational data.

Here we are:

// setjmp()/longjmp() macros:
#define TRY         if (!setjmp(stack_buffer)) {
#define CATCH  } else { /*create a proper ZOO error message */;}


// The error handler:
int error_handler(const char *msg, int fatal)
{
    if (fatal == 0) {
        fprintf(stderr, "%s\n", msg);
        return 1;
    }
    else {
        fprintf(stderr, "%s\n", msg);
        longjmp(stack_buffer, 1);
    }
    return 1;
}

/* Set the error handler */
G_set_error_routine(error_handler);

/* Catch the stdout and stderr file descriptor*/
...
/* Create the temporal mapset, compose the import arguments and import
the data*/
...
TRY
r_in_gdal_main(inargc, inargv); // or r_external_main()
CATCH

/*compose the computation arguments and start the module */
...
TRY
r_los_main(argc, argv);
CATCH

/* Compose the output arguments and export the data into the WPS
server response format*/
...
TRY
r_out_gdal_main(outargc, outargv);
CATCH
/* Create a response document for the ZOO server*/
/* Remove the temporal mapset*/


This approach do not need fork() within the calling process and may be
a bit faster than the first approach.
Also, the ZOO server may be capable to suspend and resume the forked
process and therefor the computation?
I don't know how to implement hat.



About the grass module calling process which will be forked by the ZOO server:
I can imagine that the following approach may be interesting,

Lets name the process which calls the grass modules "grass_module_handler()".
It would be nice if IMHO the argument for the grass_module_handler()
is a WPS execute XML request.
All geodata from the original XML Execute request has been downloaded
or extracted by the ZOO server and saved on the local file system.
References to the files on the local file systems replaces the input
and output parameter of the original WPS Execute document generated
for
the grass_module_handler().
The grass_module_handler() will interpret the input and output
description and calls the correct grass modules as shared objects.
At the end the grass_module_handler() will generate the response
document pointing to the result data on the local file system.
The WPS server interprets the response XML documents and replaces the
local references or streams the result data from the file system
directly to the service consumer.

So the ZOO server will not care about how to call the grass modules in
which order and so on.
This will be handled by the grass_module_handler() based on the WPS
execute request modified by the ZOO server.

If the grass_module_handler() is implemented in a generic way
(function pointer ...), it can execute most of the grass modules.

Additionally, we at the grass developer team, discussing the automatic
generation of WPS description documents by grass.
Because each grass module knows exactly what input and output is
needed, it can provide such information directly as
a WPS XML document. So you can generate automatically a valid WPS
description document for each grass module and place it on the WPS
server.

The only issue i have with this solution is the mime type of the input
and output raster/vector data in the WPS document.
A grass modules expects as input or output a local file path or
raster/vector data in the location.
My suggestion is that every grass module writes all by gdal supported
formats as input/output mime types for raster/vector data.
In this case the WPS server do not need to interpret the grass
input/output parameters, it can deliver the WPS description documents
without modification.

My knowledge of WPS XML documents are very limited, but i have the
"feeling" that this may work. It would be great if the WPS experts
could verify my idea? :)



Sorry for my egnlish and the long response,
best regards
Soeren



2009/11/9 Gérald Fenoy <gerald.fenoy at geolabs.fr>:
> Sorry for this mistake,
> we've sent this mail 30 minutes before the time I have to leave the
> conference center ....
> I will try to be more precise and to give my new thoughts about this issue
> of recode partially the main function. In fact I think about creating a new
> environment specific to ZOO Services redirecting std output, input and error
> and handling the exit function using an internal ZOO Kernel one ... let me
> think again to this idea I get into the plane about this remarks,
> thank you again Soeren for sharing your thoughts !
> Best regards,
>
> Début du message réexpédié :
>
> De : Gérald Fenoy <gerald.fenoy at geolabs.fr>
> Date : 7 novembre 2009 04:13:19 HNEC
> À : Soeren Gebbert <soerengebbert at googlemail.com>
> Objet : Rép : [Zoo-discuss] About ZOO Project
>
> Hi Soeren,
>
> Sorry for the delay in my response.
>
>
> Le 4 nov. 2009 à 12:35, Soeren Gebbert a écrit :
>
> Hello Venkatesh,
>
> 2009/11/3 Venkatesh Raghavan <raghavan at media.osaka-cu.ac.jp>
>
> .....
>
> have all the GRASS functions in the command repository.
>
> This way we will be able to run any GRASS command and
>
> also code new command using GRASS libraries.
>
> Ok, having the grass libraries in the ZOO kernel, allows developers to
>
> develop the business logic (in case of an WPS server the geo
>
> processing) of the WPS server using the ZOO api?
>
> Yes.
>
> I assume that the ZOO kernel uses threads to handle requests and to
>
> execute the business logic. So we have a multi-threaded environment?
>
> We use fork to manage the request. We do not have multi thread environment.
> if the process fails, ZOO kernel still exists in memory.
>
> Linking and using the GRASS libs in a multi-threaded environment will
>
> bring two major issues:
>
> 1.) GRASS libraries are not thread safe
>
> Yes, as many other API's like MapServer etc.
>
> * Most of the library functions using static variables for computation
>
> * parallel access to the vector database is AFAIK not allowed
>
> * to enable multi-threading support for GRASS libraries, a major
>
> update to GRASS must be done
>
> Yes, maybe but ZOO does not plan to do this, presently.
> It would be great if someone can address these issues.
>
> 2.) Many GRASS libraries call exit() on error
>
> So only child process will exit since ZOO Kernel still exists in memory.
>
> * In case an error occurred exit() is called so the WPS server will
>
> shutdown. I assume that is not practical ;)
>
> Agreed.
>
> * the error function call which invokes the exit() call can be
>
> replaced using G_set_error_routine()
>
> OK, maybe better in any case. G_set_error_routine have only to return
> SERVICE_FAILED and the error message.
>
> * To save the stack and assure a valid library state the combination
>
> of setjmp and longjmp must be used. This is implemented in
>
> vtk-grass-bridge. Have a look at this code:
>
> http://code.google.com/p/vtk-grass-bridge/source/browse/trunk/Common/vtkGRASSInit.cxx
>
> http://code.google.com/p/vtk-grass-bridge/source/browse/trunk/Common/vtkGRASSDefines.h
>
> Nice, we will have a look.
>
> * The GRASS libs need to be patched to assure a valid library state
>
> in case the error function using setjmp/longjmp was called (close open
>
> file handler, release unused memory, reinitialize static variables and
>
> so on ...)
>
> Should be tricky. Wonder if we should consider SEXTANTE. Wonder if
> SEXTANTE is thread safe?
>
> Wow, 20 GRASS examples is a lot ... . I have read the trac documentation
>
> and lurked a bit in the source code. I think i need more investigation
>
> to fully understand
>
> the functionality of the ZOO kernel. :)
>
> Aim for 20, that we could be sure to acheive 10.
>
>
> Ah, ok gdal read and write directly raster and vector data from grass
> mapsets?
>
> Thats cool. I think additionally the creation of a temporal mapset for every
>
> response is still needed (see PyWPS). Otherwise grass modules will interfere
>
> each other while computation.
>
> Have to check about the vector. GRASS raster write is supported by GDAL.
> Maybe temporal Mapset is necessary.
>
>
> ...
>
> Is there a way to connect the grass modules as shared objects to zoo to
> support
>
> i.e. the suspending of long running wps processes (without the
>
> modification of the modules)?
>
> Sure, you only have to create a shared library which use GRASS module
>
> which you want to use as a service. Suspend is already implemented
>
> if you use the storeExecuteResponse already in WPS specification.
>
> If you use this parameter this will imply that the ZOO kernel will
>
> fork to spawn new process which can run your function without
>
> having to wait for response which was already provided by the
>
> parent process.
>
> Currently all GRASS modules are stand alone C programs, shell- or python
>
> scripts. In case of shell scripts, i have no clue how to create shared
>
> libraries from.
>
> In case of the GRASS modules implemented in C, the main functions need
>
> to be replaced
>
> with a function which can be called from ZOO to invoke the service.
>
> What have to be done in GRASS to support this:
>
> 1.) In every GRASS module which should be used in ZOO the program
>
> structure needs to be changed:
>
> * Tuning the Makefiles to create an executable and a shared object
>
> * #ifdef switches to support the default main() function and the
>
> replacement function called from ZOO
>
> * Creation of a header file declaring the function which should be
>
> called from ZOO to invoke the module as a service
>
> ** i.e: r.los -> r_los_main(int argc, char **argv);
>
> sounds like a lot of work.can we find a developer to do this ?
>
> * ZOO need to provide the command line options for every module as char
> arrays
>
> ZOO Kernel use special data structure called maps.
>
> 2.) All modules call exit() in case of an error and are not thread safe
>
> * The error routine can be replaced using G_set_error_routine(), but
>
> each module must be reviewed and patched to clean up before the the
>
> error function is called and to return properly to the ZOO kernel.
>
> Thats a lot of work!
>
> I think you're right, it seems to be a lot of work. Is there any generic way
> to achieve this goals ?
>
> * Because the GRASS libraries are not thread safe, the module are
>
> neither. And in case the GRASS libs will be thread safe some times,
>
> the modules still need to be reviewed for multi threading support.
>
>
> Yes.
>
> In case grass modules are called from forked processes and not from
>
> threads of a single process, there should be no problems.
>
> My suggestion is to run GRASS modules as separate processes in
>
> temporal created mapsets.
>
> In fact this is a good sugestion.
>
> I will answer in more detail about the securre connexion later.
>
> Gérald & Venka
>
> ZOO Tribe Monkeys.
>
> Gérald Fenoy
> gerald.fenoy at geolabs.fr
>
> GEOLABS
> Siège social :
> Futur Building I
> 1280, avenue des Platanes
> 34970 Lattes
> Tél. fixe : 04 67 53 67 37
> Tél. portable : 06 70 08 25 39
>
> _______________________________________________
> Zoo-discuss mailing list
> Zoo-discuss at gisws.media.osaka-cu.ac.jp
> http://gisws.media.osaka-cu.ac.jp/mailman/listinfo/zoo-discuss
>
>