[Zoo-discuss] About ZOO Project

Wed Nov 4 03:35:51 PST 2009

Hello Venkatesh,

2009/11/3 Venkatesh Raghavan <raghavan at media.osaka-cu.ac.jp>:
> Hi Soeren,
>
> Soeren Gebbert wrote:
>
>> thanks a lot for your very informative response.
>
> Pleasure, your questions make me learn more about ZOO
> (with capital O).
> ...
>
>> I have additional questions about the grass support in zoo :)
>>
>> In which way can grass be connected to the zoo platform?
>
> Load all GRASS related libraries into the ZOO Kernel and
> have all the GRASS functions in the command repository.
> This way we will be able to run any GRASS command and
> also code new command using GRASS libraries.

Ok, having the grass libraries in the ZOO kernel, allows developers to
develop the business logic (in case of an WPS server the geo
processing) of the WPS server using the ZOO api?
I assume that the ZOO kernel uses threads to handle requests and to
execute the business logic. So we have a multi-threaded environment?
Linking and using the GRASS libs in a multi-threaded environment will
bring two major issues:
1.) GRASS libraries are not thread safe
  * Most of the library functions using static variables for computation
  * parallel access to the vector database is AFAIK not allowed
  * to enable multi-threading support for GRASS libraries, a major
update to GRASS must be done
2.) Many GRASS libraries call exit() on error
  * In case an error occurred exit() is called so the WPS server will
shutdown. I assume that is not practical ;)
  * the error function call which invokes the exit() call can be
replaced using G_set_error_routine()
  * To save the stack and assure a valid library state the combination
of setjmp and longjmp must be used. This is implemented in
vtk-grass-bridge. Have a look at this code:
  http://code.google.com/p/vtk-grass-bridge/source/browse/trunk/Common/vtkGRASSInit.cxx
http://code.google.com/p/vtk-grass-bridge/source/browse/trunk/Common/vtkGRASSDefines.h
  * The GRASS libs need to be patched to assure a valid library state
in case the error function using setjmp/longjmp was called (close open
file handler, release unused memory, reinitialize static variables and
so on ...)

>
>> Do you have a concrete concept?
>
> Yes. But at the first instance we would like to have some
> 20 GRASS examples for the developer guide to be released
> in March 2010. The concept could be concretized as we
> go along. Trac should give you a better idea of the
> concept.

Wow, 20 GRASS examples is a lot ... . I have read the trac documentation
and lurked a bit in the source code. I think i need more investigation
to fully understand
the functionality of the ZOO kernel. :)

>
>> Using the grass libraries will enable direct write and read access to the grass
>> database, but all the gis algorithms are implemented in grass as stand
>> alone modules?
>
> direct access to GRASS DB via gdal/ogr. Standalone algorithms as
> service. You could also integrate algorithms in one service.

Ah, ok gdal read and write directly raster and vector data from grass mapsets?
Thats cool. I think additionally the creation of a temporal mapset for every
response is still needed (see PyWPS). Otherwise grass modules will interfere
each other while computation.

> ...
>> Is there a way to connect the grass modules as shared objects to zoo to support
>> i.e. the suspending of long running wps processes (without the
>> modification of the modules)?
>
> Sure, you only have to create a shared library which use GRASS module
> which you want to use as a service. Suspend is already implemented
> if you use the storeExecuteResponse already in WPS specification.
> If you use this parameter this will imply that the ZOO kernel will
> fork to spawn new process which can run your function without
> having to wait for response which was already provided by the
> parent process.

Currently all GRASS modules are stand alone C programs, shell- or python
scripts. In case of shell scripts, i have no clue how to create shared
libraries from.
In case of the GRASS modules implemented in C, the main functions need
to be replaced
with a function which can be called from ZOO to invoke the service.
What have to be done in GRASS to support this:
1.) In every GRASS module which should be used in ZOO the program
structure needs to be changed:
  * Tuning the Makefiles to create an executable and a shared object
  * #ifdef switches to support the default main() function and the
replacement function called from ZOO
  * Creation of a header file declaring the function which should be
called from ZOO to invoke the module as a service
  ** i.e: r.los -> r_los_main(int argc, char **argv);
  * ZOO need to provide the command line options for every module as char arrays
2.) All modules call exit() in case of an error and are not thread safe
  * The error routine can be replaced using G_set_error_routine(), but
each module must be reviewed and patched to clean up before the the
error function is called and to return properly to the ZOO kernel.
Thats a lot of work!
  * Because the GRASS libraries are not thread safe, the module are
neither. And in case the GRASS libs will be thread safe some times,
the modules still need to be reviewed for multi threading support.

In case grass modules are called from forked processes and not from
threads of a single process, there should be no problems.
My suggestion is to run GRASS modules as separate processes in
temporal created mapsets.

>
>> This would be a huge advantage over the existing WPS implementations.
>
> Yes, that would be the aim and we need support from developers like
> you.
> ...
>> I am very interested. It would be great if you can give me trac access.
>
> Done.

Thanks. :)

> ...
>> Is the support of WSDL/SOAP planed for the future? I can imaging that
>> companies would like to integrate a WPS server in there existing SOA
>> infrastructure using WSDL/SOAP protocols.
>
> Surely, ZOO Tribe is not so clever in this domain.But looking for better
> ways for service chaining and web service orchestration.
> ...
>> Additional features like authorization, so that only authorized clients are
>> able to consume services.
>> A fine grain role support, so that different clients have access to
>> different services.
>
> Looking into GeoSheild to provide the above services,
> developed by our colleague Maxi in Swiss.
>
>> Secure connections across several server (not https) like WS-S.
>> DRM support for geodata.
>> Handling of electronic payment for geodata if requested.
>
> What is WS-S?

WebService-Security is an approach used by SOAP to establish secure
data exchange:
http://en.wikipedia.org/wiki/WS-Security

>
> Yes, we will need to look into these things. https should not

HTTPS and therefor SSL/TSL is always the first choice for secure data
transport in point to point connections. Problems arise in case there
are intermediary services which route the data from the consumer to
the service provider. In case of SSL/TSL the intermediary service need
to decrypt and encrypt the data and has therefor always access to the
decrypted data. This may be insecure, i.e: the service has been
hacked, the owner of the service is not trustable ... .
The developer of the WS-Security approach have had this in mind.

> be difficult. GeoPrisma presented at FOSS4G-Sydney could provide
> an interesting way to provide https support. Technologies
> used for any shopping cart application  should suffice for e-payment
> interface. e-payment not only for data but for WPS too, depending on
> CPU load, volume of data output urgency of request.

Indeed. E-payment based on Volume, CPU load and so on is a great idea.

Best regards
Soeren

>
>> I hope i am not offensive in my questioning.
>
> Please feel free to question. We learn more this way.
>
>> Maybe i sound a bit naive in my questions and all this is already
>> solved by using existing
>> software in the zoo software pool?
>
> Einstein said, computer were stupid because they can give answers but
> not ask questions. Very true, I think. So let the questions keep
> coming.
>
> Best regards
>
> Venka
> _______________________________________________
> Zoo-discuss mailing list
> Zoo-discuss at gisws.media.osaka-cu.ac.jp
> http://gisws.media.osaka-cu.ac.jp/mailman/listinfo/zoo-discuss
>