[GRASS-dev] [GRASS GIS] #2134: Create a general exit-safe interface to C libraries

[GRASS GIS]

#2134: Create a general exit-safe interface to C libraries
--------------------------------------------------+-------------------------
 Reporter:  wenzeslaus                            |       Owner:  grass-dev@…              
     Type:  enhancement                           |      Status:  new                      
 Priority:  normal                                |   Milestone:  7.0.0                    
Component:  Python ctypes                         |     Version:  svn-trunk                
 Keywords:  G_fatal_error, exit, multiprocessing  |    Platform:  All                      
      Cpu:  Unspecified                           |  
--------------------------------------------------+-------------------------

Comment(by huhabla):

 In my opinion the RPC approach is only meaningful for persistent
 applications that need fast access to C-library functions, or that need
 low level API access for data modification (like digitizing).

 My intention to write the RPC server was to make the temporal framework
 usable in persistent applications and to be as fast as possible.

 Extending the PyGRASS framework with signal handling will not solve the
 problem: that memory structures may be corrupted in case of a
 G_fatal_error() call or a segfault, leading to incorrect computation
 results or to segmentation faults in the persistent application.

 Implementation of persistent applications that use the GRASS C-library
 functions in their process memory is IMHO not possible. We have had this
 discussion for years.

 The PyGRASS interface is well designed for module programming not for
 persistent applications. Otherwise each C-function call should be handled
 via RPC. From my point of view and some tests that i made slows the RPC
 approach the processing significantly down.

 Example:

 {{{
 from multiprocessing import Process, Pipe, Lock
 import time

 num = 50000

 def func(a):
     b = a + 1
     c = b + 1
     d = c + b
     return d

 # DICT
 def dict_func():
     a = {}
     for i in xrange(num):
         a["%i%i%i%i"%(i,i,i,i)] = func

     start = time.time()

     for i in xrange(num):
         a["%i%i%i%i"%(i,i,i,i)](5)

     end = time.time()

     print end - start


 def target(lock, conn):
     a = [func]*num
     while True:
         # Avoid busy waiting
         conn.poll(4)
         data = conn.recv()
         lock.acquire()
         ret = a[data[0]](data[1])
         conn.send(ret)
         lock.release()

 def rpc_func():
     client_conn, server_conn = Pipe()
     lock = Lock()
     server = Process(target=target, args=(lock, server_conn))
     server.daemon = True

     server.start()


     start = time.time()
     for i in xrange(num):
         if server.is_alive() is True:
             data = [i, 5]
             client_conn.send(data)
             client_conn.recv()

     end = time.time()

     print end - start



 if __name__ == "__main__":
     # Measuring the time to call of 50000 functions that are
     # registered in a dictionary
     dict_func()
     # Measuring the time to call 50000 functions
     # using an RPC approach
     rpc_func()
 }}}

 Running the script will show that calling functions from a dict is 50 time
 faster than using a pipe with a subprocess.
 {{{
 python rpc_test.py
 0.0984511375427
 4.59985399246
 }}}

-- 
Ticket URL: <http://trac.osgeo.org/grass/ticket/2134#comment:7>
GRASS GIS <http://grass.osgeo.org>