[Qgis-developer] [GRASS-dev] GRASS & QGIS: the future

Wed Apr 23 07:02:02 PDT 2014

Hi Radim,

2014-04-23 14:50 GMT+02:00 Radim Blazek <radim.blazek at gmail.com>:
> On Tue, Apr 22, 2014 at 4:49 PM, Sören Gebbert
> <soerengebbert at googlemail.com> wrote:
>>>> IMHO we GRASS developers are  too stubborn to change the design of
>>>> GRASS GIS to work as library with persistent applications. This would
>>>> require rewriting plenty of core functionality and modification of all
>>>> C-modules. But there is a solution for persistent applications, called
>>>> Remote Procedure Call (RPC), more below.
>>>
>>> It sounds interesting but it seems to be a huge work an far future. I
>>
>> Its far less work then modifying GRASS libraries to run with
>> persistent applications.
>
> We have already the providers and the plugin working with GRASS 6, I
> just want to upgrade it to GRASS 7. The only problem I found so far is
> that G__get_window() and Rast_get_cellhd() call G_fatal_error(). I can
> only dedicate few hours/max days of my free time to that, so I have to
> consider well what is less work, RPC is not.

Yes, i absolutely understand this.

>
>> We need
>> such an approach anyway, since vector editing is a GUI task -> GUI is
>> a persistent
>> application -> we need an exit safe interface to the GRASS libraries
>> to implement a reliable
>> vector editing GUI in GRASS itself.
>
> Vector lib does not call G_fatal_error(), it was written in parallel
> with QGIS plugin with vector editing in mind. I hope it is still true.

I fear that might be not the case anymore. For example, all memory
allocation functions G_calloc(), G_malloc() and G_realloc() will call
G_fatal_error() in case the memory allocation fails. As far as i can
tell the vector library make often use of this functions. So we have
the problem that basic core functions call G_fatal_error() that are
used all over the place in GRASS GIS.

>
>>> am also concerned about complexity, e.g. starting / keeping running
>>> servers on various platforms. I can imagine endless bug reports about
>>> "server not running". I am looking for something which will not
>>
>> Since only QGIS is spawning the "server" process, it is always able to
>> check if the
>> process is running and can restart it when needed. We already make use
>> of this concept in GRASS
>> and it works nicely[1]. Spawning of processes and keeping track of
>> them is an easy task with Qt,
>> and it is completely OS independent. The Qt Bitcoin client for example
>> makes use of RPC
>> and runs on Windows, Mac OS and various Linux systems. So i don't see
>> a problem here.
>
> The raster provider is already using something like that. Raster data
> are read from executed GRASS module output (a module written for that
> in QGIS in fact) and identify tool keeps another GRASS module running
> and it is communicating with that through a  pipe. I wrote that as an
> experiment and because I know, how much unnecessary additional code
> and potential problems it is, I don't want to continue in that
> direction.

Ok.

>
>>> require permanent assistance. It is not KISS enough for me.
>>
>> What would be a better KISS solution?
>>
>> IMO the RPC approach will make the implementation of the QGIS-GRASS
>> data provider and the
>> plugin simpler. There will be no mixing of C++ and C-code anymore.
>
> We are no starting the plugin from scratch, it is already written, we
> just need to make it work with GRASS 7.
>
>> No setjmp/longjmp/exception workaround that is really complex
>
> Why?

If you patch GRASS to use setjmp/longjmp to implement an exception
like behavior then you need to write clean up code for static
variables and dynamically allocated memory. A G_fatal_error() call can
be deeply nested in a library function. Cleaning only the static
variables to be in usable state will lead to problematic memory leaks.
So it is a complex task in GRASS.

Using setjmp/longjmp in QGIS wrapping all GRASS functions that might
call G_fatal_error() is complex, since a G_fatal_error() call can be
deeply nested in a library function. You need to keep track of this
resulting in plenty setjmp wrappers around GRASS functions.
You need to cleanup global static structures as well and you will have
no control over pointers pointing to dynamic allocated memory that got
lost in a longjmp.

IMHO booth cases require plenty of work to make them usable.

Having a kind of garbage collector in GRASS, implemented in the
G_*alloc() functions, may solve many problems. But i have no idea
howto implement the garbage collector to detect pointers that got
orphaned in a longjmp to automagically clean them.

The next step would be to centralize global variables to clean them up
more easily (as Glynn mentioned).

>
>> and will not prevent QGIS from
>> terminating or segfaulting eventually in case of a fatal error.
>
> Why?

Because of the undefined state of global variables and structures and
because of potential memory leaks that may sum up making QGIS
unusable.

Best regards
Soeren

>
> Radim
>
>> The QGIS GRASS data provider/plugin will be a clean C++ implementation
>> using the thrift library.
>> It will be better maintainable since developers do not need to know
>> all the different
>> GRASS libraries (gis, raster, vector, database) but only the RPC
>> interface that make
>> use of a subset of the GRASS library functions, bundling them in an
>> object oriented API.
>> QGIS do not need to link against GRASS libraries anymore, future GRASS
>> versions will be supported out of the box.
>>
>> This is much more KISS then the current approach.
>>
>> Best regards
>> Soeren
>>
>> [1] http://grass.osgeo.org/programming7/namespacepython_1_1temporal_1_1c__libraries__interface.html
>>>
>>> Radim
>>>
>>>> 2014-04-18 11:31 GMT+02:00 Radim Blazek <radim.blazek at gmail.com>:
>>>>> I have upgraded  the vector provider to GRASS 7, layers may be added
>>>>> by drag from browser. The raster and the plugin are disabled. Be
>>>>> careful about multiple versions on the same system
>>>>> (LD_LIBRARY_PATH..., check with ldd if does not work).
>>>>>
>>>>> Unfortunately GRASS 7 moved ahead towards its aim "to make life harder
>>>>> for anyone trying to use the GRASS libraries" [1]. Basically more and
>>>>> more low level functions are calling exit() instead of returning error
>>>>> code if something failed. As I am not willing to implement GRASS
>>>>> module call for each simple function, we have to think again about
>>>>> hacks we can use:
>>>>>
>>>>> 1) add a requirement that GRASS 7 used with QGIS must be compiled with
>>>>> -fexceptions
>>>>>
>>>>> 2) add a requirement that a patch (it is a single line comment in
>>>>> fact) must be applied to GRASS 7 to make it usable with QGIS
>>>>>
>>>>> 3) use setjmp()/longjmp()
>>>>>
>>>>> 4) let QGIS crash whenever GRASS lib function fails
>>>>
>>>> I fear that none of these suggestions are good working solutions,
>>>> since in case a fatal error or a segfault occurs nothing (no
>>>> exception, no setjmp) will prevent QGIS from crashing. So i would like
>>>> to suggest the following:
>>>>
>>>> 5.) Using a RPC interface for map metadata, vector and raster map
>>>> access. Hence one or several GRASS "server" processes provide an RPC
>>>> interface to access GRASS library functions that are needed in a
>>>> persistent application. Most important is the vector editing, vector
>>>> reading/writing, database access, raster reading and map metadata
>>>> support. Everything else can be done using modules. Hence the RPC
>>>> interface will support only a limited subset of the GRASS library
>>>> functions. This RPC interface should only be used in the GRASS
>>>> provider classes, the GRASS plugin itself can be written in C++ or
>>>> Python.
>>>>
>>>> This approach will decouple GRASS from QGIS, since all communication
>>>> is done via Inter Process Communication (IPC) using pipes or sockets.
>>>> There is no need anymore to catch a fatal error or to link GRASS
>>>> libraries directly to QGIS. The GRASS plugin can be implemented GRASS
>>>> version independently and so the GRASS data provider, since they will
>>>> not use GRASS functions directly only the RPC interface. The RPC
>>>> implementation on the side of GRASS will provide a consistent
>>>> interface that will not change in case the underlying GRASS API
>>>> changes.
>>>>
>>>> I strongly suggest to use an existing RPC framework to implement fast
>>>> binary data exchange and the IPC. My favorite is apache thrift[1],
>>>> since it supports plenty of programming languages (C/C++, Python,
>>>> Java, JavaScript, ...) and provides operating system independent
>>>> client and server functionality. It supports exceptions on client side
>>>> that can be emitted in case a GRASS server process died because of a
>>>> fatal error, SIGINT or segfault.
>>>>
>>>> I am absolutely willing to implement the RPC server side in GRASS
>>>> using thrift and C++, providing plenty of Python unit tests that show
>>>> howto use it on the client side. We just need to decide what kind of
>>>> GRASS library functionality is needed as RPC interface and what can be
>>>> done in QGIS with C++/Python directly (gisrc creation, GRASS
>>>> environmental variable settings, ...) or using GRASS modules
>>>> (g.region, g.gisenv, ...).
>>>>
>>>> [1] https://thrift.apache.org/
>>>>
>>>>
>>>> Best regards
>>>> Soeren
>>>>
>>>>>
>>>>> Radim
>>>>>
>>>>> [1]https://trac.osgeo.org/grass/ticket/869#comment:1
>>>>>
>>>>> On Thu, Mar 27, 2014 at 11:18 AM, Paolo Cavallini <cavallini at faunalia.it> wrote:
>>>>>> Hi all.
>>>>>> I learned during dinner that GRASS7 RC1 is due very soon. This opens the
>>>>>> issue of its functioning in QGIS. IMHO:
>>>>>>
>>>>>> * the qgis-grass-plugin might stop working (this has to be tested)
>>>>>> * some of the module options will be different
>>>>>> * new modules will not be available in QGIS.
>>>>>>
>>>>>> I think we can deal with this in several ways:
>>>>>>
>>>>>> * dropping the plugin, and concentrate the work on Processing
>>>>>> * upgrading both the plugin and Processing.
>>>>>>
>>>>>> In the first case, we have two major issues:
>>>>>>
>>>>>> * upgrading Processing GRASS modules
>>>>>> * changing the current Processing behaviour, avoiding the import-export
>>>>>> phase when piping consecutive GRASS commands; this makes GRASS modules
>>>>>> slower than the equivalent commands of other backends.
>>>>>>
>>>>>> While the first issue can be solved easily by a couple of people in a
>>>>>> few days, the second one is more tricky, and requires hard coding skills.
>>>>>> In addition, we'll no longer be able to edit GRASS vectors directly.
>>>>>>
>>>>>> In the second case, we'll have more work, and a not-so-nice duplication.
>>>>>>
>>>>>> I would like to have an open discussion on this, avoiding things to just
>>>>>> happen, with the possible negative consequences.
>>>>>>
>>>>>> All the best.
>>>>>> --
>>>>>> Paolo Cavallini - www.faunalia.eu
>>>>>> QGIS & PostGIS courses: http://www.faunalia.eu/training.html
>>>>>> _______________________________________________
>>>>>> grass-dev mailing list
>>>>>> grass-dev at lists.osgeo.org
>>>>>> http://lists.osgeo.org/mailman/listinfo/grass-dev
>>>>> _______________________________________________
>>>>> Qgis-developer mailing list
>>>>> Qgis-developer at lists.osgeo.org
>>>>> http://lists.osgeo.org/mailman/listinfo/qgis-developer