[Qgis-developer] [GRASS-dev] GRASS & QGIS: the future

Wed Apr 23 05:50:30 PDT 2014

On Tue, Apr 22, 2014 at 4:49 PM, Sören Gebbert
<soerengebbert at googlemail.com> wrote:
>>> IMHO we GRASS developers are  too stubborn to change the design of
>>> GRASS GIS to work as library with persistent applications. This would
>>> require rewriting plenty of core functionality and modification of all
>>> C-modules. But there is a solution for persistent applications, called
>>> Remote Procedure Call (RPC), more below.
>>
>> It sounds interesting but it seems to be a huge work an far future. I
>
> Its far less work then modifying GRASS libraries to run with
> persistent applications.

We have already the providers and the plugin working with GRASS 6, I
just want to upgrade it to GRASS 7. The only problem I found so far is
that G__get_window() and Rast_get_cellhd() call G_fatal_error(). I can
only dedicate few hours/max days of my free time to that, so I have to
consider well what is less work, RPC is not.

> We need
> such an approach anyway, since vector editing is a GUI task -> GUI is
> a persistent
> application -> we need an exit safe interface to the GRASS libraries
> to implement a reliable
> vector editing GUI in GRASS itself.

Vector lib does not call G_fatal_error(), it was written in parallel
with QGIS plugin with vector editing in mind. I hope it is still true.

>> am also concerned about complexity, e.g. starting / keeping running
>> servers on various platforms. I can imagine endless bug reports about
>> "server not running". I am looking for something which will not
>
> Since only QGIS is spawning the "server" process, it is always able to
> check if the
> process is running and can restart it when needed. We already make use
> of this concept in GRASS
> and it works nicely[1]. Spawning of processes and keeping track of
> them is an easy task with Qt,
> and it is completely OS independent. The Qt Bitcoin client for example
> makes use of RPC
> and runs on Windows, Mac OS and various Linux systems. So i don't see
> a problem here.

The raster provider is already using something like that. Raster data
are read from executed GRASS module output (a module written for that
in QGIS in fact) and identify tool keeps another GRASS module running
and it is communicating with that through a  pipe. I wrote that as an
experiment and because I know, how much unnecessary additional code
and potential problems it is, I don't want to continue in that
direction.

>> require permanent assistance. It is not KISS enough for me.
>
> What would be a better KISS solution?
>
> IMO the RPC approach will make the implementation of the QGIS-GRASS
> data provider and the
> plugin simpler. There will be no mixing of C++ and C-code anymore.

We are no starting the plugin from scratch, it is already written, we
just need to make it work with GRASS 7.

> No setjmp/longjmp/exception workaround that is really complex

Why?

> and will not prevent QGIS from
> terminating or segfaulting eventually in case of a fatal error.

Why?

Radim

> The QGIS GRASS data provider/plugin will be a clean C++ implementation
> using the thrift library.
> It will be better maintainable since developers do not need to know
> all the different
> GRASS libraries (gis, raster, vector, database) but only the RPC
> interface that make
> use of a subset of the GRASS library functions, bundling them in an
> object oriented API.
> QGIS do not need to link against GRASS libraries anymore, future GRASS
> versions will be supported out of the box.
>
> This is much more KISS then the current approach.
>
> Best regards
> Soeren
>
> [1] http://grass.osgeo.org/programming7/namespacepython_1_1temporal_1_1c__libraries__interface.html
>>
>> Radim
>>
>>> 2014-04-18 11:31 GMT+02:00 Radim Blazek <radim.blazek at gmail.com>:
>>>> I have upgraded  the vector provider to GRASS 7, layers may be added
>>>> by drag from browser. The raster and the plugin are disabled. Be
>>>> careful about multiple versions on the same system
>>>> (LD_LIBRARY_PATH..., check with ldd if does not work).
>>>>
>>>> Unfortunately GRASS 7 moved ahead towards its aim "to make life harder
>>>> for anyone trying to use the GRASS libraries" [1]. Basically more and
>>>> more low level functions are calling exit() instead of returning error
>>>> code if something failed. As I am not willing to implement GRASS
>>>> module call for each simple function, we have to think again about
>>>> hacks we can use:
>>>>
>>>> 1) add a requirement that GRASS 7 used with QGIS must be compiled with
>>>> -fexceptions
>>>>
>>>> 2) add a requirement that a patch (it is a single line comment in
>>>> fact) must be applied to GRASS 7 to make it usable with QGIS
>>>>
>>>> 3) use setjmp()/longjmp()
>>>>
>>>> 4) let QGIS crash whenever GRASS lib function fails
>>>
>>> I fear that none of these suggestions are good working solutions,
>>> since in case a fatal error or a segfault occurs nothing (no
>>> exception, no setjmp) will prevent QGIS from crashing. So i would like
>>> to suggest the following:
>>>
>>> 5.) Using a RPC interface for map metadata, vector and raster map
>>> access. Hence one or several GRASS "server" processes provide an RPC
>>> interface to access GRASS library functions that are needed in a
>>> persistent application. Most important is the vector editing, vector
>>> reading/writing, database access, raster reading and map metadata
>>> support. Everything else can be done using modules. Hence the RPC
>>> interface will support only a limited subset of the GRASS library
>>> functions. This RPC interface should only be used in the GRASS
>>> provider classes, the GRASS plugin itself can be written in C++ or
>>> Python.
>>>
>>> This approach will decouple GRASS from QGIS, since all communication
>>> is done via Inter Process Communication (IPC) using pipes or sockets.
>>> There is no need anymore to catch a fatal error or to link GRASS
>>> libraries directly to QGIS. The GRASS plugin can be implemented GRASS
>>> version independently and so the GRASS data provider, since they will
>>> not use GRASS functions directly only the RPC interface. The RPC
>>> implementation on the side of GRASS will provide a consistent
>>> interface that will not change in case the underlying GRASS API
>>> changes.
>>>
>>> I strongly suggest to use an existing RPC framework to implement fast
>>> binary data exchange and the IPC. My favorite is apache thrift[1],
>>> since it supports plenty of programming languages (C/C++, Python,
>>> Java, JavaScript, ...) and provides operating system independent
>>> client and server functionality. It supports exceptions on client side
>>> that can be emitted in case a GRASS server process died because of a
>>> fatal error, SIGINT or segfault.
>>>
>>> I am absolutely willing to implement the RPC server side in GRASS
>>> using thrift and C++, providing plenty of Python unit tests that show
>>> howto use it on the client side. We just need to decide what kind of
>>> GRASS library functionality is needed as RPC interface and what can be
>>> done in QGIS with C++/Python directly (gisrc creation, GRASS
>>> environmental variable settings, ...) or using GRASS modules
>>> (g.region, g.gisenv, ...).
>>>
>>> [1] https://thrift.apache.org/
>>>
>>>
>>> Best regards
>>> Soeren
>>>
>>>>
>>>> Radim
>>>>
>>>> [1]https://trac.osgeo.org/grass/ticket/869#comment:1
>>>>
>>>> On Thu, Mar 27, 2014 at 11:18 AM, Paolo Cavallini <cavallini at faunalia.it> wrote:
>>>>> Hi all.
>>>>> I learned during dinner that GRASS7 RC1 is due very soon. This opens the
>>>>> issue of its functioning in QGIS. IMHO:
>>>>>
>>>>> * the qgis-grass-plugin might stop working (this has to be tested)
>>>>> * some of the module options will be different
>>>>> * new modules will not be available in QGIS.
>>>>>
>>>>> I think we can deal with this in several ways:
>>>>>
>>>>> * dropping the plugin, and concentrate the work on Processing
>>>>> * upgrading both the plugin and Processing.
>>>>>
>>>>> In the first case, we have two major issues:
>>>>>
>>>>> * upgrading Processing GRASS modules
>>>>> * changing the current Processing behaviour, avoiding the import-export
>>>>> phase when piping consecutive GRASS commands; this makes GRASS modules
>>>>> slower than the equivalent commands of other backends.
>>>>>
>>>>> While the first issue can be solved easily by a couple of people in a
>>>>> few days, the second one is more tricky, and requires hard coding skills.
>>>>> In addition, we'll no longer be able to edit GRASS vectors directly.
>>>>>
>>>>> In the second case, we'll have more work, and a not-so-nice duplication.
>>>>>
>>>>> I would like to have an open discussion on this, avoiding things to just
>>>>> happen, with the possible negative consequences.
>>>>>
>>>>> All the best.
>>>>> --
>>>>> Paolo Cavallini - www.faunalia.eu
>>>>> QGIS & PostGIS courses: http://www.faunalia.eu/training.html
>>>>> _______________________________________________
>>>>> grass-dev mailing list
>>>>> grass-dev at lists.osgeo.org
>>>>> http://lists.osgeo.org/mailman/listinfo/grass-dev
>>>> _______________________________________________
>>>> Qgis-developer mailing list
>>>> Qgis-developer at lists.osgeo.org
>>>> http://lists.osgeo.org/mailman/listinfo/qgis-developer