[SoC] [GRASS-dev] Weekly report #3 - GRASS Interactive Scatter Plot Tool

[Sören Gebbert]

Hi Štěpán,
some notes to you implementation approach:

2013/7/6 Markus Metz <markus.metz.giswork at gmail.com>
>
> On Sat, Jul 6, 2013 at 4:19 PM, Štěpán Turek <stepan.turek at seznam.cz> wrote:
> > Hi Hamish and Michael,
> >
> > your questions are connected I will try to explain both.
> >
> > The scatter plot tool backend is run in same process as wxGUI. If you select
> > some area in scatter plot, then data from numpy arrays are passed to the
> > backend (through ctypes), which describes selected areas in scatters plots.
>
> You probably don't need ctypes for that, have a look at
> lib/python/script/array.py.
>
> If you want to use ctypes, keep in mind that GRASS is designed for
> modular use. Tools using library functions must run as a separate
> process, i.e. the wxGUI should run just fine also in the absence of
> ctypes.

I would avoid to use ctypes in the same process as wxGUI. The grass
libraries call exit() in case a fatal error occurs. This will also
crash the GUI in this case. I would strongly suggest to use
multiprocessing Python module to spawn new processes that make use of
ctypes. However, what is the reason that you use ctypes directly
instead of PyGRASS that was designed to provide an abstract layer
above several grass libraries? Maybe you can add an OO layer in
PyGRASS providing access to your backend?

> > And the backend computes corresponding areas which will be highlighted in
> > all opened scatter plots. To avoid frozing of the wxGUI, the computation is
> > done in separate python thread.

Unfortunately are python threads not a good idea if you use ctypes,
since in case of a fatal error it will crash the GUI. Besides of that
are Python threads not really parallel. [1]

[1] https://en.wikipedia.org/wiki/Global_Interpreter_Lock

> >
> > Currently r.out.bin is used to bypass inability to set region in raster
> > library for different rasters. When user choose raster group for analysis in
> > the tool, it is exported into binary files in current region and the backend
> > works directly with the binary files. This is not very nice.
> >
> > If the backend would be written as module, then I would need to create
> > temporary files to pass the selected areas, which does not seem very elegant
> > to me.

Please use the multiprocessing [2] Python module to spawn separate
processes and then use PyGRASS to set the process specific region and
to access the raster maps. With this approach each raster map can have
its own region. This will avoid GUI crash in case of a fatal error, it
will avoid the need to export raster maps or to create temporary
files.  The multiprocessing modules provides inter process
communication based on serialized Python objects. Hence it is very
simple an convenient to exchange Python objects between different
processes (using a queue object).

[2] http://docs.python.org/2/library/multiprocessing.html

>
>
> When processing raster data, temporary files are created pretty much
> all the time (by the raster lib). Regarding the scatterplot tool which
> works with raster data which can be very large, an advantage of
> temporary files is that out-of-memory errors are more easily avoided.
> >
> > In simplified way the idea is that the library should give you option
> > whether you want to open and then work with raster file according to the
> > statically set variables (in structure R__) or open it with dynamically
> > defined variables.
>
> With regard to the current region, it seems to me that this would
> violate the concept of the current region. Rather run the scatterplot
> tool as a separate process, then you can modify the region without
> changing the regular current region. When using raster lib functions
> directly, maybe Rast_set_input_window() can help you?

Please use PyGRASS in separate processes to access raster maps.

> > It could be also useful for Pietro's pyGRASS API. And it would be first
> > small step to achieve compilation of raster modules as library instead of
> > different programs. I am working on more concrete proposal of the changes in
> > raster library during this weekend.
>
> Raster modules, as all GRASS modules, should (IMHO must definitively)
> stay modules because GRASS is designed for modular use. Modules can
> easily call other modules, effectively using them instead of some,
> just like some library function. Again, the advantage is that modules
> are run as a separate process.

Absolutely. PyGRASS offers a very convenient way to call GRASS modules
of any kind (C/C++, Python) using a simple function.

Best regards and happy hacking :)
Soeren

>
> my 2c
>
> Markus M
> >
> > I am new to this problematic, so all these suggestions can be totally
> > wrong..
> >
> > Best
> > Stepan
> >
> > ---------- Původní zpráva ----------
> > Od: Michael Barton <Michael.Barton at asu.edu>
> > Datum: 6. 7. 2013
> > Předmět: Re: [SoC] Weekly report #3 - GRASS Interactive Scatter Plot Tool
> >
> >
> > So I take it you are not using r.stats to generate the data to plot, but are
> > getting data directly accessing rasters via GRASS libraries?
> >
> > Michael
> > ______________________________
> >
> >
> >
> > Hi,
> >
> > I am interested to hear more about the library limitations.. you
> > mean the WIND region, right? I'd be surprised if WIND_OVERRIDE or
> > GRASS_REGION enviro variables (and the python wrappers for them)
> > didn't provide full control for whatever someone might like to do.
> >
> > I would guess if the scatter tool was spawned as an independent
> > process, it could maintain its own environment & so it's own region
> > override enviro variables. or simply unset the wxGUI display overrides
> > and just use the mapset's real computational region read directly from
> > the WIND file?
> >
> >
> > regards,
> > Hamish
> >
> >
> >
> > _______________________________________________
> > grass-dev mailing list
> > grass-dev at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/grass-dev
> _______________________________________________
> SoC mailing list
> SoC at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/soc