[GRASS-dev] Adding an expert mode to the parser

[Sören Gebbert]

Hi,
Here an example of hidden region settings for modules utilized in the
temporal framework:

Imagine you have a Landsat location that contains many thousand scenes
(time series) from all over the world including all bands. Each band of the
time series is managed in a specific space-time dataset. You may want to
use the temporal raster algebra to compute the NDVI in parallel on all
scenes at once. This is not possible at the moment, because of the current
region settings. Except you can set the current region to cover the whole
world, which is not preferable.

So the idea is to have options that will set the computational region for
each process independently, when the process is invoked. Hence the raster
computational window is set in the parser, so that the usual region get
methods in the modules will work untouched. This avoids the current
approach that is covering a single module call (like r.mapclalc, r.series,
r.neighbors, ... ) in several g.region and g.remove calls in a dedicated
subprocess.

This option is only useful for parallel processing of raster layers with
different spatial extents. A special feature for the temporal framework,
that should not be exposed to all the users, only to those who implement
massive parallel working temporal modules. Hence using the same approach as
for WPS, GUI and HTML output:

r.mapclac --raster-region= --north= --south= --west= --east= --res=
--ewres= --nsres= --vect-region --raster-align= ...

Just my 2 cent
Sören

2016-09-25 20:49 GMT+01:00 Markus Neteler <neteler at osgeo.org>:

> On Fri, Sep 23, 2016 at 11:30 PM, Markus Metz
> <markus.metz.giswork at gmail.com> wrote:
> > On Fri, Sep 23, 2016 at 11:22 PM, Markus Neteler <neteler at osgeo.org>
> wrote:
> >> On Fri, Sep 23, 2016 at 11:05 PM, Markus Metz
> >> <markus.metz.giswork at gmail.com> wrote:
> >>> On Fri, Sep 23, 2016 at 1:11 PM, Markus Neteler <neteler at osgeo.org>
> wrote:
> >> ...
> >>> Your motivation is to provide a specialized CLI interface for HPC
> >>> processing?
> >>
> >> No, not the case.
> >>
> >>> We used GRASS with HPC processing for years and the
> >>> problems we faced were causes by the HPC hardware and software
> >>> infrastructure, not by GRASS. What exactly is the problem with using
> >>> GRASS and HPC processing?
> >>
> >> There is no problem. There is just the issue that with an increasing
> >> amount of additions (e.g. maybe the need to provide region/resolution
> >> to individual modules for independent parallel processing without the
> >> overhead of always opening a new mapset)
> >
> > Getting closer it seems. Can you quantify "the overhead of always
> > opening a new mapset"?
>
> As an example, when aiming at processing all Sentinel-2 tiles
> globally, we speak about currently 73000 scenes * up-to-16 tiles along
> with global data, analysis on top of other global data is more complex
> when doing each job in its own mapset and reintegrate it in a single
> target mapset as if able to process then in parallel in one mapset by
> simply specifying the respective region to the command of interest.
> Yes, different from the current paradigm and not for G7.
>
> But my original comment was targeted at the increasing number of
> module parameters and how to handle that (with some new HPC related
> idea as an example).
>
> I'm fine to archive this question for now, it will likely come up
> again in the future.
>
> markusN
> _______________________________________________
> grass-dev mailing list
> grass-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/grass-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-dev/attachments/20160925/d6e7073b/attachment.html>