[GRASS-dev] Compute mahalanobis distance using Scipy

[Paulo van Breugel]

Hi Javier,

Thanks. I need to test whether the mahalanobis function you provided is
faster than the one I used (which Glynn wrote, based on the numpy function
I believe); I think I can use both together with the function to run it in
parallel, so it would be a matter of selecting the fastest one (or the one
using less memory, if there is such a difference).

I am not sure if I understand the parallel part, but it seems the data is
'sliced' in subsets to compute the distance. If so, that should indeed
solve the problem with large files? Anyway, I'll try how it works for me.

For now I do the partitioning of the data in subsets in GRASS (create a
dummy layer, split that in tiles using r.tile, set the region to match tile
1, import the input layers in python (which will be restricted to the
region defined by tile 1), run the function and import the result back in
GRASS, set region to match tile 2, impor the data... etc). Much slower I
guess (not really tested yet), but should be able to handle any data size
that GRASS can handle.

To create a grass plugin to compute the mahalanobis distance based on that
code would be trivial I think, I can give that a try if you want. As for
the rest of the code, I would be interested to help, but my grass/python
programming are very limited, so hopefully there are others interested as
well.

Cheers,

Paulo



On Mon, Feb 16, 2015 at 11:35 AM, Javier Martínez-López <
javi.martinez.lopez at gmail.com> wrote:

> Hi Paulo,
>
> to use it with python just copy the mahalanobis function definitions,
> import all necessary libraries and use it as in line 597 (the
> covariance matrix and the mean are computed just before this line). I
> am sorry that the code is not documented yet. I am not sure if this
> will solve the large raster layers problem but you can try it. As you
> can see in the script, I did not write the parallel mahalanobis
> functions (Sturla Molden in CC did it) and it works perfectly for me,
> so I agree that it would be great to implement it also in GRASS (maybe
> as an add on?). However, I am not so experienced with that. The
> ehab.py script also does a lot data preprocessing to allow batch runs
> using heterogeneous data sources and finally performs some patch
> analysis and metrics out of the resulting similarity maps using the
> ndimage library (scipy). Any help on how to implement it in GRASS
> would be more than welcome! Cheers, Javier
>
> On Sun, Feb 15, 2015 at 10:21 PM, Paulo van Breugel
> <p.vanbreugel at gmail.com> wrote:
> > Hi Javier,
> >
> > This looks really useful, thanks! My python skills are very limited, so I
> > will have to look at it better to see what I can understand. But to
> start,
> > the parallel function, does that also solve the handling of potentially
> very
> > large raster layers?
> >
> > Btw, this does look like a very useful set of functions, any plan to
> develop
> > this into a GRASS addon or function?
> >
> > Paulo
> >
> >
> > On Sun, Feb 15, 2015 at 9:22 PM, Javier Martínez-López
> > <javi.martinez.lopez at gmail.com> wrote:
> >>
> >> Hi Paulo,
> >>
> >> you can see an implementation of the mahalanobis distance computed in
> >> parallel (using all computer processors) for a single image here:
> >>
> >> https://github.com/javimarlop/eHabpy/blob/master/ehab.py
> >>
> >> see lines 32-86 and 597. I hope it helps! Cheers, Javier
> >>
> >> On Sat, Feb 14, 2015 at 7:29 PM, Vaclav Petras <wenzeslaus at gmail.com>
> >> wrote:
> >> >
> >> > On Sat, Feb 14, 2015 at 11:47 AM, Paulo van Breugel
> >> > <p.vanbreugel at gmail.com>
> >> > wrote:
> >> >>
> >> >>
> >> >> For a quick solution, what about using r.tile to split the input data
> >> >> in
> >> >> tiles and compute the mahalanobis distance per tile.
> >> >
> >> >
> >> > See PyGRASS GridModule class which will do tiling (and a lot of other
> >> > things) for you.
> >> >
> >> >
> >> >
> http://grass.osgeo.org/grass70/manuals/libpython/pygrass.modules.grid.html
> >> >
> >> > _______________________________________________
> >> > grass-dev mailing list
> >> > grass-dev at lists.osgeo.org
> >> > http://lists.osgeo.org/mailman/listinfo/grass-dev
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-dev/attachments/20150216/bcad925b/attachment.html>