Fwd: [GRASS-dev] grass 7 and pixel random access

[Yann Chemin]

2008/4/28 Yann Chemin <yann.chemin at gmail.com>:
> We have experimented a bit with parallel processing, and for a given
>  processor power, there is a minimum amount of operations on a pixel
>  that needs to be done before there is any time benefit of using
>  parallelization. I would also believe that multi-core parallelization
>  (OpenMP: easy to code in most of the case) would benefit earlier that
>  multi-cpus (MPICH: requires some stronger preparation about coding),
>  for the physical reason of transport distance of bits.
>
>  I honestly would not know at which "level" we should integrate
>  parallelization capacities, it would be great indeed to have the
>  library understand a direct row loop and choose to split the loop by
>  the number of core/cpus available, if we speak about direct raster
>  processing. This would mean, potentially every direct raster
>  processing module could benefit, if they actually need it (which is
>  the main question, actually).
>
>  Now many processing in raster (and vector) may now be parallelized
>  straight-forwardly. And this is the main thing, we often face complex
>  algorithms, those ones are connecting pixels together from places to
>  places in the map (i.e. interpolation), those required taylor-made
>  parallelization. Here we should need tools to ease the task of
>  parallelization of those complex systems.
>
>  Finally, the codes I am working on are mostly pixel-based, either
>  temporal signal processing, energy-balance, adata assimilation of 1-D
>  vertical models, some take minutes, some hours/days, and some may go
>  up to a month (i dont run those last ones, I am waiting for CPU
>  improvement). So I would benefit directly by anything even just
>  running half of the raster on each core of my dual-core laptop/desktop.
>
>  Well, I am sure there are more experienced people on this list about
>  parallelization, so I am waiting for comments.
>
>  Yann
>
>  2008/4/28 Hamish <hamish_b at yahoo.com>:
>
>
> > > Yann Chemin wrote:
>  >  > >  I would like to know the planned changes for the raster library,
>  >  > >  especially the random access of pixels in the raster.
>  >  Markus:
>  >
>  > > Not sure if all of those are actually planned, but here is a list:
>  >  > http://grass.osgeo.org/wiki/GRASS_7_ideas_collection#Raster
>  >  >
>  >  > >  I wanted to work on it some months back, but my daily job got more
>  >  > >  intense.
>  >  > >  In the coming future, we will need to access easily any row for
>  >  > >  parallel processing.
>  >
>  >
>  >  One thing I wonder about for parallel processing of (serial) raster
>  >  modules- do we really need random read access to send each individual row
>  >  into a separate thread? The overhead with that seems counter-productive.
>  >  Couldn't we read some GRASS_NPROC envrio variable and then split the
>  >  overall number of rows by that number and create a small number of
>  >  threads, ie matching the system.
>  >
>  >
>  >  like: ceil(rows/nproc)
>  >  nproc=4
>  >  rows=1035
>  >  chunk size=259
>  >
>  >
>  >  the last thread reads/processes/writes fewer rows than the others as it
>  >  finds the EOF.
>  >
>  >
>  >
>  >  another thing I still wonder about (see thread from a month or so back)
>  >  is  where to start? Modify the libs to support the concept, then tackle
>  >  each module on their own? ie concentrate on the non-I/O limited and
>  >  can't-do- much-about-it but throw more processor at the problem modules,
>  >  and leave non-number crunching modules alone? -- concentrate on areas
>  >  where we'll get the most bang for the buck / pick off low hanging fruit /
>  >  etc?
>  >
>  >
>  >  Hamish
>  >  (showing off his multi-proc naivet'e)
>  >
>  >
>  >
>  >
>  >       ____________________________________________________________________________________
>  >  Be a better friend, newshound, and
>  >  know-it-all with Yahoo! Mobile.  Try it now.  http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ
>  >
>  >
>  >
>  >  _______________________________________________
>  >  grass-dev mailing list
>  >  grass-dev at lists.osgeo.org
>  >  http://lists.osgeo.org/mailman/listinfo/grass-dev
>  >
>
>
>
>
>
> --
>  Yann Chemin
>  International Rice Research Institute
>  Office: http://www.irri.org/gis
>  Perso: http://www.freewebs.com/ychemin
>



-- 
Yann Chemin
International Rice Research Institute
Office: http://www.irri.org/gis
Perso: http://www.freewebs.com/ychemin