[gdal-dev] gdal_rasterize in parallel processing

[Even Rouault]

Selon Giuseppe Amatulli <giuseppe.amatulli at gmail.com>:

> Hi all,
> I solved the down reported issues. I change -ot Float32  to -ot
> Float64 and it works.
> Anyway Float32 should be enough to store the max value of
> 560459513003129 < 3.4E38
> but probably during the multi-core several values are stored in the
> same array and it needs Float64.

I'm extremly skeptical that what you have seen initially is linked to
multi-processing and I would have recommanded you to double check your findings
that the results with -P 1 or -P 10 were really different. From what I've seen,
the -P option of xargs forks separate processes, so I can't imagine how they
could interact badly (this is different from having several threads inside a
single process).

But what happens likely with Float32 is that the precision isn't sufficient to
keep all the digits of the integral value. So 5000000000001 and 5000000000002
might be represented as the same Float32 binary value (let's say 5000000000000),
hence the impression that they are burnt with the same value. Float64 will give
you more precision (and more range, but that's not what you need).

>
>
> ciao ciao
> Giuseppe
>
> On 24 September 2012 17:53, Giuseppe Amatulli
> <giuseppe.amatulli at gmail.com> wrote:
> > Hi,
> > i'm running  gdal_rasterize for  several *.shp using xargs, sending
> > the process to 10 cpu.
> >
> > 1) ls /weldgfs/p51/gius_urban/pop_urban/2010/shp_albers/*.shp   |
> > xargs -n 1 -P 10  bash
> > /weldgfs/p51/gius_urban/pop_urban/scripts/sc2a_shp_rasterizeP30_2010.sh
> >
> > the same process can be run using one processor.
> >
> > 2) ls /weldgfs/p51/gius_urban/pop_urban/2010/shp_albers/*.shp   |
> > xargs -n 1 -P 1  bash
> > /weldgfs/p51/gius_urban/pop_urban/scripts/sc2a_shp_rasterizeP30_2010.sh
> >
> > Inputs and outputs are independent in each process so no issues in
> > reading/writing the same file.
> >
> > The obtained tifs  produced by the gdal_rasterize
> > (gdal_rasterize -a_nodata 0 -ot Float32 -co 'COMPRESS=LZW'  -te
> > $geo_string_tif   -tr 30 30    -l $filename     -a BLOCKID10
> > $INDIR/$filename.shp   $OUTDIR/$filename"_"$tile".tif"  ;
> >  where BLOCKID10 is an integer number if  e.g 550 019 505 012 101 )
> >
> > should be same but is not the case.
> > Line 2) produce the tif perfect similar with the attribute of the shp;
> > so no issues.
> > Line 1) burn the same BLOCKID10 value for several polygons, resulting
> > in less final polygons.
> >            Seems to me that this phenomena manly effect BLOCKID10
> > values having several "0" (e.g. 10979900000033 ; 10030101002035)
> >
> > May question is. Can be possible that the  gdal driver used inside C
> > code not support multi-core processing?
> > If yes it is possible to find a solution to run gdal_rasterize in parallel.
> > Any idea?
> >
> > Thanks in advance
> > Giuseppe
> >
> > --
> > Web: www.spatial-ecology.net
>
>
>
> --
> Giuseppe Amatulli
> Web: www.spatial-ecology.net
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>