[gdal-dev] Raster calculation optimalisation

[Paul Meems]

Thanks all for your suggestions.

@Rutger and @Damian:
Thanks for your suggestion about the blocks.
I had a look at Rutger's links,
I create the input file myself so I can add 'TILES=YES' but I'm not sure
how to change my calculation code.
I see in this first link xbs, ybs = b1.GetBlockSize() But I don't see when
to use the xbs or ybs variables.
I assume I need to change the reading of the data: band.ReadRaster(0, line,
sizeX, 1, scanline, sizeX, 1, 0, 0); but I'm not sure how.
And how should I write back the block?

BTW. The main bottle-neck seems to be in the used formula parser.
With the formula it takes 68s, without the formule - just setting the
pixelvalue takes 3.3s and with the formula written in code pixelValue =
(float)Math.Exp(3.1 + 0.9 * f); it only takes 3.4s.
So not using mathparser has the highest benefits. I will do that first, but
I also want to understand the block reading and writing.

@Jan, thanks for your link to the other parser. I had a quick look and it
looks very promising. Sadly I couldn't get their C# example working.
I will look at as well.

Just a more general question.
Doesn't it makes sense if gdal would provide a gdal_calculate tool which
also is librified like VectorTranslate?
It seems lots of people are implementing it on their own.

Thanks,

Paul


2017-06-14 9:29 GMT+02:00 Rutger <kassies at gmail.com>:

> Damian Dixon wrote
> > It is usually better to process the pixels in a block rather than walking
> > across each row especially if you are processing a TIFF as these are
> > usually stored as tiles (blocks).
>
> Other layouts are common as well. For example, the Landsat TIFF's provided
> by the USGS have a row based-layout.  If you can choose it yourself, bocks
> are prefered in my opinion, since GDAL VRT's have a fixed blocksize
> of128*128. So when writing TIFF's, setting "TILED=YES" is a good default.
>
> I think your spot on by mentioning the blocks. Don't assume the layout at
> all, look at the blocksize of the file and use it. If the blocks are
> relatively small (memoy-wise), using a multiple of the size can increase
> performance a bit more. So if its row-based and you have plenty of memory
> to
> spare, why not read blocks of (xsize, 128). Or if the blocksize is 128x128,
> use blocks of 256x256 etc.
>
> If the volume of data is really large, increasing GDAL's block cache can be
> helpful. Although its best to avoid relying on the cache (if possible) by
> specifying an appropriate blocksize.
>
> Here are a few mini-benchmarks:
> http://nbviewer.jupyter.org/gist/RutgerK/4faa43873ee6389873dd4de85de7c450
>
> https://stackoverflow.com/questions/41742162/gdal-
> readasarray-for-vrt-extremely-slow/41853759#41853759
>
> Regards,
> Rutger
>
>
>
>
> --
> View this message in context: http://osgeo-org.1560.x6.
> nabble.com/gdal-dev-Raster-calculation-optimalisation-
> tp5324014p5324120.html
> Sent from the GDAL - Dev mailing list archive at Nabble.com.
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20170615/a622049b/attachment.html>