[gdal-dev] Raster calculation optimalisation

[Damian Dixon]

Personally I would agree with your approach at looking at the cost of
running the
formula first.


However if you do change your code to read/write blocks than there are
additional
opportunities for speeding the processing up.

Once you have the block size reading working consider using multiple
threads.
You will need to use a GDALDataset for each thread.

You will however need to consider how you write the results out again
as you will not be able to write to a single output. You will have to write
to
a separate raster for each thread.

This is the approach I have taken.

I can't help you with writing the block back as once I've got the raster
data I
processes it into a format that allows me to load and display the data a lot
quicker than going back to GDAL for the data.

You need to find the equivalent to:

http://www.gdal.org/classGDALRasterBand.html#a30786c81246455321e96d73047b8edf1
Also these may help...

http://www.gdal.org/classGDALRasterBand.html#ad80cecc562fd48f5783677de625360ac

http://www.gdal.org/classGDALRasterBand.html#a457ae6645dd2680289734aed0afce357



On 15 June 2017 at 14:04, Paul Meems <bontepaarden at gmail.com> wrote:

> Thanks all for your suggestions.
>
> @Rutger and @Damian:
> Thanks for your suggestion about the blocks.
> I had a look at Rutger's links,
> I create the input file myself so I can add 'TILES=YES' but I'm not sure
> how to change my calculation code.
> I see in this first link xbs, ybs = b1.GetBlockSize() But I don't see
> when to use the xbs or ybs variables.
> I assume I need to change the reading of the data: band.ReadRaster(0,
> line, sizeX, 1, scanline, sizeX, 1, 0, 0); but I'm not sure how.
> And how should I write back the block?
>
> BTW. The main bottle-neck seems to be in the used formula parser.
> With the formula it takes 68s, without the formule - just setting the
> pixelvalue takes 3.3s and with the formula written in code pixelValue =
> (float)Math.Exp(3.1 + 0.9 * f); it only takes 3.4s.
> So not using mathparser has the highest benefits. I will do that first,
> but I also want to understand the block reading and writing.
>
> @Jan, thanks for your link to the other parser. I had a quick look and it
> looks very promising. Sadly I couldn't get their C# example working.
> I will look at as well.
>
> Just a more general question.
> Doesn't it makes sense if gdal would provide a gdal_calculate tool which
> also is librified like VectorTranslate?
> It seems lots of people are implementing it on their own.
>
> Thanks,
>
> Paul
>
>
> 2017-06-14 9:29 GMT+02:00 Rutger <kassies at gmail.com>:
>
>> Damian Dixon wrote
>> > It is usually better to process the pixels in a block rather than
>> walking
>> > across each row especially if you are processing a TIFF as these are
>> > usually stored as tiles (blocks).
>>
>> Other layouts are common as well. For example, the Landsat TIFF's provided
>> by the USGS have a row based-layout.  If you can choose it yourself, bocks
>> are prefered in my opinion, since GDAL VRT's have a fixed blocksize
>> of128*128. So when writing TIFF's, setting "TILED=YES" is a good default.
>>
>> I think your spot on by mentioning the blocks. Don't assume the layout at
>> all, look at the blocksize of the file and use it. If the blocks are
>> relatively small (memoy-wise), using a multiple of the size can increase
>> performance a bit more. So if its row-based and you have plenty of memory
>> to
>> spare, why not read blocks of (xsize, 128). Or if the blocksize is
>> 128x128,
>> use blocks of 256x256 etc.
>>
>> If the volume of data is really large, increasing GDAL's block cache can
>> be
>> helpful. Although its best to avoid relying on the cache (if possible) by
>> specifying an appropriate blocksize.
>>
>> Here are a few mini-benchmarks:
>> http://nbviewer.jupyter.org/gist/RutgerK/4faa43873ee6389873dd4de85de7c450
>>
>> https://stackoverflow.com/questions/41742162/gdal-readasarra
>> y-for-vrt-extremely-slow/41853759#41853759
>>
>> Regards,
>> Rutger
>>
>>
>>
>>
>> --
>> View this message in context: http://osgeo-org.1560.x6.nabbl
>> e.com/gdal-dev-Raster-calculation-optimalisation-tp5324014p5324120.html
>> Sent from the GDAL - Dev mailing list archive at Nabble.com.
>> _______________________________________________
>> gdal-dev mailing list
>> gdal-dev at lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>
>
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20170615/9b016d87/attachment.html>