[Qgis-developer] Provider native raster band stats

[Tim Sutton]

Hi

On Tue, Aug 2, 2011 at 7:06 PM, Etienne <etiennesky at yahoo.com> wrote:
> Tim, my impression is that it speeds things up, but I have a suggestion to make it faster.
>
Cool!

> From what I have seen in your code, the  QgsGdalProvider::bandStatistics function calls GDALComputeRasterStatistics (), which computes the statistics every time - it doesn't fetch pre-calculated data ( which is stored in gtiff metadata, or aux.xml file).
>
> A better way would be to use the GDALGetRasterStatistics() which calls GDALComputeRasterStatistics() when needed (when bForce = TRUE)
> double myerval =  GDALGetRasterStatistics ( myGdalBand, bApproxOK, TRUE, &pdfMin, &pdfMax, &pdfMean, &pdfStdDev);
>

Ah - actually I thought GDALComputeRasterStatistics was using aux.xml
cached stats already. I guess I never read the docs carefully enough.
I applied your change and my ~8min to load raster from my original
email took about a minute to load the first time (while gdal was
creating an aux.xml) and subsequent use of the same raster resulted in
it opening instantly! So thanks for your excellent advice. I have
pushed an updated version to my repo that incorporates your changes
should others like to try. I will try to write some unit tests this
week and then merge my branch into QGIS master over the weekend.

>
>
> I have tested it on a few rasters, it gets the same stats.
>
> The downside is that GDALGetRasterStatistics() does not offer a progress callback, rendering the progressbar useless.
> If it"s really needed, I could re-implement the function in  QgsGdalProvider (about 50 lines of code), or try to have GDALGetRasterStatistics support a progress callback.
>

Yes this is quite noticeable the first time the raster is loaded and
you sit wondering what is happening. I guess the more elegant would be
to have the change carried out in gdal itself if that is acceptible,
otherwise your patch as per your former suggestion would be greatly
appreciated.

Best regards

Tim

> regards, Etienne
>
> ----- Original Message -----
> From: Tim Sutton <lists at linfiniti.com>
> To: Marco Hugentobler <marco.hugentobler at sourcepole.ch>
> Cc: qgis-developer at lists.osgeo.org
> Sent: Monday, July 11, 2011 7:04:08 PM
> Subject: Re: [Qgis-developer] Provider native raster band stats
>
> Hi
>
>
> On Mon, Jul 11, 2011 at 5:29 PM, Marco Hugentobler
> <marco.hugentobler at sourcepole.ch> wrote:
>> Hi Tim
>>
>> I'm not a raster guru, but the changes look good to me.
>> +1 for merging.
>
> Thanks for your feedback Marco. I guess I'll write some unit tests
> (wrong way around I know :-) ) and then merge it if they all pass and
> there are no other objections. :-)
>
> Regards
>
> Tim
>
>
>>
>> Regards,
>> Marco
>>
>> Am Donnerstag, 7. Juli 2011, 00.47:25 schrieb Tim Sutton:
>>> Hi
>>>
>>> During the Lisbon hackfest (between all those meetings!) I worked on
>>> updating the raster providers to be able to generate stats themselves.
>>> The raster provider base class has a default (and historically
>>> inefficient as we all know) implementation for collecting stats by
>>> walking over every cell in the raster twice. This method remains, but
>>> the providers can now supply stats themselves (heopfully using more
>>> efficient mechanisms). I have implemented gdal native stats already.
>>> Over the last few weeks I did some further testing and cleanups to
>>> this work. From my testing some of my worst case files opened in
>>> ~1minute rather than ~8+ minutes. It would be great if interested
>>> parties could test (details below) before I put it into master.
>>>
>>> git remote add timlinux git://github.com/timlinux/Quantum-GIS.git
>>> git fetch timlinux
>>> git branch --track timlinux raster-stats
>>> git checkout raster-stats
>>>
>>> I look forward to your feedback.
>>>
>>> Regards
>>
>>
>> --
>> Dr. Marco Hugentobler
>> Sourcepole -  Linux & Open Source Solutions
>> Churerstrasse 22, CH-8808 Pfäffikon SZ, Switzerland
>> marco.hugentobler at sourcepole.ch http://www.sourcepole.ch
>> Technical Advisor QGIS Project Steering Committee
>>
>
>
>
> --
> Tim Sutton - QGIS Project Steering Committee Member (Release  Manager)
> ==============================================
> Please do not email me off-list with technical
> support questions. Using the lists will gain
> more exposure for your issues and the knowledge
> surrounding your issue will be shared with all.
>
> Visit http://linfiniti.com to find out about:
>  * QGIS programming and support services
>  * Mapserver and PostGIS based hosting plans
>  * FOSS Consulting Services
> Skype: timlinux
> Irc: timlinux on #qgis at freenode.net
> ==============================================
> _______________________________________________
> Qgis-developer mailing list
> Qgis-developer at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/qgis-developer
>
> _______________________________________________
> Qgis-developer mailing list
> Qgis-developer at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/qgis-developer
>



-- 
Tim Sutton - QGIS Project Steering Committee Member (Release  Manager)
==============================================
Please do not email me off-list with technical
support questions. Using the lists will gain
more exposure for your issues and the knowledge
surrounding your issue will be shared with all.

Visit http://linfiniti.com to find out about:
 * QGIS programming and support services
 * Mapserver and PostGIS based hosting plans
 * FOSS Consulting Services
Skype: timlinux
Irc: timlinux on #qgis at freenode.net
==============================================