[Qgis-user] Raster Math on very very large files

[Prasun Kumar Gupta]

Hi Nicolas,
I work with lots of *very* large rasters.
Here are my thoughts on doing geoprocessing with them on Windows
environment.
The steps are in increasing order of speed and user/coding complexity.

1. QGIS front-end generally tends to hang when dealing with large rasters.
R is said to be slow [1]. [1] also suggests that you use ORFEO's toolbox [4]
It is recommended that you try the Python program gdal_calc.py
You may explore compression options [2]

2. I would personally opt for the IDL [3] programming language. The
compiler is, however, not free.
It is really fast and powerful for array intensive operations.

3. Multi-threaded / Parallel / GPU programming may help. However it has a
steep learning curve (atleast for me).
This presentation [5] in FOSS4G PDX 2014 on "Fast Big Data? A
High-Performance System for Creating Global Satellite Image Time Series"
was very informative and used HDF data cubes and GDAL to solve the problem.

I hope you have got some useful pointers.

Regards,
Prasun


[1]
http://gis.stackexchange.com/questions/88996/per-pixel-statistical-calculations-on-a-raster-stack-using-gdal
[2]
http://gis.stackexchange.com/questions/69462/gdal-calc-py-outputs-huge-files
[3] https://en.wikipedia.org/wiki/IDL_%28programming_language%29
[4] http://orfeo-toolbox.org/Applications/BandMath.html
[5] http://vimeo.com/107532162


On Thu, Jan 15, 2015 at 9:38 PM, Nicolas Cadieux <
nicolas.cadieux at archeotec.ca> wrote:

> Hi,
>
> In the next few weeks, I will be trying to do some Raster Math on some
> very large Rasters.  Rasters will likely be too large for the system memory
> (72Go).  I will have a PCIe solid state drive to help out.
>
> Does anyone know how QGIS will manage?  Can we use the raster calculator
> without loading the files into memory? Can the Python api solve this
> problem?
>
> Doing the math on subsets of the data could help but I still need to
> process the entire extent at some point to get to the raster statistics.
>
> I know I can use the R Raster package to do it but that package is not
> flexible when dealing with grids with different resolutions.
>
> Any thoughts on this would be appreciated.
>
> Thanks
>
> Nicolas Cadieux M.Sc.
> Les Entreprises Archéotec inc.
> 8548, rue Saint-Denis Montréal H2P 2H2
> Téléphone: 514.381.5112  Fax: 514.381.4995
> _______________________________________________
> Qgis-user mailing list
> Qgis-user at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/qgis-user
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-user/attachments/20150116/d015afb7/attachment.html>