[Qgis-user] Raster Math on very very large files

[Paulo van Breugel]

Another option you may want to check out is GRASS GIS 7.0 (the RC1 just got
released a day ago, but it is rock solid, very fast especially on rasters,
and able to deal with very large raster layers. It has a bit of a learning
curve, but once you get the basics it offers easy scripting in shell or
python. And you can access it through QGIS (processing toolbox).


On Fri Jan 16 2015 at 6:12:52 AM Prasun Kumar Gupta <prasunkgupta at gmail.com>
wrote:

> Hi Nicolas,
> I work with lots of *very* large rasters.
> Here are my thoughts on doing geoprocessing with them on Windows
> environment.
> The steps are in increasing order of speed and user/coding complexity.
>
> 1. QGIS front-end generally tends to hang when dealing with large rasters.
> R is said to be slow [1]. [1] also suggests that you use ORFEO's toolbox
> [4]
> It is recommended that you try the Python program gdal_calc.py
> You may explore compression options [2]
>
> 2. I would personally opt for the IDL [3] programming language. The
> compiler is, however, not free.
> It is really fast and powerful for array intensive operations.
>
> 3. Multi-threaded / Parallel / GPU programming may help. However it has a
> steep learning curve (atleast for me).
> This presentation [5] in FOSS4G PDX 2014 on "Fast Big Data? A
> High-Performance System for Creating Global Satellite Image Time Series"
> was very informative and used HDF data cubes and GDAL to solve the problem.
>
> I hope you have got some useful pointers.
>
> Regards,
> Prasun
>
>
> [1]
> http://gis.stackexchange.com/questions/88996/per-pixel-statistical-calculations-on-a-raster-stack-using-gdal
> [2]
> http://gis.stackexchange.com/questions/69462/gdal-calc-py-outputs-huge-files
> [3] https://en.wikipedia.org/wiki/IDL_%28programming_language%29
> [4] http://orfeo-toolbox.org/Applications/BandMath.html
> [5] http://vimeo.com/107532162
>
>
> On Thu, Jan 15, 2015 at 9:38 PM, Nicolas Cadieux <
> nicolas.cadieux at archeotec.ca> wrote:
>
>> Hi,
>>
>> In the next few weeks, I will be trying to do some Raster Math on some
>> very large Rasters.  Rasters will likely be too large for the system memory
>> (72Go).  I will have a PCIe solid state drive to help out.
>>
>> Does anyone know how QGIS will manage?  Can we use the raster calculator
>> without loading the files into memory? Can the Python api solve this
>> problem?
>>
>> Doing the math on subsets of the data could help but I still need to
>> process the entire extent at some point to get to the raster statistics.
>>
>> I know I can use the R Raster package to do it but that package is not
>> flexible when dealing with grids with different resolutions.
>>
>> Any thoughts on this would be appreciated.
>>
>> Thanks
>>
>> Nicolas Cadieux M.Sc.
>> Les Entreprises Archéotec inc.
>> 8548, rue Saint-Denis Montréal H2P 2H2
>> Téléphone: 514.381.5112  Fax: 514.381.4995
>> _______________________________________________
>> Qgis-user mailing list
>> Qgis-user at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/qgis-user
>
>
> _______________________________________________
> Qgis-user mailing list
> Qgis-user at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/qgis-user
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-user/attachments/20150116/4d9b647a/attachment.html>