[gdal-dev] GDAL raster processing library?

Mon Aug 8 08:18:34 PDT 2016

Hi Ari,
I began some work to clarify my ideas here: www.github.com/JamesRamm/GeoAlg
perhaps there is potential for merging with your project? For
neighbourhoods I provide 2 iterators - one simple block based and one
'buffered'. The block can be the natural block or user defined. I also
planned to provide a 'mosaic' iterator to support the planned 'reduce'
function. For callbacks I initially planned a simple function, but then
realised that many algorithms may wish to save state, so I changed to a
simple interface class where there is a single function to override. My
final plan was to expose opencv and other functionality (eg maybe taudem)
as a library of ready made callbacks.

If you think it worthwhile and possible, I would be open to porting
relevant parts to your code. Features I see as absolutely essential are:
- 'map' and 'reduce' functions for raster datasets, which handle large data
and user defined callbacks
- ability to save state in callbacks. This is essential for more complex
algorithms.
- A rich set of block iterators

A ready made library of a number of callbacks (preferably by exposing
functionality from existing well known libraries) would also be useful and
help build interest I think.

I will be in Bonn at foss4g from the Tuesday evening, it would be good to
meet!

On 8 Aug 2016 1:56 p.m., "Ari Jolma-2 [via OSGeo.org]" <
ml-node+s1560n5279906h35 at n6.nabble.com> wrote:

> Hi,
>
> Sorry for missing this last month due to holidays.
>
> In fact in my approach I do not want to limit to single pixel operations
> but support neighborhood operations too. That is necessary for watershed
> analysis for example. Making sure that the neighborhood is readily
> available for computations is probably what is causing the biggest problem
> for the code. I also have a callback support. See for example
> https://github.com/ajolma/gdal/blob/trunk/gdal/map_algebra/test.cpp where
> on the lines 5 to 12 is a callback function.
>
> When I left the code for my extended holidays I was doing profiling and
> intended to test another approach for caching. Now the caching is based on
> the native GDAL blocks, which is efficient for going through a single
> dataset. Since the block may be different for different datasets this
> approach may not be the best when there are more than one dataset. For
> example the raster algebra in QGIS uses an approach where the block is the
> same for all datasets. That simplifies the code and may be very efficient
> as a whole.
>
> I'll be at the code sprint in FOSS4G 2016 and that would be a good place
> to discuss the RFC too.
>
> Best,
>
> Ari
>
> 13.07.2016, 10:09, James Ramm kirjoitti:
>
> Peter,
> I think 'Grid Algebra' would be what Ari Jolma is proposing here:
> https://trac.osgeo.org/gdal/wiki/rfc62_raster_algebra
>
> As Even pointed out, there is some overlap, though my proposal is
> technically very different.
> The key differences I see are:
>
> - Users can submit functions which operate on each sub window of the
> raster, rather than an algebraic expression. This potentially allows for
> much more complicated algorithms to be used (e.g. I dont think it would be
> possible to run a watershed segmentation with a raster algebra
> implementation, or to have algorithms which behave differently depending on
> the 'location' within the raster or the values of surrounding pixels etc
> etc).
>
> - Functions can be chained together for a complex processing toolchain.
> Some overlap with VRT here, although again this introduces a little more
> flexibility.
>
> On 12 July 2016 at 15:47, Peter Halls <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=5279906&i=0>> wrote:
>
>> James,
>>
>>       in reality, are you not requesting an implementation of Tomlin's
>> 'Grid Algebra' in GDAL?  That defines the whole range of functions from
>> whole raster to pixel and has the distinct advantage of being both
>> published and extremely well known because of other implementations ...
>> which also provide ready-made reference bases for the GDAL implementors ...
>>
>> Best wishes,
>> Peter
>>
>> On 12 July 2016 at 15:39, James Ramm <[hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=5279906&i=1>> wrote:
>>
>>> Hi Even,
>>>
>>> The difference I see with pixel functions is that, as far as I
>>> understand, the pixel function is applied per pixel, so there is no
>>> possibility of e.g. the pixel buffer when have the function apply to
>>> 'blocks'.
>>> I may be way off, but many of the algorithms we deal with require some
>>> kind of neighbourhood search - a polygonise algorithm or flow direction
>>> algorithm being good examples.
>>> I dont think VRT pixel functions allow this?
>>>
>>> So in that sense I'd see a VRT being 'just' another potential input data
>>> source.
>>>
>>> Perhaps VRT pixel functions could be expanded to also allow 'window'
>>> functions?
>>>
>>> A downside is it requires creating a VRT even when you only want to
>>> apply a such a function to a single dataset. Small effort, but still a bit
>>> more than throwing in any GDALDataset to be processed.
>>>
>>> I see the overlap with raster algebra, although yes technically very
>>> different.
>>>
>>>
>>>
>>> On 12 July 2016 at 14:55, Even Rouault <[hidden email]
>>> <http:///user/SendEmail.jtp?type=node&node=5279906&i=2>> wrote:
>>>
>>>> James,
>>>>
>>>> There's some intersection with Ari's proposal :
>>>> https://trac.osgeo.org/gdal/wiki/rfc62_raster_algebra . At least
>>>> regarding the
>>>> overall purposes, since technically this is quite different.
>>>>
>>>> Actually what you propose is very close to the existing VRT pixel
>>>> functions of
>>>> derived bands. See "Using Derived Bands" in
>>>> http://www.gdal.org/gdal_vrttut.html . In the last days, we've merged
>>>> Antonio's work regarding a predefined set of pixel functions.
>>>> Perhaps some extension to allow passing user parameters to the pixel
>>>> func
>>>> could be useful. It is possible to use pixel functions from Python as
>>>> shown in
>>>> https://github.com/OSGeo/gdal/blob/trunk/autotest/gcore/
>>>> testnonboundtoswig.py#L303
>>>> although this is a bit ugly as it uses ctypes and not SWIG. But should
>>>> be
>>>> possible through SWIG by introducing proper types similarly to what is
>>>> done
>>>> for the progress functions or error handler functions.
>>>>
>>>> Even
>>>>
>>>>
>
> _______________________________________________
> gdal-dev mailing list
> [hidden email] <http:///user/SendEmail.jtp?type=node&node=5279906&i=3>
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://osgeo-org.1560.x6.nabble.com/GDAL-raster-processing-library-
> tp5275948p5279906.html
> To unsubscribe from GDAL raster processing library?, click here
> <http://osgeo-org.1560.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5275948&code=amFtZXNzcmFtbUBnbWFpbC5jb218NTI3NTk0OHwyMTIyMDM1OTQ5>
> .
> NAML
> <http://osgeo-org.1560.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>

--
View this message in context: http://osgeo-org.1560.x6.nabble.com/GDAL-raster-processing-library-tp5275948p5279934.html
Sent from the GDAL - Dev mailing list archive at Nabble.com.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20160808/b6638e0c/attachment.html>