[gdal-dev] VRT derived band pixel functions written in Python

[Rutger]

I overlooked the fact that it still moves through Python, is that the 'only'
hurdle preventing parallel IO? Since gdalwarp for example has the -multi
flag, it seems as if GDAL is capable of it, or is that a
specific/specialized implementation?

Numba has several options which might eliminate using Python during
execution. There are c-callbacks:
http://numba.pydata.org/numba-doc/dev/user/cfunc.html

It probably works by using the Numpy C-API (which i have zero experience
with). I don't know if its possible that other programs, like GDAL, can also
use those compiled functions without moving to Python first.

There is also ahead-of-time compilation (AOT):
http://numba.pydata.org/numba-doc/dev/user/pycc.html

AOT has the benefit that users only need Numpy as a dependency, and don't
need Numba/llvm. There some drawbacks as well, like no longer being able to
compile optimizations for the hardware its running on.


Regards,
Rutger


Even Rouault-2 wrote
> Le mardi 13 septembre 2016 09:02:09, Rutger a écrit :
> 
> I've only scratched up the surface of Numba (didn't know it 2 days ago). I 
> guess this might work (might only be interested if the 
> VRTDerivedRasterBand::IRasterIO() is called with a big enough region.
> Which 
> depends on the pixel access pattern of upper layers). The VRT drivers just 
> calls a Python function that takes numpy arrays and a few extra args.
> 
>> And would that give parallel
>> processing for both IO and calculations?
> 
> Only calculations. I/O is done before going to Python and after returning
> from 
> it.
> 
> Actually... if you specify zero source in the VRT, then it is up to you to
> do 
> the read operations the way you like in Python, so you could possibly 
> parallelize them there.
> 
> 
> Even
> 
> -- 
> Spatialys - Geospatial professional services
> http://www.spatialys.com
> _______________________________________________
> gdal-dev mailing list

> gdal-dev at .osgeo

> http://lists.osgeo.org/mailman/listinfo/gdal-dev





--
View this message in context: http://osgeo-org.1560.x6.nabble.com/gdal-dev-VRT-derived-band-pixel-functions-written-in-Python-tp5285323p5285482.html
Sent from the GDAL - Dev mailing list archive at Nabble.com.