[gdal-dev] VRT derived band pixel functions written in Python

Sean Gillies sean at mapbox.com
Thu Sep 15 06:56:23 PDT 2016


Hi Even, James:

I suspect users will be find it very surprising if `which python` and the
python interpreter used by VRT are not the same. Imagine starting python in
an environment created by virtualenv or conda that contains extension
modules like scikit-image, scikit-learn, &c. You import osgeo.gdal, open
your super duper VRT with `gdal.Open()`, and get an error because your
environment's extension modules aren't available in the VRT script. This
could be tough to debug for a beginning to intermediate level Python
programmer.


On Thu, Sep 15, 2016 at 10:29 AM, James Ramm <jamessramm at gmail.com> wrote:

> excellent, working setting the PYTHONSO variable. Trying to match the
> version of python on the path would be smart, but it is probably 'good
> enough' to give PYTHONSO some prominence in the documentation - some kind
> of warning on how python is discovered perhaps?
>
> On 14 September 2016 at 20:20, James Ramm <jamessramm at gmail.com> wrote:
>
>> Ah, that makes sense. I'll have to try the config option in the morning,
>> but it sounds like that could be it
>>
>> On 14 Sep 2016 8:06 p.m., "Even Rouault" <even.rouault at spatialys.com>
>> wrote:
>>
>>> Le mercredi 14 septembre 2016 20:50:04, James Ramm a écrit :
>>> > Yes, it is loading a different python....The path printed by sys.path
>>> is
>>> > different to if I open the command window and type:
>>> >
>>> > python
>>> >
>>> > >>> import sys
>>> > >>> print(sys.path)
>>> >
>>> > Gdal and the python bindings were compiled with vs2015 and python 3.5,
>>> and
>>> > I can correctly import python in the 3.5 interpreter, yet somehow a
>>> > different python DLL (2.7) is being loaded at runtime.
>>>
>>> Might be related to the default try order in
>>>
>>>         const char* const apszPythonSO[] = { "python27.dll",
>>>                                             "python26.dll",
>>>                                             "python34.dll",
>>>                                             "python35.dll",
>>>                                             "python36.dll",
>>>                                             "python33.dll",
>>>                                             "python32.dll" };
>>>
>>> First found, first served.
>>>
>>> Hum maybe we should try to match the version that issuing python on the
>>> command line would start. We could potentially look at the PATH to see if
>>> there's something like "bla:\pythonXX" and try the related .dll... Or
>>> more
>>> costly, but more reliable, try issuing a 'python -c "import sys;
>>> print(str(sys.version_info[0]) + str(sys.version_info[1]))"' command
>>>
>>> That's one of the downside of requiring no dependency at build time.
>>>
>>> Anyway you can override the default guess by setting the PYTHONSO config
>>> option
>>> to point to the desired python dll.
>>>
>>> By the way I've committed the doc in
>>> https://trac.osgeo.org/gdal/changeset/35441 . Should be reflected
>>> online in a
>>> few hours.
>>>
>>>
> On 14 September 2016 at 20:20, James Ramm <jamessramm at gmail.com> wrote:
>
>> Ah, that makes sense. I'll have to try the config option in the morning,
>> but it sounds like that could be it
>>
>> On 14 Sep 2016 8:06 p.m., "Even Rouault" <even.rouault at spatialys.com>
>> wrote:
>>
>>> Le mercredi 14 septembre 2016 20:50:04, James Ramm a écrit :
>>> > Yes, it is loading a different python....The path printed by sys.path
>>> is
>>> > different to if I open the command window and type:
>>> >
>>> > python
>>> >
>>> > >>> import sys
>>> > >>> print(sys.path)
>>> >
>>> > Gdal and the python bindings were compiled with vs2015 and python 3.5,
>>> and
>>> > I can correctly import python in the 3.5 interpreter, yet somehow a
>>> > different python DLL (2.7) is being loaded at runtime.
>>>
>>> Might be related to the default try order in
>>>
>>>         const char* const apszPythonSO[] = { "python27.dll",
>>>                                             "python26.dll",
>>>                                             "python34.dll",
>>>                                             "python35.dll",
>>>                                             "python36.dll",
>>>                                             "python33.dll",
>>>                                             "python32.dll" };
>>>
>>> First found, first served.
>>>
>>> Hum maybe we should try to match the version that issuing python on the
>>> command line would start. We could potentially look at the PATH to see if
>>> there's something like "bla:\pythonXX" and try the related .dll... Or
>>> more
>>> costly, but more reliable, try issuing a 'python -c "import sys;
>>> print(str(sys.version_info[0]) + str(sys.version_info[1]))"' command
>>>
>>> That's one of the downside of requiring no dependency at build time.
>>>
>>> Anyway you can override the default guess by setting the PYTHONSO config
>>> option
>>> to point to the desired python dll.
>>>
>>> By the way I've committed the doc in
>>> https://trac.osgeo.org/gdal/changeset/35441 . Should be reflected
>>> online in a
>>> few hours.
>>>
>>>
>>> > I am on a
>>> > 'inherited' PC right now, so final thing to do is to ensure that the
>>> > gdal_translate I am running is the one I compiled and there isn't
>>> another
>>> > version lurking somewhere....
>>> >
>>> >
>>> > On 14 September 2016 at 16:50, Even Rouault <
>>> even.rouault at spatialys.com>
>>> >
>>> > wrote:
>>> > > Le mercredi 14 septembre 2016 17:24:53, jramm a écrit :
>>> > > > Trying to run this using a function relying on scipy.ndimage...
>>> > > >
>>> > > > When running gdal_translate on the VRT, I get ImportError: No
>>> module
>>> > >
>>> > > named
>>> > >
>>> > > > scipy.ndimage
>>> > > > This comes after successfully import numpy. scipy.ndimage will
>>> happily
>>> > > > import within the python interpreter.
>>> > >
>>> > > Works for me for both inline or offline functions.
>>> > >
>>> > > Are you sure GDAL loads the same python lib as the python version
>>> used in
>>> > > the
>>> > > python interpreter ? (check the debug traces with CPL_DEBUG=ON)
>>> > >
>>> > > You can also add at the top of your script
>>> > >
>>> > > import sys
>>> > > print(sys.path)
>>> > >
>>> > > and check if the output points to a location where your scipy
>>> package can
>>> > > be
>>> > > found.
>>> > >
>>> > > > Any tips on how to track this down/debug?
>>> > >
>>> > > > The entire VRT file is as follows:
>>> > > I guess this is not the entire VRT since it refers to an inline
>>> > > definition of
>>> > > the script but <PixelFunctionCode> has empty content.
>>> > >
>>> > > > <VRTDataset RasterXSize="111090" RasterYSize="259376">
>>> > > >
>>> > > >   <SRS>PROJCS["OSGB 1936 / British National Grid",GEOGCS["OSGB
>>> > > >
>>> > > > 1936",DATUM["OSGB_1936",SPHEROID["Airy
>>> > > > 1830",6377563.396,299.3249646,AUTHORITY["EPSG","7001"]],TOWG
>>> > >
>>> > > S84[446.448,-12
>>> > >
>>> > > > 5.157,542.06,0.15,0.247,0.842,-20.489],AUTHORITY["EPSG","627
>>> > >
>>> > > 7"]],PRIMEM["Gr
>>> > >
>>> > > > eenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532
>>> > >
>>> > > 925199433,AUTHO
>>> > >
>>> > > > RITY["EPSG","9122"]],AUTHORITY["EPSG","4277"]],PROJECTION["
>>> > >
>>> > > Transverse_Merca
>>> > >
>>> > > > tor"],PARAMETER["latitude_of_origin",49],PARAMETER["central_
>>> > >
>>> > > meridian",-2],P
>>> > >
>>> > > > ARAMETER["scale_factor",0.9996012717],PARAMETER["false_easti
>>> > >
>>> > > ng",400000],PAR
>>> > >
>>> > > > AMETER["false_northing",-100000],UNIT["metre",1,AUTHORITY["
>>> > >
>>> > > EPSG","9001"]],A
>>> > >
>>> > > > XIS["Easting",EAST],AXIS["Northing",NORTH],AUTHORITY["EPSG",
>>> > >
>>> > > "27700"]]</SRS>
>>> > >
>>> > > > <GeoTransform>100180.0,5.0,0.0,1215730.0,0.0,-5.0</GeoTransform>
>>> > > > <VRTRasterBand dataType="Float32" band="1"
>>> > > > subClass="VRTDerivedRasterBand">
>>> > > >
>>> > > >     <SimpleSource>
>>> > > >
>>> > > >       <SourceFilename
>>> > > >
>>> > > > relativeToVrt="0">F:\tif_data\large_sparse.tif</SourceFilename>
>>> > > >
>>> > > >       <SourceProperties BlockXSize="256" BlockYSize="256"
>>> > > >
>>> > > > RasterXSize="111090" RasterYSize="259376"/>
>>> > > >
>>> > > >         <OpenOptions>
>>> > > >
>>> > > >               <OOI key="NUM_THREADS">4</OOI>
>>> > > >               <OOI key="SPARSE_OK">TRUE</OOI>
>>> > > >
>>> > > >         </OpenOptions>
>>> > > >
>>> > > >     </SimpleSource>
>>> > > >     <PixelFunctionType>extract_blobs</PixelFunctionType>
>>> > > >     <PixelFunctionLanguage>Python</PixelFunctionLanguage>
>>> > > >     <PixelFunctionCode>
>>> > > >
>>> > > >       </PixelFunctionCode>
>>> > > >
>>> > > >     <BufferRadius>5</BufferRadius>
>>> > > >     <PixelFunctionArguments nodata="0" count_threshold="5"/>
>>> > > >
>>> > > >   </VRTRasterBand>
>>> > > >
>>> > > > </VRTDataset>
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > > --
>>> > > > View this message in context:
>>> > > > http://osgeo-org.1560.x6.nabble.com/gdal-dev-VRT-derived-
>>> > >
>>> > > band-pixel-functi
>>> > >
>>> > > > ons-written-in-Python-tp5285323p5285882.html Sent from the GDAL -
>>> Dev
>>> > > > mailing list archive at Nabble.com.
>>> > > > _______________________________________________
>>> > > > gdal-dev mailing list
>>> > > > gdal-dev at lists.osgeo.org
>>> > > > http://lists.osgeo.org/mailman/listinfo/gdal-dev
>>> > >
>>> > > --
>>> > > Spatialys - Geospatial professional services
>>> > > http://www.spatialys.com
>>>
>>> --
>>> Spatialys - Geospatial professional services
>>> http://www.spatialys.com
>>>
>>
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>



-- 
Sean Gillies
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20160915/1c4ee305/attachment-0001.html>


More information about the gdal-dev mailing list