Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0

Jason Roberts jason.roberts at duke.edu
Thu Jan 6 12:25:49 EST 2011


Tamas,

 

I think with either option #2 or #3, gdal.py will have to be modified in
order for everything to be invisible to the user. In #2, gdal.py would have
to modify the PATH variable. In #3, it would have to import the
_gdal_dll_helper module. In either case, it would just be a few lines of
code at the top of the file that occurs before any other GDAL extension
modules are loaded (_gdal, _gdal_array, etc). It would also need to be done
for other modules such as ogr.py, osr.py, or whatever also imports GDAL
extension modules (unless those ones also import gdal.py first).

 

So #3 is not better by virtue of not having to modify the bindings; it does
have to modify the bindings. But #3 is appealing because setting the PATH
from Python code sometimes has weird issues. For example, when I tried it in
my code, it produced a weird problem when I tried to use the "nose" Python
testing framework to run test cases that involved calling Python modules
from GDAL and ArcGIS. The problem there may ultimately be a Microsoft issue
in which the environment settings for a process are maintained by the
Microsoft C runtime library (msvcrt*.dll) and Python, ArcGIS, and GDAL may
use different versions of that library. I do not want to digress into this
here, but suffice to say, I prefer #3 to #2 because it did not have this
problem.

 

Here is an outline of tasks. We probably should put it in an RFC to keep
track.

 

1.    Modify the makefiles for your SDK so that it runs
release-XXXX-dev\gdal\swig\python\setup.py with the "bdist
--formats=wininst" option. This will produce an installation program such as
gdal-1.7.3.win32-py2.5.exe. This is what the user will run to install the
Python bindings together with a private copy of the GDAL DLLs used just by
those bindings. On Python 2.6 and later, we probably want "bdist
--formats=msi" to produce a .msi file rather than a .exe, because Windows
likes .msi files much better than .exes for security purposes. IIRC, Python
2.5 and earlier do not have the msi option.

 

2.    Modify release-XXXX-dev\gdal\swig\python\setup.py to include the GDAL
DLLs and data files in the data_files list that is passed to the setup()
function. Make sure it is only done for Windows (Python code can check
that). The goal is to have the installation program create the following
kind of installation:

 

Existing Python bindings:

 

C:\PythonXX\Lib\site-packages\osgeo\gdal.py

C:\PythonXX\Lib\site-packages\osgeo\ogr.py

C:\PythonXX\Lib\site-packages\osgeo\...

C:\PythonXX\Lib\site-packages\osgeo\_gdal.pyd

C:\PythonXX\Lib\site-packages\osgeo\_ogr.pyd

C:\PythonXX\Lib\site-packages\osgeo\...

 

New bin, data, and plugins directories, which are private to this
installation of the bindings:

 

C:\PythonXX\Lib\site-packages\osgeo\bin\gdal17.dll

C:\PythonXX\Lib\site-packages\osgeo\bin\proj.dll

C:\PythonXX\Lib\site-packages\osgeo\bin\...

 

C:\PythonXX\Lib\site-packages\osgeo\bin\gdal-data\coordinate_axis.csv

C:\PythonXX\Lib\site-packages\osgeo\bin\gdal-data\...

 

C:\PythonXX\Lib\site-packages\osgeo\bin\gdal-plugins\ogr_PG.dll

C:\PythonXX\Lib\site-packages\osgeo\bin\gdal-plugins\...

 

3.    Introduce a new file
release-XXXX-dev\gdal\swig\python\extensions\_gdal_dll_loader.cpp. This file
will call GetDllDirectory to find the current DLL directory setting, call
SetDllDirectory to C:\PythonXX\Lib\site-packages\osgeo\bin (or wherever the
Python instance is installed), call LoadLibrary("gdal17.dll"), and call
SetDllDirectory back to what it was before. There are more details to
consider. For example, we might want to have it call LoadLibrary first to
see if it can load the GDAL DLL from the PATH, allowing the user to use
their own GDAL DLLs without having to overwrite the ones in the Python
directory. Or we might not want to do that, with the idea that the GDAL
Python bindings must be tightly coupled to a particular version of GDAL's
DLLs.

 

4.    Modify setup.py to compile _gdal_dll_loader.cpp, only for Windows.
This will result in _gdal_dll_loader.pyd alongside the other .pyd files.

 

5.    Modify gdal.py, osr.py, and so on to do something like this:

 

import sys

if sys.platform == 'win32':     # TODO: add check for Windows x64 as well

    try:

        import osgeo._gdal_dll_loader          # As part of this, Python
will call an "init" function inside. That function will do the
SetDllDirectory, etc.

    except:

        pass

 

6.    Do whatever is necessary to ensure the GDAL_DATA etc are properly set
up inside GDAL. I can't remember if this will "just work" using directories
named gdal-data, gdal-plugins, or if it is necessary for the
_gdal_dll_loader.pyd to call some GDAL functions to make it happen.

 

Does that sound like the right direction?

 

Jason

 

From: Tamas Szekeres [mailto:szekerest at gmail.com] 
Sent: Thursday, January 06, 2011 11:07 AM
To: Jason Roberts
Cc: gdal-dev at lists.osgeo.org
Subject: Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0

 

 

2011/1/6 Jason Roberts <jason.roberts at duke.edu>

2.    Build an installation package as above. Have it install the GDAL DLLs
as a subdirectory of the osgeo directory, e.g.
C:\PythonXY\Lib\site-packages\osgeo\bin. Modify gdal.py to set
os.environ['PATH'] = os.environ['PATH'] + ';' gdalInstallDir to modify the
PATH to include that directory prior to importing _gdal.pyd. The PATH will
be modified for the running process only, for the duration of that process.

 

3.    Same as #2 but rather than modifying gdal.py to set the PATH variable,
instead create a new Python extension module called _gdal_dll_helper.pyd.
The job of this C extension module is simply to get gdal.dll and other DLLs
loaded without resorting to modifying the system PATH which can sometimes
have weird consequences (I can explain more if needed). The extension module
would call the Windows SetDllDirectory
<http://msdn.microsoft.com/en-us/library/ms686203%28v=vs.85%29.aspx>
function, call LoadLibrary to explicitly load gdal17.dll into the current
process, then call SetDllDirectory again to set the DLL directory back to
what it was previously. Then, when gdal.py wants to load _gdal.pyd,
gdal17.dll is already loaded and the binding succeeds.

I know #2 and #3 sound scary but they can be done cleanly. I currently use a
variation of #3 in my own project that embeds GDAL and its Python bindings.

 

 


Jason,

Good writeup, thanks for that. You mention #2 and #3 which is exactly the
same I have in my mind. Already posted a code for #2 related to the csharp
bindings to demonstrate this option.

#2 would be fairly transparent for the user (no additional modules
imported), the only thing we could add is how gdal.py would find out the
install location of the gdal files. This may probably set in the registry
added by the installer. Not sure how this addition would affect the gdal
binding on other platforms (if we implement this in gdal)

#3 seems to be better for gdal (doesn't require to modify the bindings)
however the user will require to use an additional module (which is not
required normally when using gdal.py)


I would personally vote on having an extension (like #3) which is imported
by gdal.py in case if it is installed. If this extension is not installed,
gdal.py would work as before. This extension would scan the registry to find
out the install location of the corresponding files (probably based on an
unique key) and perform the required actions to make the dll-s loadable.


Best regards,

Tamas




-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/gdal-dev/attachments/20110106/a1b52e8a/attachment-0001.html


More information about the gdal-dev mailing list