Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0

[Christopher Barker]

Lots of stuff in this thread, I'll try to contribute what I can inline:

On 1/6/11 2:15 AM, Ari Jolma wrote:
> When I started there was only ActivePerl for Windows that was usable,
> but it turned out impossible for me to use because they use MS
> compilers. Now there is Strawberry Perl, which I might try again (or pay
> someone to try). It is good because it uses MinGW.

The situation is a bit different for Python -- the Python.org binaries 
are almost universal (and I think the ActiveState ones are compatible). 
So it's the obvious choice to build against. They are built with MS 
compilers, but you can compile extensions with MinGW (at least for 2.6 
and above). However, ideally the MS compilers are users -- there are 
freely available versions that work (not Open source though!)

> Then there could also be a more end-user type Windows installation
> package project for GDAL, which installs the stuff into some standard
> location (c:\Program Files\share?).

I think so too, but it's a bit unclear to me what's best as far as the 
various language bindings go...given all the versions of Python (and I 
assume others) it's a bit messy.

>  In fact
> Geoinformatica should be simply installable through Perl's cpan
> application - but that does not even work in Linux yet :( (the reason is
> that the GDAL Perl module installation script is not intelligent enough
> and the version of the module in CPAN is horribly old and I really suck
> at managing that)

Wow! given how much folks rave about CPAN, it sounds like PyPi is better 
in this regard, at least for binary packages.

On 1/6/11 4:00 AM, Jan Hartmann wrote:
> Three months ago I put an announcement on the MapServer-Users list about
> a shell script I wrote to compile the whole MapServer-GDAL-PostGIS-PLR
> suite, with and without PHP/Python,

Did you provide Python, to use an existing python.org build?

> Since then I got no (zero) replies, so I don't know whether it doesn't
> work, isn't interesting, or people just didn't look at it.

well, for what it's worth, on LInux, users can go a long way with distro 
packages and the plain old ./configure; make.. dance.

Binaries on Windows are in much higher demand.

On 1/6/11 4:24 AM, Tamas Szekeres wrote:
> Modifying the PATH globally is a bad practice in 99%
> of the cases.

really? for command line tools, I do it all the time! It seems the ay to 
go if you want it to just work". Maybe it makes for more dll hell?

> I would expect at least a development or a
> stable version (and their x86/x64 variants) to coexist which should be
> used by the same user.

This is mostly an issue for developers, and you are smart enough to take 
care of yourselves... As for me, I usually have a system wide default 
for things like GDAL on the PATH, and then will have a startup script or 
something that re-sets things if I want to run test versions, or I just 
call the exe I want explicitly

Anyway, putting a binary in a "standard" location is separate from 
having an installer modify the users PATH -- that second step should be 
optional, or just leave it for the user to do manually if they want.

> The same problem may arise when we would like to
> install multiple versions to the site packages directory, how the
> versions of the files are maintained by the python runtime?

Python is very weak in that regard, There are three ways to handl eit now:

1)a roll-your-open version selection mechanism -- wxPython and I think 
PyGTK do this.

2) virtualenv, or similar -- they set up a "virtual" python install, so 
that you can have multiple installs, each with different versions of 
packages

3) setuptools provides some mechanism for egg version selection, though 
i've never really figure out how to use it.

I think (2) is the best recommendation -- the nice thing about it is 
that the packager doesn't have to do anything special -- it's handled by 
how the user installs a package -- if they use setup.py or easy_install, 
or pip, it just works -- maybe not with an msi, though.

So this points to building binary eggs, ideally.

>  The individual packages may
> specify the required minimum version of the referred packages loaded by
> the .NET runtime. Is this something that can be done with the python
> environment as well?

That's kind of what setuptools provides, but it's pretty kludgy.

> (As opposed to this, the dumb solution of having a starting script to
> open a command prompt (and setting PYTHONPATH properly) would ensure
> multiple versions to be used at the same time, since those settings are
> applied to the cmd process solely.)

I suspect that's how virtualenv works, more or less.

On 1/6/11 5:01 AM, Tamas Szekeres wrote:
> I did mention to set the environment for the cmd process solely, and not
> by a systemwide setting. In this regard it doesn't matter how many
> python runtime is installed on a particular system, the only issue is to
> refer to the desired one (by setting the PATH to the python executable
> or using absolute path when running python) from the command prompt.

This is a fine solution for sophisticated developers, but believe Jason 
and me when we tell you that that's simply not how most Python users 
work. I think the goal here to to provide something that fits well into 
the common work flow.

> 3. Similar issue may exist with the multiple CRT dependencies privided
> by the packages, for example when I would package my vesion of mapserver
> compiled with MSVC2008 but we have only gdal-dev packages compiled with
> MSVC2003 out there.

This is a big Python issue -- extensions must be compiled with the same 
CRT as the interpreter -- that's  MSVC2008 for 2.6 and 2.7 (and 3.8??), 
I think 2.5 is MSVC2003 (but that may be 2.4, my memory is fading...). 
Interestingly, MingGW is OK, too.

Which is why I love your binaries that specify what they were built with!

 > But indeed we may speak about different things, you may probably
> want to install a single version of the gdal packages throughout the
> system, while I'd be curious about the desired approach to host multiple
> versions side by side.

see above -- virtualenv is probably the most common way to handle that now.

On 1/6/11 5:10 AM, Ari Jolma wrote:

> I think it would not make sense to include GDAL into such a binary perl
> module package. Thus GDAL would need to be separately installable -

This is a key question for Python, too. I lean toward having GDAL 
separate, and ideally it would then provide the command line tools, as 
well as the support for Perl, Python, etc...

2) we make GDAL-dev.msi  available at an URL.

yup.

I think having a python binary depend on a GDAL install is OK, as long 
as there is a simple installer to point folks to to get it.

On 1/6/11 5:49 AM, Ari Jolma wrote:
> :) I'm not completely satisfied (I'm also not sure I really understand
> the whole f* issue) with how Perl searches for dll's - especially dll
> import libs - in Windows.

me neither, with Python (or anything Windows, for that matter). Maybe 
Jason knows better, but I *think* we should be OK with a standard 
location for GDAL. But could you have:

Program Files\GDAL\1.6\gdal.dll
  and
Program Files\GDAL\1.6\gdal.dll

and have one Python binding find 1.6, and one find 1.7 -- I'm not sure 
how to do that, but it SHOULD be doable! OH what i'd give for a symlink!

On 1/6/11 6:50 AM, Tamas Szekeres wrote:
> The only wokaround which is satisfactory to me is to set the required
> enviroment at run time before the gdal bindings are used. In csharp we
> can do something like:
>
>                      string path =
> Environment.GetEnvironmentVariable("PATH");
>                      if (!path.Contains(gdal_bin_path))
>                      {
>                          Environment.SetEnvironmentVariable("PATH",
> gdal_bin_path + ";" + path);
>                      }
>                      Gdal.SetConfigOption("GDAL_DRIVER_PATH",
> gdal_driver_path);
>
> But it requires that loading the gdal dll is happening when the
> interface is used actually (in Gdal.SetConfigOption with the code
> above). This is the case with the .NET runtime,  but I'm not sure how do
> we stand with the other bindings in this regard.

I'm pretty sure the entry point for gdal on Python is a python file, so 
you could do something similar, before the binary module is loaded. But 
I don't know if that is standard practice.

On 1/6/11 7:25 AM, Jason Roberts wrote:

> Apologies in advance for repeating stuff below that you already know.

Great summary -- thanks! And you clearly have a better grasp that I on 
Windows dll hell.


> 1.The minimum case: build an installation package for the Python
> bindings but still require the user to manually store the GDAL binaries
> someplace and set PATH, PROJ_LIB, GDAL_DATA, etc.

I think this is a pretty sub-optimal approach, maybe not worth doing. 
(And maybe what Howard already has?)

> 2.Build an installation package as above. Have it install the GDAL DLLs
> as a subdirectory of the osgeo directory, e.g.
> C:\PythonXY\Lib\site-packages\osgeo\bin.

This is a good option if we decide to make the python bindings stand 
alone -- though I'm not sure if that's the best goal, but it may be.

> Modify gdal.py to set
> os.environ['PATH'] = os.environ['PATH'] + ';' gdalInstallDir to modify
> the PATH to include that directory prior to importing _gdal.pyd. The
> PATH will be modified for the running process only, for the duration of
> that process.

I didn't think that was required, if the dll is in the same place as the 
*.pyd, but maybe it is. And it seems it would work fine. Might it mess 
things up if someone where to do a system call from Python? Also, would 
that support py2exe?

I wonder what is donefor theones here:

http://www.lfd.uci.edu/~gohlke/pythonlibs/

No time to poke into that right now, sorry. We should look at them 
though, maybe He's solved our problems for us!

> 3.Same as #2 but rather than modifying gdal.py to set the PATH variable,
> instead create a new Python extension module called
> _gdal_dll_helper.pyd. The job of this C extension module is simply to
> get gdal.dll and other DLLs loaded without resorting to modifying the
> system PATH which can sometimes have weird consequences (I can explain
> more if needed). The extension module would call the Windows
> SetDllDirectory
> <http://msdn.microsoft.com/en-us/library/ms686203%28v=vs.85%29.aspx>
> function, call LoadLibrary to explicitly load gdal17.dll into the
> current process, then call SetDllDirectory again to set the DLL
> directory back to what it was previously. Then, when gdal.py wants to
> load _gdal.pyd, gdal17.dll is already loaded and the binding succeeds.

could all that be done with some ctypes calls, rather than another 
extension?

> 4.Statically link all necessary code to _gdal.pyd, _gdal_array.pyd, etc.
> This would eliminate the need to store the GDAL DLLs on the machine at
> all, as all code would be in the bindings .pyd DLLs. This would likely
> be a big job, and probably not even possible given the variety of
> libraries that GDAL leverages.

But is a good way to go if possible -- we try to do that for all 
Macintosh python binaries. Though I don't anyone has succeeded (or even 
tried) for GDAL.

On 1/6/11 8:07 AM, Tamas Szekeres wrote:
> I would personally vote on having an extension (like #3) which is
> imported by gdal.py in case if it is installed. If this extension is not
> installed, gdal.py would work as before. This extension would scan the
> registry to find out the install location of the corresponding files
> (probably based on an unique key) and perform the required actions to
> make the dll-s loadable.

I like that, except for the registry part -- I am fearful of the 
registry anyway, and it wouldn't work well for easy_install, etc.

Maybe the GDAL paths could just be put in a config file (or python file, 
for that matter) in the osgeo package dir.

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov