[osgeo4w-dev] Alternative osgeo4w infrastructure

[Jürgen E. Fischer]

Hi Hugo,

On Thu, 27. Apr 2017 at 18:23:35 +0200, Hugo Mercier wrote:
> We've setup a compilation / CI infrastructure based on Gitlab CI that
> triggers Virtual box virtual machines for the actual compilation and
> packaging.

Sounds like something on my "one-fine-day" TODO list for long.


> So far, we noticed some issues with the current osgeo4w project and
> infrastructure:

> - the main repository seems heavily loaded and hard to reach sometimes

limited disk space is also a constantly reoccuring issue (although not actually visible
from the outside)

> - the build of the entire distribution is hard to reproduce

Indeed.

> - the points listed above also result in a situation where there is a mix of
>   compiler versions used for packages. Which should be avoided
>   (http://siomsystems.com/mixing-visual-studio-versions/)

That's certainly a fragile issue.  We don't track which compilers were used to
produce the individual packages - so it's also unclear on which version of the
msvcrt they depend on (msvcrt used to have several, although it was meanwhile
split up, the dependencies were still not refined everywhere).

But I think it's unavoidable sometimes - I wouldn't like to have three GDALs
and all their dependencies for QGIS 2 (msvc2010/py2), 3 (msvc2015/py3) and
GRASS (MinGW) in OSGeo4W (not to mention the GRASS plugin where all three would
meet anyway; also newer version of GDAL will not work with msvc2010 because of
c++11).  I think it's meant to work and it does, still it should be avoided
where possible.

Others:
* not listing of build dependencies (libraries and tools)
* no build recipes for everything
* those that exist don't have a form that would allow automatic invocation in a
  uniform way
* source packages don't contain sources nor automatically usable pointers
  to download them.

So there's much to do on that front before all this can work.

Also there are some proprietary SDKs (esp. for use with GDAL: ECW, MrSID,
FileGDB etc.) that I'm not sure you are allowed to make them public in such a
setup - although you are allowed to ship the redistributable runtime parts.


> We actually hit a problem probably related to that
> (https://trac.osgeo.org/osgeo4w/ticket/529) and that could mean to recompile
> gdal and all its dependencies with msvc 2015.

I think that is just due to the fact that numpy wasn't installed, when the
package was built.

That that problem is not already fixed, is just because I'm working on
transitioning building gdal with msvc2015 (needed for 2.2).

Having all this in place would have certainly helped with this of course.

The nightly should already work again - 2.2 not yet.


> So here are some propositions:
> - find a way to add a mirroring system for the osgeo4w distribution. The idea
> is to register a list of mirrors that would receive a copy of files
> when they are uploaded to the main repository.

AFAIK there is also a OSUSL file server/proxy that could be set in front
download.osgeo.org to help with this - bandwidth should be better.  Not sure if
that extra mirror architecture would still be necessary.


> - share access to our gitlab-ci infrastructure and vbox VMs (and allow
>   it to be copied somewhere else if needed). The idea would be to have a
>   unique repository with all package "sources" and all the machinery needed
>   to trigger a build of the whole distribution or of some selected packages
>   on a known environment.

Not sure if that also would be candidates for osgeo hosting - but self hosting
is probably easier and more flexible (also a reason why I didn't push this).
Perhaps leased hosted servers are even be cheaper than hosting "own" hardware
at OSUSL.


> Regarding the issue of mixing compiler versions, we apparently still
> need to compile python2.7 modules with msvc 2010 (which is the version
> used to compile python27 and all wheel binaries).

> Apart from that (which may imply to maintain two versions of the same
> libraries), do you see problems if all packages get compiled with the
> same (last) version of msvc (2015) ?

For GRASS this will certainly not work - that requires MinGW - no version of
msvc will work for that matter.


> Are there any interests for that work? Raise your hand if so, and raise
> both hands if you could contribute (time or funds) :) we also still need
> to find a way to fund this alternative osgeo4w / CI server.

Both hands full ;) - but consider them raised.

Although I have somewhat mixed feelings about this.  Except for Martin Landa
who took over GRASS of course, I have been pretty much on my own on this for a
while...


Jürgen

-- 
Jürgen E. Fischer           norBIT GmbH             Tel. +49-4931-918175-31
Dipl.-Inf. (FH)             Rheinstraße 13          Fax. +49-4931-918175-50
Software Engineer           D-26506 Norden             http://www.norbit.de
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 827 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/osgeo4w-dev/attachments/20170427/0b9261b1/attachment.sig>