[QGIS-Developer] using conda for development

Sun Aug 2 15:12:57 PDT 2020

Hi,

Following the discussion on QGIS dependencies on conda-forge, I had a chat
with one of conda-forge most active maintainers.

It seems that the conda-forge community is aware of the dangers of building
only bleeding edge versions of the packages. There's a CFEP (same as our
QEP) to improve the situation.

https://hackmd.io/N1hoJGJBSqGTFd83pxCyYA

Nevertheless, regarding the QGIS packages, and because they are not
dependencies of any other packages, I was told that it would be ok to
create local pinning configuration files for both our stable and LTR
branches. This means we could have the control of the most important
dependencies. I will try to mimic osgeo4w for 3.16 onwards, and make sure
the pinning is not changed during the LTR life cycle.

This does not fix the issue with no one maintaining the QtWebKit package,
but I don't think no one really worked actively on it.

My intentions are not to replace our packaging system (poor me), that, like
Mathias said, works fine for now. I will just try to keep QGIS available on
conda, as it seems useful for some folks, and it doesn't really take me too
much time.

Thanks,

Alexandre Neto

A domingo, 2/08/2020, 10:04, Matthias Kuhn <matthias at opengis.ch> escreveu:

> Hi Ari, all
>
> I can follow your reasoning. Various discussions have taken place about
> this already.
>
> Currently it's also a question of feasibility. Windows and macOS have
> completely different dependency build chains / repositories. Synchronizing
> those within their current completely separate frameworks will be hard to
> achieve. (other OS'es come with their own set of dependencies already, so
> we only can ignore them - or could build complete packages including all
> deps in the future if we do have a standard set).
>
> I had a look at conda-forge in the past months to evaluate if it could be
> considered fit for a reference build system. One precaution I had from the
> start was the rolling release dependency hell we had experienced with
> homebrew. It still needs to be proven if we get away with pinning. The
> major roadblock I hit was already updating Qt. I tried to lift it to the
> latest version with no success (and nobody else succeeded either despite
> constant efforts in the last 4 months since that -- because of CI
> limitations / build issues). If we would have a proposal how to continue
> here (custom hosted CI with a responsible person taking care of it?), I am
> open to explore this further.
>
> In summary, the current process we have works good enough, with the
> occasional troubles of varying dependency versions you mention. Changing it
> would be risky but possible and it is a major effort for which we currently
> lack a widely accepted plan, resources and/or (long-term committed)
> volunteers to do so.
>
> For your particular case here, since there are no standard versions of
> dependencies, conda might be as good as any other guess. Or you go for the
> installer of your main target platform instead.
>
> Best regards
>
> Matthias
>
>
> On 8/2/20 8:49 AM, Ari Meyer wrote:
>
> Obrigado, Alexandre, for all the details. :-)  I guess I'm still baffled
> by, "Conda-forge community always tries to use the most recent version of
> all dependencies."  It just seems like a "recipe" for disaster -- just
> built on assumptions.  Maybe you get lucky and everything just happens to
> work.  Or maybe you get slightly less lucky and encounter breakages
> immediately, forcing you to fiddle with the build until you find some
> combination that seems to work, if you can't pin the versions.  Or maybe
> you get unlucky and everything builds and *seems* to work based on the
> extent of your testing, but your testing may have important gaps.
>
> In my previous contract I was using PyViz, which was rebranded as
> HoloViz.  As that set of libraries was immature and constantly in flux, and
> we wanted to take advantage of new features/fixes, our policy was to go
> with the latest releases.  I wasn't comfortable with this, esp. after
> enduring many problems, but we just dealt with bugs and regressions as they
> came up.  QGIS, though, is much more mature, after more than a decade since
> 1.0 was released (2009 according to Wiki).  I assumed, therefore, that they
> wouldn't trust such a hit-or-miss strategy with dependencies.  But maybe
> I'm just being a worry wart. ;-)  In any case, I'm going to be hard-pressed
> to sell conda to my team if the dependencies are out of sync with those of
> the installers.
>
> On Fri, Jul 31, 2020 at 8:49 AM Alexandre Neto <senhor.neto at gmail.com>
> wrote:
>
>> Hi,
>>
>> I will take this opportunity to give a brief explanation of how conda /
>> conda-forge works. At least what I was able to understand so far.
>>
>> Full disclaimer: I know very little about building and maintaining
>> packages and installers, including the QGIS ones, but I try to trigger the
>> update of QGIS versions on Conda-forge.
>>
>> *Conda* - an open source package management system and environment
>> management system developed mainly by anaconda. It provides tools to build,
>> install and run packages, and setup environments. One interesting feature
>> is that, in conda, the same building recipe can be used to build the
>> packages for Windows, Mac and Linux. Normally, to build a package, you use
>> prebuild packages of its dependencies which are available in conda online
>> channels. This doesn't mean that the same versions of the dependencies are
>> used for all platforms, as some dependency may not be available or doesn't
>> work for a specific platform.
>>
>> *Conda-forge* - a community that maintains a big amount of conda recipes
>> for many open source packages, which are stored in github repositories.
>> Those are called feedstocks. Among them, there is a qgis-feedstock and also
>> feedstocks for almost all of its dependencies. Collaboration to keep those
>> recipes is normally very welcome.
>>
>> https://conda-forge.org/feedstocks/
>>
>> https://github.com/conda-forge/qgis-feedstock
>>
>> All conda-forge recipes (each feedstock github branch) are built
>> automatically using Azure Pipelines and uploaded to the conda-forge channel
>> in Anaconda cloud, making it easily deployed for installation. QGIS has a
>> master (stable) and a LTR branch.
>>
>> *Anaconda cloud* is a repository of prebuilt packages, which is
>> organized in channels.  Anyone can have a channel and upload their locally
>> built packages so that others can use them. If you search for QGIS in
>> Anaconda cloud you will find several channels with several versions
>> available.
>>
>> Answering Ari question:
>>
>> On Fri, Jul 31, 2020 at 6:15 AM Ari Meyer <ari.meyer at gmail.com> wrote:
>>
>>> @Greg: I found that many of the exact dependencies were not even
>>> available through the main channels.  So if the QGIS devs compile/build
>>> against such versions, I'm not sure there's an easy way to even specify
>>> such a conda recipe, unless all those dependencies are also made available
>>> somehow.  I didn't expect that those versions would not even be there with
>>> the others for the various libraries.  Could this imply that the library
>>> developers don't want those versions to be used?  Not sure.
>>>
>>
>> I can only speak for Conda Forge and mainly QGIS feedstock. I normally
>> try to keep the recipes updated for every single new version of QGIS and
>> QGIS LTR, but I may forget to do so. Then, sometimes the recipes don't work
>> out of the box for a certain patch release of QGIS, the conda-forge folks
>> try to fix them, but meanwhile, a new patch release is out, so we may just
>> abandon that version recipe and move on. I expect the same may happen with
>> other packages... There is no intention of excluding versions for any
>> reason.
>>
>> Regarding the version of the dependencies. Conda-forge community always
>> tries to use the most recent version of all dependencies. Sometimes that
>> doesn't work, and one needs to pin the version of a dependency. Also, my
>> idea was to keep the LTR main dependencies pinned during all its lifecycle,
>> to avoid regressions.
>>
>> *Cool things about QGIS on conda (and conda-forge):*
>> - One single transparent recipe to build QGIS in all three main platforms;
>> - As users, to install one or several packages for a certain workflow,
>> you normally create an environment to isolate the install, which allows
>> having several versions of the same software in parallel in your machine.
>> Nevertheless, the packages shared by different environments/QGIS versions
>> are only downloaded once and don't take more space in the disk. Symbolic
>> links are used.
>> - Once set, you can share an environment with others, making it perfect
>> for collaborative development.
>> - Updating QGIS is like osfeo4w in all platforms, only the necessary
>> packages are downloads and install.
>> - Using Conda-forge infrastructure, you only need to update the recipe,
>> the build and upload to anaconda cloud are done in azure pipelines
>> "automagically".
>> - It's a great way to use QGIS PyQGIS and the command line tools (the
>> reason why I try to update QGIS versions in conda-forge)
>>
>> *Caveats of  conda-forge:*
>> - As QGIS project, we have no direct control over dependencies used on
>> the master version, as the community has the goal of using the most recent
>> dependency versions that is possible. The same is not necessarily true for
>> LTR, where I feel we would be allowed to control and pin the dependencies
>> - Currently, there are some dependencies missing. For example, QtWebkit,
>> which limits QGIS functionality, including some nice plugins like
>> DataPlotly, and resource sharing plugin.
>>
>> It seems to me that, if the QGIS project wanted to use conda as their
>> main building/packaging system, we would need to:
>> - Keep a fork of conda-forge recipes, and upload the result to a
>> dedicated channel, probably even outside anaconda cloud.
>> - We could use dependencies available on conda-forge or other channels,
>> but we would need to be ready to maintain abandoned packages that greatly
>> impact QGIS, like QtWebkit;
>> - Recreate the installers (at least Windows and Mac) to use conda tools
>> for downloading, installing, and updating packages.
>>
>> Last bit, when I was working for Boundless, Larry Shaffer was trying to
>> do all this mentioned above, but then the company was bought by Planet...
>> Not sure what is the status of this work now.
>>
>> https://github.com/osgeo-forge
>>
>> Thanks, Sorry for the long post.
>>
>> Alexandre Neto
>>
>>
>>
>>
> _______________________________________________
> QGIS-Developer mailing listQGIS-Developer at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20200802/11b30f00/attachment-0001.html>