[QGIS-Developer] using conda for development

[Alexandre Neto]

Hi,

I will take this opportunity to give a brief explanation of how conda /
conda-forge works. At least what I was able to understand so far.

Full disclaimer: I know very little about building and maintaining packages
and installers, including the QGIS ones, but I try to trigger the update of
QGIS versions on Conda-forge.

*Conda* - an open source package management system and environment
management system developed mainly by anaconda. It provides tools to build,
install and run packages, and setup environments. One interesting feature
is that, in conda, the same building recipe can be used to build the
packages for Windows, Mac and Linux. Normally, to build a package, you use
prebuild packages of its dependencies which are available in conda online
channels. This doesn't mean that the same versions of the dependencies are
used for all platforms, as some dependency may not be available or doesn't
work for a specific platform.

*Conda-forge* - a community that maintains a big amount of conda recipes
for many open source packages, which are stored in github repositories.
Those are called feedstocks. Among them, there is a qgis-feedstock and also
feedstocks for almost all of its dependencies. Collaboration to keep those
recipes is normally very welcome.

https://conda-forge.org/feedstocks/

https://github.com/conda-forge/qgis-feedstock

All conda-forge recipes (each feedstock github branch) are built
automatically using Azure Pipelines and uploaded to the conda-forge channel
in Anaconda cloud, making it easily deployed for installation. QGIS has a
master (stable) and a LTR branch.

*Anaconda cloud* is a repository of prebuilt packages, which is organized
in channels.  Anyone can have a channel and upload their locally built
packages so that others can use them. If you search for QGIS in Anaconda
cloud you will find several channels with several versions available.

Answering Ari question:

On Fri, Jul 31, 2020 at 6:15 AM Ari Meyer <ari.meyer at gmail.com> wrote:

> @Greg: I found that many of the exact dependencies were not even available
> through the main channels.  So if the QGIS devs compile/build against such
> versions, I'm not sure there's an easy way to even specify such a conda
> recipe, unless all those dependencies are also made available somehow.  I
> didn't expect that those versions would not even be there with the others
> for the various libraries.  Could this imply that the library developers
> don't want those versions to be used?  Not sure.
>

I can only speak for Conda Forge and mainly QGIS feedstock. I normally try
to keep the recipes updated for every single new version of QGIS and QGIS
LTR, but I may forget to do so. Then, sometimes the recipes don't work out
of the box for a certain patch release of QGIS, the conda-forge folks try
to fix them, but meanwhile, a new patch release is out, so we may just
abandon that version recipe and move on. I expect the same may happen with
other packages... There is no intention of excluding versions for any
reason.

Regarding the version of the dependencies. Conda-forge community always
tries to use the most recent version of all dependencies. Sometimes that
doesn't work, and one needs to pin the version of a dependency. Also, my
idea was to keep the LTR main dependencies pinned during all its lifecycle,
to avoid regressions.

*Cool things about QGIS on conda (and conda-forge):*
- One single transparent recipe to build QGIS in all three main platforms;
- As users, to install one or several packages for a certain workflow, you
normally create an environment to isolate the install, which allows having
several versions of the same software in parallel in your machine.
Nevertheless, the packages shared by different environments/QGIS versions
are only downloaded once and don't take more space in the disk. Symbolic
links are used.
- Once set, you can share an environment with others, making it perfect for
collaborative development.
- Updating QGIS is like osfeo4w in all platforms, only the necessary
packages are downloads and install.
- Using Conda-forge infrastructure, you only need to update the recipe, the
build and upload to anaconda cloud are done in azure pipelines
"automagically".
- It's a great way to use QGIS PyQGIS and the command line tools (the
reason why I try to update QGIS versions in conda-forge)

*Caveats of  conda-forge:*
- As QGIS project, we have no direct control over dependencies used on the
master version, as the community has the goal of using the most recent
dependency versions that is possible. The same is not necessarily true for
LTR, where I feel we would be allowed to control and pin the dependencies
- Currently, there are some dependencies missing. For example, QtWebkit,
which limits QGIS functionality, including some nice plugins like
DataPlotly, and resource sharing plugin.

It seems to me that, if the QGIS project wanted to use conda as their main
building/packaging system, we would need to:
- Keep a fork of conda-forge recipes, and upload the result to a dedicated
channel, probably even outside anaconda cloud.
- We could use dependencies available on conda-forge or other channels, but
we would need to be ready to maintain abandoned packages that greatly
impact QGIS, like QtWebkit;
- Recreate the installers (at least Windows and Mac) to use conda tools for
downloading, installing, and updating packages.

Last bit, when I was working for Boundless, Larry Shaffer was trying to do
all this mentioned above, but then the company was bought by Planet... Not
sure what is the status of this work now.

https://github.com/osgeo-forge

Thanks, Sorry for the long post.

Alexandre Neto
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20200731/c8905a74/attachment.html>