[QGIS-Developer] using conda for development

Sun Aug 2 22:40:37 PDT 2020

Thanks for explaining, Matthias.  That's particularly concerning with Qt,
as it's core to so many projects and prides itself on being
cross-platform.  I still think conda (using pinned versions) is worth
trying.

Cheers!
Ari

On Sun, Aug 2, 2020 at 4:04 AM Matthias Kuhn <matthias at opengis.ch> wrote:

> Hi Ari, all
>
> I can follow your reasoning. Various discussions have taken place about
> this already.
>
> Currently it's also a question of feasibility. Windows and macOS have
> completely different dependency build chains / repositories. Synchronizing
> those within their current completely separate frameworks will be hard to
> achieve. (other OS'es come with their own set of dependencies already, so
> we only can ignore them - or could build complete packages including all
> deps in the future if we do have a standard set).
>
> I had a look at conda-forge in the past months to evaluate if it could be
> considered fit for a reference build system. One precaution I had from the
> start was the rolling release dependency hell we had experienced with
> homebrew. It still needs to be proven if we get away with pinning. The
> major roadblock I hit was already updating Qt. I tried to lift it to the
> latest version with no success (and nobody else succeeded either despite
> constant efforts in the last 4 months since that -- because of CI
> limitations / build issues). If we would have a proposal how to continue
> here (custom hosted CI with a responsible person taking care of it?), I am
> open to explore this further.
>
> In summary, the current process we have works good enough, with the
> occasional troubles of varying dependency versions you mention. Changing it
> would be risky but possible and it is a major effort for which we currently
> lack a widely accepted plan, resources and/or (long-term committed)
> volunteers to do so.
>
> For your particular case here, since there are no standard versions of
> dependencies, conda might be as good as any other guess. Or you go for the
> installer of your main target platform instead.
>
> Best regards
>
> Matthias
>
>
> On 8/2/20 8:49 AM, Ari Meyer wrote:
>
> Obrigado, Alexandre, for all the details. :-)  I guess I'm still baffled
> by, "Conda-forge community always tries to use the most recent version of
> all dependencies."  It just seems like a "recipe" for disaster -- just
> built on assumptions.  Maybe you get lucky and everything just happens to
> work.  Or maybe you get slightly less lucky and encounter breakages
> immediately, forcing you to fiddle with the build until you find some
> combination that seems to work, if you can't pin the versions.  Or maybe
> you get unlucky and everything builds and *seems* to work based on the
> extent of your testing, but your testing may have important gaps.
>
> In my previous contract I was using PyViz, which was rebranded as
> HoloViz.  As that set of libraries was immature and constantly in flux, and
> we wanted to take advantage of new features/fixes, our policy was to go
> with the latest releases.  I wasn't comfortable with this, esp. after
> enduring many problems, but we just dealt with bugs and regressions as they
> came up.  QGIS, though, is much more mature, after more than a decade since
> 1.0 was released (2009 according to Wiki).  I assumed, therefore, that they
> wouldn't trust such a hit-or-miss strategy with dependencies.  But maybe
> I'm just being a worry wart. ;-)  In any case, I'm going to be hard-pressed
> to sell conda to my team if the dependencies are out of sync with those of
> the installers.
>
> On Fri, Jul 31, 2020 at 8:49 AM Alexandre Neto <senhor.neto at gmail.com>
> wrote:
>
>> Hi,
>>
>> I will take this opportunity to give a brief explanation of how conda /
>> conda-forge works. At least what I was able to understand so far.
>>
>> Full disclaimer: I know very little about building and maintaining
>> packages and installers, including the QGIS ones, but I try to trigger the
>> update of QGIS versions on Conda-forge.
>>
>> *Conda* - an open source package management system and environment
>> management system developed mainly by anaconda. It provides tools to build,
>> install and run packages, and setup environments. One interesting feature
>> is that, in conda, the same building recipe can be used to build the
>> packages for Windows, Mac and Linux. Normally, to build a package, you use
>> prebuild packages of its dependencies which are available in conda online
>> channels. This doesn't mean that the same versions of the dependencies are
>> used for all platforms, as some dependency may not be available or doesn't
>> work for a specific platform.
>>
>> *Conda-forge* - a community that maintains a big amount of conda recipes
>> for many open source packages, which are stored in github repositories.
>> Those are called feedstocks. Among them, there is a qgis-feedstock and also
>> feedstocks for almost all of its dependencies. Collaboration to keep those
>> recipes is normally very welcome.
>>
>> https://conda-forge.org/feedstocks/
>>
>> https://github.com/conda-forge/qgis-feedstock
>>
>> All conda-forge recipes (each feedstock github branch) are built
>> automatically using Azure Pipelines and uploaded to the conda-forge channel
>> in Anaconda cloud, making it easily deployed for installation. QGIS has a
>> master (stable) and a LTR branch.
>>
>> *Anaconda cloud* is a repository of prebuilt packages, which is
>> organized in channels.  Anyone can have a channel and upload their locally
>> built packages so that others can use them. If you search for QGIS in
>> Anaconda cloud you will find several channels with several versions
>> available.
>>
>> Answering Ari question:
>>
>> On Fri, Jul 31, 2020 at 6:15 AM Ari Meyer <ari.meyer at gmail.com> wrote:
>>
>>> @Greg: I found that many of the exact dependencies were not even
>>> available through the main channels.  So if the QGIS devs compile/build
>>> against such versions, I'm not sure there's an easy way to even specify
>>> such a conda recipe, unless all those dependencies are also made available
>>> somehow.  I didn't expect that those versions would not even be there with
>>> the others for the various libraries.  Could this imply that the library
>>> developers don't want those versions to be used?  Not sure.
>>>
>>
>> I can only speak for Conda Forge and mainly QGIS feedstock. I normally
>> try to keep the recipes updated for every single new version of QGIS and
>> QGIS LTR, but I may forget to do so. Then, sometimes the recipes don't work
>> out of the box for a certain patch release of QGIS, the conda-forge folks
>> try to fix them, but meanwhile, a new patch release is out, so we may just
>> abandon that version recipe and move on. I expect the same may happen with
>> other packages... There is no intention of excluding versions for any
>> reason.
>>
>> Regarding the version of the dependencies. Conda-forge community always
>> tries to use the most recent version of all dependencies. Sometimes that
>> doesn't work, and one needs to pin the version of a dependency. Also, my
>> idea was to keep the LTR main dependencies pinned during all its lifecycle,
>> to avoid regressions.
>>
>> *Cool things about QGIS on conda (and conda-forge):*
>> - One single transparent recipe to build QGIS in all three main platforms;
>> - As users, to install one or several packages for a certain workflow,
>> you normally create an environment to isolate the install, which allows
>> having several versions of the same software in parallel in your machine.
>> Nevertheless, the packages shared by different environments/QGIS versions
>> are only downloaded once and don't take more space in the disk. Symbolic
>> links are used.
>> - Once set, you can share an environment with others, making it perfect
>> for collaborative development.
>> - Updating QGIS is like osfeo4w in all platforms, only the necessary
>> packages are downloads and install.
>> - Using Conda-forge infrastructure, you only need to update the recipe,
>> the build and upload to anaconda cloud are done in azure pipelines
>> "automagically".
>> - It's a great way to use QGIS PyQGIS and the command line tools (the
>> reason why I try to update QGIS versions in conda-forge)
>>
>> *Caveats of  conda-forge:*
>> - As QGIS project, we have no direct control over dependencies used on
>> the master version, as the community has the goal of using the most recent
>> dependency versions that is possible. The same is not necessarily true for
>> LTR, where I feel we would be allowed to control and pin the dependencies
>> - Currently, there are some dependencies missing. For example, QtWebkit,
>> which limits QGIS functionality, including some nice plugins like
>> DataPlotly, and resource sharing plugin.
>>
>> It seems to me that, if the QGIS project wanted to use conda as their
>> main building/packaging system, we would need to:
>> - Keep a fork of conda-forge recipes, and upload the result to a
>> dedicated channel, probably even outside anaconda cloud.
>> - We could use dependencies available on conda-forge or other channels,
>> but we would need to be ready to maintain abandoned packages that greatly
>> impact QGIS, like QtWebkit;
>> - Recreate the installers (at least Windows and Mac) to use conda tools
>> for downloading, installing, and updating packages.
>>
>> Last bit, when I was working for Boundless, Larry Shaffer was trying to
>> do all this mentioned above, but then the company was bought by Planet...
>> Not sure what is the status of this work now.
>>
>> https://github.com/osgeo-forge
>>
>> Thanks, Sorry for the long post.
>>
>> Alexandre Neto
>>
>>
>>
>>
> _______________________________________________
> QGIS-Developer mailing listQGIS-Developer at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20200803/91e7f1c3/attachment-0001.html>