[QGIS-Developer] using conda for development

[Ari Meyer]

Good to hear, Alexandre.  Your efforts are much appreciated!
Ari

On Sun, Aug 2, 2020 at 5:13 PM Alexandre Neto <senhor.neto at gmail.com> wrote:

> Hi,
>
> Following the discussion on QGIS dependencies on conda-forge, I had a chat
> with one of conda-forge most active maintainers.
>
> It seems that the conda-forge community is aware of the dangers of
> building only bleeding edge versions of the packages. There's a CFEP (same
> as our QEP) to improve the situation.
>
> https://hackmd.io/N1hoJGJBSqGTFd83pxCyYA
>
> Nevertheless, regarding the QGIS packages, and because they are not
> dependencies of any other packages, I was told that it would be ok to
> create local pinning configuration files for both our stable and LTR
> branches. This means we could have the control of the most important
> dependencies. I will try to mimic osgeo4w for 3.16 onwards, and make sure
> the pinning is not changed during the LTR life cycle.
>
> This does not fix the issue with no one maintaining the QtWebKit package,
> but I don't think no one really worked actively on it.
>
> My intentions are not to replace our packaging system (poor me), that,
> like Mathias said, works fine for now. I will just try to keep QGIS
> available on conda, as it seems useful for some folks, and it doesn't
> really take me too much time.
>
> Thanks,
>
> Alexandre Neto
>
> A domingo, 2/08/2020, 10:04, Matthias Kuhn <matthias at opengis.ch> escreveu:
>
>> Hi Ari, all
>>
>> I can follow your reasoning. Various discussions have taken place about
>> this already.
>>
>> Currently it's also a question of feasibility. Windows and macOS have
>> completely different dependency build chains / repositories. Synchronizing
>> those within their current completely separate frameworks will be hard to
>> achieve. (other OS'es come with their own set of dependencies already, so
>> we only can ignore them - or could build complete packages including all
>> deps in the future if we do have a standard set).
>>
>> I had a look at conda-forge in the past months to evaluate if it could be
>> considered fit for a reference build system. One precaution I had from the
>> start was the rolling release dependency hell we had experienced with
>> homebrew. It still needs to be proven if we get away with pinning. The
>> major roadblock I hit was already updating Qt. I tried to lift it to the
>> latest version with no success (and nobody else succeeded either despite
>> constant efforts in the last 4 months since that -- because of CI
>> limitations / build issues). If we would have a proposal how to continue
>> here (custom hosted CI with a responsible person taking care of it?), I am
>> open to explore this further.
>>
>> In summary, the current process we have works good enough, with the
>> occasional troubles of varying dependency versions you mention. Changing it
>> would be risky but possible and it is a major effort for which we currently
>> lack a widely accepted plan, resources and/or (long-term committed)
>> volunteers to do so.
>>
>> For your particular case here, since there are no standard versions of
>> dependencies, conda might be as good as any other guess. Or you go for the
>> installer of your main target platform instead.
>>
>> Best regards
>>
>> Matthias
>>
>>
>> On 8/2/20 8:49 AM, Ari Meyer wrote:
>>
>> Obrigado, Alexandre, for all the details. :-)  I guess I'm still baffled
>> by, "Conda-forge community always tries to use the most recent version of
>> all dependencies."  It just seems like a "recipe" for disaster -- just
>> built on assumptions.  Maybe you get lucky and everything just happens to
>> work.  Or maybe you get slightly less lucky and encounter breakages
>> immediately, forcing you to fiddle with the build until you find some
>> combination that seems to work, if you can't pin the versions.  Or maybe
>> you get unlucky and everything builds and *seems* to work based on the
>> extent of your testing, but your testing may have important gaps.
>>
>> In my previous contract I was using PyViz, which was rebranded as
>> HoloViz.  As that set of libraries was immature and constantly in flux, and
>> we wanted to take advantage of new features/fixes, our policy was to go
>> with the latest releases.  I wasn't comfortable with this, esp. after
>> enduring many problems, but we just dealt with bugs and regressions as they
>> came up.  QGIS, though, is much more mature, after more than a decade since
>> 1.0 was released (2009 according to Wiki).  I assumed, therefore, that they
>> wouldn't trust such a hit-or-miss strategy with dependencies.  But maybe
>> I'm just being a worry wart. ;-)  In any case, I'm going to be hard-pressed
>> to sell conda to my team if the dependencies are out of sync with those of
>> the installers.
>>
>> On Fri, Jul 31, 2020 at 8:49 AM Alexandre Neto <senhor.neto at gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I will take this opportunity to give a brief explanation of how conda /
>>> conda-forge works. At least what I was able to understand so far.
>>>
>>> Full disclaimer: I know very little about building and maintaining
>>> packages and installers, including the QGIS ones, but I try to trigger the
>>> update of QGIS versions on Conda-forge.
>>>
>>> *Conda* - an open source package management system and environment
>>> management system developed mainly by anaconda. It provides tools to build,
>>> install and run packages, and setup environments. One interesting feature
>>> is that, in conda, the same building recipe can be used to build the
>>> packages for Windows, Mac and Linux. Normally, to build a package, you use
>>> prebuild packages of its dependencies which are available in conda online
>>> channels. This doesn't mean that the same versions of the dependencies are
>>> used for all platforms, as some dependency may not be available or doesn't
>>> work for a specific platform.
>>>
>>> *Conda-forge* - a community that maintains a big amount of conda
>>> recipes for many open source packages, which are stored in github
>>> repositories. Those are called feedstocks. Among them, there is a
>>> qgis-feedstock and also feedstocks for almost all of its dependencies.
>>> Collaboration to keep those recipes is normally very welcome.
>>>
>>> https://conda-forge.org/feedstocks/
>>>
>>> https://github.com/conda-forge/qgis-feedstock
>>>
>>> All conda-forge recipes (each feedstock github branch) are built
>>> automatically using Azure Pipelines and uploaded to the conda-forge channel
>>> in Anaconda cloud, making it easily deployed for installation. QGIS has a
>>> master (stable) and a LTR branch.
>>>
>>> *Anaconda cloud* is a repository of prebuilt packages, which is
>>> organized in channels.  Anyone can have a channel and upload their locally
>>> built packages so that others can use them. If you search for QGIS in
>>> Anaconda cloud you will find several channels with several versions
>>> available.
>>>
>>> Answering Ari question:
>>>
>>> On Fri, Jul 31, 2020 at 6:15 AM Ari Meyer <ari.meyer at gmail.com> wrote:
>>>
>>>> @Greg: I found that many of the exact dependencies were not even
>>>> available through the main channels.  So if the QGIS devs compile/build
>>>> against such versions, I'm not sure there's an easy way to even specify
>>>> such a conda recipe, unless all those dependencies are also made available
>>>> somehow.  I didn't expect that those versions would not even be there with
>>>> the others for the various libraries.  Could this imply that the library
>>>> developers don't want those versions to be used?  Not sure.
>>>>
>>>
>>> I can only speak for Conda Forge and mainly QGIS feedstock. I normally
>>> try to keep the recipes updated for every single new version of QGIS and
>>> QGIS LTR, but I may forget to do so. Then, sometimes the recipes don't work
>>> out of the box for a certain patch release of QGIS, the conda-forge folks
>>> try to fix them, but meanwhile, a new patch release is out, so we may just
>>> abandon that version recipe and move on. I expect the same may happen with
>>> other packages... There is no intention of excluding versions for any
>>> reason.
>>>
>>> Regarding the version of the dependencies. Conda-forge community always
>>> tries to use the most recent version of all dependencies. Sometimes that
>>> doesn't work, and one needs to pin the version of a dependency. Also, my
>>> idea was to keep the LTR main dependencies pinned during all its lifecycle,
>>> to avoid regressions.
>>>
>>> *Cool things about QGIS on conda (and conda-forge):*
>>> - One single transparent recipe to build QGIS in all three main
>>> platforms;
>>> - As users, to install one or several packages for a certain workflow,
>>> you normally create an environment to isolate the install, which allows
>>> having several versions of the same software in parallel in your machine.
>>> Nevertheless, the packages shared by different environments/QGIS versions
>>> are only downloaded once and don't take more space in the disk. Symbolic
>>> links are used.
>>> - Once set, you can share an environment with others, making it perfect
>>> for collaborative development.
>>> - Updating QGIS is like osfeo4w in all platforms, only the necessary
>>> packages are downloads and install.
>>> - Using Conda-forge infrastructure, you only need to update the recipe,
>>> the build and upload to anaconda cloud are done in azure pipelines
>>> "automagically".
>>> - It's a great way to use QGIS PyQGIS and the command line tools (the
>>> reason why I try to update QGIS versions in conda-forge)
>>>
>>> *Caveats of  conda-forge:*
>>> - As QGIS project, we have no direct control over dependencies used on
>>> the master version, as the community has the goal of using the most recent
>>> dependency versions that is possible. The same is not necessarily true for
>>> LTR, where I feel we would be allowed to control and pin the dependencies
>>> - Currently, there are some dependencies missing. For example, QtWebkit,
>>> which limits QGIS functionality, including some nice plugins like
>>> DataPlotly, and resource sharing plugin.
>>>
>>> It seems to me that, if the QGIS project wanted to use conda as their
>>> main building/packaging system, we would need to:
>>> - Keep a fork of conda-forge recipes, and upload the result to a
>>> dedicated channel, probably even outside anaconda cloud.
>>> - We could use dependencies available on conda-forge or other channels,
>>> but we would need to be ready to maintain abandoned packages that greatly
>>> impact QGIS, like QtWebkit;
>>> - Recreate the installers (at least Windows and Mac) to use conda tools
>>> for downloading, installing, and updating packages.
>>>
>>> Last bit, when I was working for Boundless, Larry Shaffer was trying to
>>> do all this mentioned above, but then the company was bought by Planet...
>>> Not sure what is the status of this work now.
>>>
>>> https://github.com/osgeo-forge
>>>
>>> Thanks, Sorry for the long post.
>>>
>>> Alexandre Neto
>>>
>>>
>>>
>>>
>> _______________________________________________
>> QGIS-Developer mailing listQGIS-Developer at lists.osgeo.org
>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20200803/e4ed5cba/attachment-0001.html>