<div dir="ltr"><div>Good to hear, Alexandre. Your efforts are much appreciated!</div><div>Ari<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Aug 2, 2020 at 5:13 PM Alexandre Neto <<a href="mailto:senhor.neto@gmail.com">senhor.neto@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">Hi,<div dir="auto"><br></div><div dir="auto">Following the discussion on QGIS dependencies on conda-forge, I had a chat with one of conda-forge most active maintainers. </div><div dir="auto"><br></div><div dir="auto">It seems that the conda-forge community is aware of the dangers of building only bleeding edge versions of the packages. There's a CFEP (same as our QEP) to improve the situation.</div><div dir="auto"><br></div><div dir="auto"><a href="https://hackmd.io/N1hoJGJBSqGTFd83pxCyYA" target="_blank">https://hackmd.io/N1hoJGJBSqGTFd83pxCyYA</a><br></div><div dir="auto"><br></div><div dir="auto">Nevertheless, regarding the QGIS packages, and because they are not dependencies of any other packages, I was told that it would be ok to create local pinning configuration files for both our stable and LTR branches. This means we could have the control of the most important dependencies. I will try to mimic osgeo4w for 3.16 onwards, and make sure the pinning is not changed during the LTR life cycle.</div><div dir="auto"><br></div><div dir="auto">This does not fix the issue with no one maintaining the QtWebKit package, but I don't think no one really worked actively on it.</div><div dir="auto"><br></div><div dir="auto">My intentions are not to replace our packaging system (poor me), that, like Mathias said, works fine for now. I will just try to keep QGIS available on conda, as it seems useful for some folks, and it doesn't really take me too much time.</div><div dir="auto"><br></div><div dir="auto">Thanks,</div><div dir="auto"><br></div><div dir="auto">Alexandre Neto</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">A domingo, 2/08/2020, 10:04, Matthias Kuhn <<a href="mailto:matthias@opengis.ch" target="_blank">matthias@opengis.ch</a>> escreveu:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hi Ari, all<br>
</p>
<p>I can follow your reasoning. Various discussions have taken place
about this already.</p>
<p>Currently it's also a question of feasibility. Windows and macOS
have completely different dependency build chains / repositories.
Synchronizing those within their current completely separate
frameworks will be hard to achieve. (other OS'es come with their
own set of dependencies already, so we only can ignore them - or
could build complete packages including all deps in the future if
we do have a standard set).</p>
<p>I had a look at conda-forge in the past months to evaluate if it
could be considered fit for a reference build system. One
precaution I had from the start was the rolling release dependency
hell we had experienced with homebrew. It still needs to be proven
if we get away with pinning. The major roadblock I hit was already
updating Qt. I tried to lift it to the latest version with no
success (and nobody else succeeded either despite constant efforts
in the last 4 months since that -- because of CI limitations /
build issues). If we would have a proposal how to continue here
(custom hosted CI with a responsible person taking care of it?), I
am open to explore this further.<br>
</p>
<p>In summary, the current process we have works good enough, with
the occasional troubles of varying dependency versions you
mention. Changing it would be risky but possible and it is a major
effort for which we currently lack a widely accepted plan,
resources and/or (long-term committed) volunteers to do so.<br>
</p>
<p>For your particular case here, since there are no standard
versions of dependencies, conda might be as good as any other
guess. Or you go for the installer of your main target platform
instead.</p>
<p>Best regards</p>
<p>Matthias<br>
</p>
<p><br>
</p>
<div>On 8/2/20 8:49 AM, Ari Meyer wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Obrigado, Alexandre, for all the details. :-) I guess I'm
still baffled by, "Conda-forge community always tries to use
the most recent version of all dependencies." It just seems
like a "recipe" for disaster -- just built on assumptions.
Maybe you get lucky and everything just happens to work. Or
maybe you get slightly less lucky and encounter breakages
immediately, forcing you to fiddle with the build until you
find some combination that seems to work, if you can't pin the
versions. Or maybe you get unlucky and everything builds and
*seems* to work based on the extent of your testing, but your
testing may have important gaps.</div>
<div><br>
</div>
<div>In my previous contract I was using PyViz, which was
rebranded as HoloViz. As that set of libraries was immature
and constantly in flux, and we wanted to take advantage of new
features/fixes, our policy was to go with the latest
releases. I wasn't comfortable with this, esp. after enduring
many problems, but we just dealt with bugs and regressions as
they came up. QGIS, though, is much more mature, after more
than a decade since 1.0 was released (2009 according to
Wiki). I assumed, therefore, that they wouldn't trust such a
hit-or-miss strategy with dependencies. But maybe I'm just
being a worry wart. ;-) In any case, I'm going to be
hard-pressed to sell conda to my team if the dependencies are
out of sync with those of the installers.<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Jul 31, 2020 at 8:49
AM Alexandre Neto <<a href="mailto:senhor.neto@gmail.com" rel="noreferrer" target="_blank">senhor.neto@gmail.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>Hi,</div>
<div><br>
</div>
I will take this opportunity to give a brief explanation of
how conda / conda-forge works. At least what I was able to
understand so far.
<div><br>
</div>
<div>Full disclaimer: I know very little about building and
maintaining packages and installers, including the QGIS
ones, but I try to trigger the update of QGIS versions on
Conda-forge.</div>
<div><br>
</div>
<b>Conda</b> - an open source package management system and
environment management system developed mainly by anaconda.
It provides tools to build, install and run packages, and
setup environments. One interesting feature is that, in
conda, the same building recipe can be used to build the
packages for Windows, Mac and Linux. Normally, to build a
package, you use prebuild packages of its dependencies which
are available in conda online channels. This doesn't mean
that the same versions of the dependencies are used for all
platforms, as some dependency may not be available or
doesn't work for a specific platform.
<div><br>
</div>
<div>
<div><b>Conda-forge</b> - a community that maintains a big
amount of conda recipes for many open source packages,
which are stored in github repositories. Those are
called feedstocks. Among them, there is a
qgis-feedstock and also feedstocks for almost all of
its dependencies. Collaboration to keep those recipes is
normally very welcome.<br>
</div>
<div><br>
</div>
<div><a href="https://conda-forge.org/feedstocks/" rel="noreferrer" target="_blank">https://conda-forge.org/feedstocks/</a><br>
</div>
<div><br>
</div>
<div>
<div>
<div><a href="https://github.com/conda-forge/qgis-feedstock" rel="noreferrer" target="_blank">https://github.com/conda-forge/qgis-feedstock</a><br>
</div>
</div>
</div>
<div><br>
</div>
<div>All conda-forge recipes (each feedstock github
branch) are built automatically using Azure Pipelines
and uploaded to the conda-forge channel in Anaconda
cloud, making it easily deployed for installation. QGIS
has a master (stable) and a LTR branch.<br>
</div>
<div><br>
</div>
<div><b>Anaconda cloud</b> is a repository of
prebuilt packages, which is organized in
channels. Anyone can have a channel and upload their
locally built packages so that others can use them. If
you search for QGIS in Anaconda cloud you will find
several channels with several versions available.</div>
<div><br>
</div>
<div>Answering Ari question:</div>
<div><br>
</div>
<div>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Jul 31, 2020
at 6:15 AM Ari Meyer <<a href="mailto:ari.meyer@gmail.com" rel="noreferrer" target="_blank">ari.meyer@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>@Greg: I found that many of the exact
dependencies were not even available through the
main channels. So if the QGIS devs
compile/build against such versions, I'm not
sure there's an easy way to even specify such a
conda recipe, unless all those dependencies are
also made available somehow. I didn't expect
that those versions would not even be there with
the others for the various libraries. Could
this imply that the library developers don't
want those versions to be used? Not sure.<br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>I can only speak for Conda Forge and mainly QGIS
feedstock. I normally try to keep the recipes
updated for every single new version of QGIS and
QGIS LTR, but I may forget to do so. Then, sometimes
the recipes don't work out of the box for a certain
patch release of QGIS, the conda-forge folks try to
fix them, but meanwhile, a new patch release is out,
so we may just abandon that version recipe and move
on. I expect the same may happen with other
packages... There is no intention of excluding
versions for any reason.</div>
<div><br>
</div>
<div>Regarding the version of the dependencies.
Conda-forge community always tries to use the most
recent version of all dependencies. Sometimes that
doesn't work, and one needs to pin the version of a
dependency. Also, my idea was to keep the LTR main
dependencies pinned during all its lifecycle, to
avoid regressions.</div>
<div><br>
</div>
<div><b>Cool things about QGIS on conda (and
conda-forge):</b></div>
<div>- One single transparent recipe to build QGIS in
all three main platforms;</div>
<div>- As users, to install one or several packages
for a certain workflow, you normally create an
environment to isolate the install, which allows
having several versions of the same software in
parallel in your machine. Nevertheless, the packages
shared by different environments/QGIS versions are
only downloaded once and don't take more space in
the disk. Symbolic links are used.</div>
<div>- Once set, you can share an environment with
others, making it perfect for collaborative
development.</div>
<div>- Updating QGIS is like osfeo4w in all platforms,
only the necessary packages are downloads and
install.</div>
<div>- Using Conda-forge infrastructure, you only need
to update the recipe, the build and upload to
anaconda cloud are done in azure pipelines
"automagically".</div>
<div>- It's a great way to use QGIS PyQGIS and the
command line tools (the reason why I try to update
QGIS versions in conda-forge)</div>
<div><br>
</div>
<div><b>Caveats of conda-forge:</b></div>
<div>- As QGIS project, we have no direct control over
dependencies used on the master version, as the
community has the goal of using the most recent
dependency versions that is possible. The same is
not necessarily true for LTR, where I feel we would
be allowed to control and pin the dependencies</div>
<div>- Currently, there are some dependencies missing.
For example, QtWebkit, which limits QGIS
functionality, including some nice plugins like
DataPlotly, and resource sharing plugin.</div>
<div><br>
</div>
<div>It seems to me that, if the QGIS project wanted
to use conda as their main building/packaging
system, we would need to:</div>
<div>- Keep a fork of conda-forge recipes, and upload
the result to a dedicated channel, probably even
outside anaconda cloud.</div>
<div>- We could use dependencies available on
conda-forge or other channels, but we would need to
be ready to maintain abandoned packages that greatly
impact QGIS, like QtWebkit;</div>
<div>- Recreate the installers (at least Windows and
Mac) to use conda tools for downloading, installing,
and updating packages. </div>
<div><br>
</div>
<div>Last bit, when I was working for Boundless, Larry
Shaffer was trying to do all this mentioned above,
but then the company was bought by Planet... Not
sure what is the status of this work now.</div>
<div><br>
</div>
<div><a href="https://github.com/osgeo-forge" rel="noreferrer" target="_blank">https://github.com/osgeo-forge</a><br>
</div>
<div><br>
</div>
<div>Thanks, Sorry for the long post.</div>
<div><br>
</div>
<div>Alexandre Neto</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
QGIS-Developer mailing list
<a href="mailto:QGIS-Developer@lists.osgeo.org" rel="noreferrer" target="_blank">QGIS-Developer@lists.osgeo.org</a>
List info: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-developer" rel="noreferrer" target="_blank">https://lists.osgeo.org/mailman/listinfo/qgis-developer</a>
Unsubscribe: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-developer" rel="noreferrer" target="_blank">https://lists.osgeo.org/mailman/listinfo/qgis-developer</a></pre>
</blockquote>
</div>
</blockquote></div>
</blockquote></div>