[GeoNode-devel] GeoNode dependencies breaking & Pypi consistency

Sat Oct 21 12:24:42 PDT 2017

I understand from Alessio's message that 2.7 already pins versions properly in install_requires, which I am glad for.

pipenv looks like it could be a nice tool, and pipfile looks like a reasonable way to replace requirements.txt, especially for cases where you want hash checking. I see no reason GeoNode or anyone else should not use it, if it is mature and its features are wanted.

However, I believe adoption of this tool stands alone as a new issue, since it does not solve strictly the same problem as reliable version pins in setup.py install_requires - the problem of making a package that is user-friendly as consumed from PyPI in any of the normal ways, not only pipenv.

Everyone who runs pip install geonode should get a working package, that also will not stop working if a dependency introduces a breaking change - whether or not they use pipenv, and I can already see use cases where one might not, even if it eventually becomes a Schelling point like setup.py and pip and install_requires.

I think having the package work well from PyPI is one important part of encouraging code sharing... in contrast with making yet another fork of GeoNode and editing it in place until it becomes hard to rebase on upstream geonode, and then eventually it has diverged so far that nobody would be willing to pay for the annoying reconciliation. Maybe we can't fix all the institutional barriers to sharing, but we can remove excuses for developers.

That is why I am personally glad this was thoughtfully fixed in 2.7 - thanks.

Sasha

On Oct 21, 2017, 9:30 AM, at 9:30 AM, Francesco Bartoli <xbartolone at gmail.com> wrote:
>A more sophisticated approach would be to follow the recommendations
>from Python Packaging Authority and evolve our workflow to be adapted
>with Pipfile and Pipfile.lock [1] which would mean to going forward and
>using pipenv [2] rather than pip. The lock file would produce a
>deterministic build for releases because at the end it uses sha256
>hashes for each package [3].
>
>But I think that would be worth having it discussed and accepted by all
>developers. I’ve never used it, looks promising though. 
>
>[1] https://github.com/pypa/pipfile
>[2] https://docs.pipenv.org/
>[3] https://docs.pipenv.org/basics.html#pipfile-lock-security-features
>
>Francesco
>
>Il giorno 20/ott/2017, alle ore 10:15, Alessio Fabiani
><alessio.fabiani at geo-solutions.it> ha scritto:
>
>> +1
>> 
>> Already like this in GeoNode 2.7+
>> 
>> https://github.com/GeoNode/geonode/blob/master/requirements.txt
>> 
>> https://github.com/GeoNode/geonode/blob/master/setup.py
>> 
>> 
>> 
>> Regards,
>> Alessio Fabiani
>> ==
>> GeoServer Professional Services from the experts! Visit
>http://goo.gl/it488V for more information.
>> ==
>> 
>> Ing. Alessio Fabiani
>> @alfa7691
>> Founder/Technical Lead
>> 
>> GeoSolutions S.A.S.
>> Via di Montramito 3/A
>> 55054  Massarosa (LU)
>> Italy
>> phone: +39 0584 962313
>> fax:     +39 0584 1660272
>> mob:   +39 331 6233686
>> 
>> http://www.geo-solutions.it
>> http://twitter.com/geosolutions_it
>> 
>> -------------------------------------------------------
>> AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
>> Le informazioni contenute in questo messaggio di posta elettronica
>e/o nel/i file/s allegato/i sono da considerarsi strettamente
>riservate. Il loro utilizzo è consentito esclusivamente al destinatario
>del messaggio, per le finalità indicate nel messaggio stesso. Qualora
>riceviate questo messaggio senza esserne il destinatario, Vi preghiamo
>cortesemente di darcene notizia via e-mail e di procedere alla
>distruzione del messaggio stesso, cancellandolo dal Vostro sistema.
>Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo
>ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse,
>costituisce comportamento contrario ai principi dettati dal D.Lgs.
>196/2003.
>>  
>> The information in this message and/or attachments, is intended
>solely for the attention and use of the named addressee(s) and may be
>confidential or proprietary in nature or covered by the provisions of
>privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New
>Data Protection Code).Any use not in accord with its purpose, any
>disclosure, reproduction, copying, distribution, or either
>dissemination, either whole or partial, is strictly forbidden except
>previous formal approval of the named addressee(s). If you are not the
>intended recipient, please contact immediately the sender by telephone,
>fax or e-mail and delete the information in this message that has been
>received in error. The sender does not give any warranty or accept
>liability as the content, accuracy or completeness of sent messages and
>accepts no responsibility  for changes made after they were sent or for
>other risks which arise as a result of e-mail transmission, viruses,
>etc.
>> 
>> On Fri, Oct 20, 2017 at 12:50 AM, Jeremiah Cooper
><jcooper at jeremiahcooper.com> wrote:
>> +1 to this.  MapStory has seen these breaking changes as well. 
>Pinning everything would help alleviate having to track down which
>dependency broke the build every time it happens.  It's happened
>several times just in the past few weeks.
>> 
>> Thanks.
>> 
>> Jeremiah Cooper
>> 
>> 
>> On Thu, Oct 19, 2017 at 5:54 PM, Travis Brundage
><tbrundage at boundlessgeo.com> wrote:
>> Hi all,
>> 
>> Recently I have seen some core GeoNode behaviour breaking due to
>changes in our dependencies. This isn't the first time it's happened.
>What happens is that we have many dependencies pulling in the latest
>version via >= in our requirements, but something changes which
>conflicts with current technology or otherwise is broken, causing core
>GeoNode to break as well.
>> 
>> In addition, I'd like for us to have consistency in regards to
>GeoNode and its sub projects in pypi. Establishing the pypi repo as
>consistent with the official repos for the dependencies will ensure
>changes are visible and (hopefully) reviewed. This will help at least
>partially with preventing broken dependencies. One possibility is
>introducing some automation here through a continuous integration tool,
>so there are regular builds to pypi straight off the repos in GeoNode.
>This method may help establish a consistent cycle and also ensure we're
>diligent about quality in changes to the code.
>> 
>> For the dependencies breaking, we can start pinning versions in our
>requirements (that is, using == instead of >= in requirements.txt). The
>upside of this is we will have consistent, expected behaviour that
>won't change due to updates. The downside of this is we won't
>necessarily have the most recent versions of dependencies, which we
>will want to occasionally update. I feel pinning versions makes sense,
>but we should make sure to update dependencies regularly. Before making
>this change, we should check what versions are actually being used and
>working with GeoNode, as >= will pull later versions, so simply doing a
>find+replace of >= to == won't be correct.
>> 
>> Cheers,
>> Travis
>> 
>> -- 
>> Travis Brundage
>> Software Engineer | Boundless
>> tbrundage at boundlessgeo.com
>> 250.888.2820
>> @boundlessgeo
>> 
>> 
>> _______________________________________________
>> geonode-devel mailing list
>> geonode-devel at lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/geonode-devel
>> 
>> 
>> 
>> _______________________________________________
>> geonode-devel mailing list
>> geonode-devel at lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/geonode-devel
>> 
>> 
>> _______________________________________________
>> geonode-devel mailing list
>> geonode-devel at lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/geonode-devel
>
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>geonode-devel mailing list
>geonode-devel at lists.osgeo.org
>https://lists.osgeo.org/mailman/listinfo/geonode-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/geonode-devel/attachments/20171021/246972ca/attachment.html>