[Board] Travis-CI & OSGeo

Mon Apr 30 14:34:24 PDT 2018

Jachym,

Yes.  I currently manage PostGIS Jenkins, though I'd like to stream line the process so I can then apply to a more global OSGeo use.

I sadly haven't been keeping up to date with all the new features that have come into Jenkins like use of pipelines and travis like CI yaml scripts (there is even one plugin that can consume travis scripts)  which would be something that would be needed for all projects to be able to use a centralized Jenkins ci.

So I envision an Osgeo.org/Jenkins would be able to take advantage of these.  Right now the setup I have is not very extendable yet.

There is also some discussion of reviving the old buildbot and buildbot itself has changed quite a bit.

https://buildbot.net/index.html#basics

Not sure people's interest in reviving that.  I'm guessing in long run it would be easier to maintain than Jenkins, but I haven't set it up to know.

I started using Jenkins because of its WYSIWYG features that I could figure out without having to read and my inability to stand reading any documentation for longer than 5 minutes.

Thanks,

Regina

From: Jachym Cepicky [mailto:jachym.cepicky at gmail.com] 
Sent: Monday, April 30, 2018 7:14 AM
To: lr at pcorp.us
Cc: Jeffrey Johnson <ortelius at gmail.com>; OSGeo-Board <board at lists.osgeo.org>; Sandro Santilli <strk at kbt.io>
Subject: Re: [Board] Travis-CI & OSGeo

Hi,

sorry jumping so late into this discussion. 

For PyWPS, we are using travis-ci infrastructure.

But I can imagine, that having osgeo.org/jenkins <http://osgeo.org/jenkins>  instance at hand, we would move straight way. I had chance to use jenkins in the past and it works nice. And it can be used for automatic build of packages etc .. did you consider this? I hope, I have not missed someones mail to this topic already

Jachym

út 24. 4. 2018 v 7:09 odesílatel Regina Obe <lr at pcorp.us <mailto:lr at pcorp.us> > napsal:

>  In many cases the code is developed primarily on github with Travis running tests (and often other tools in the build) and then mirrored onto a private 

> gitlab and Jenkins setup where further quality assurance, release and deploy steps are run. 

That's kind of the thing I wanted to push for thus my reason for joining SAC.  We have a couple of things we have in pipe – which we will do once we get new hardware and also using infrastructure Funtoo org was offered us. I'd use PostGIS as a proof of concept though it needs some cleanup.

> I would be in favor of this kind of integrated setup for OSGeo with both sides of the equation represented, 

> but haven’t seen evidence that we are capable of organizing that, setting cross project standards and coming up with a more coherent plan that represents 

> the organization clearly and professionally. 

True admittedly there hasn't been evidence which has concerned me too, thus I wanted to be part of the solution and make evidence.

There is a certain self-defeating helplessness in projects that bothers me.  It's always a SAC does nothing for me. OSGeo does nothing for me.

In a sense it's self-defeating because if you expect nothing you will get nothing and you will frustrate those of us trying with (well they certainly don't care for my input).

On PostGIS side there is a lot of sentiment like that too – but I think hey I depend on OSGeo for a place to put my downloads, for my mailing lists, and at least currently 

Sure old ways like Travis are useful and good for the present, but I worry about too much reliance of a single company especially one that has to answer to investors who could care less about open source.  The bigness of them to me is an artificial thing, and I'd hate that to stifle smaller players with great ideas or Uni kids wanting to explore administration and testing and thinking (why bother cause GitHub/Amazon etc. has already solved the world's problems)

To Even's point:

> That's the kind of workaround/cheating which causes others to pay for resources you

> use. And thus increase their bills... and/or may decrease the quality of the free tier.

There are several schools of thought to this. Mine are the following, which is why I feel little guilt

a)  In my mind, my use is advertising they can add to their statistics.  Private companies that use my work should be footing the bill and we/they should do more to extract money from these private companies.

b)  These companies (not speaking necessarily of Travis) are venture funded, so their progression is not organic and could be stifling growth of smaller companies who can provide similar services who are growing organically.  I'd rather see many more Travis's, AppVeyors, and Github than one big one for the same reason I appreciate having thousands of cloud hosters to choose from.

> I also hear your argument that there are solutions like Drone or GitLab that use

> open source software, which is great. But at some point you need to make that

> software run on servers, that you must buy or rent, and that you must monitor, maintain, etc.

The value I see in using an open solution is not just that I can run it on my own servers and tweak it for my own needs, also it's that it is more likely to lead to me having more choices of providers I can rely on that are easy to move to. Granted this starts getting messy with the plethora of OSL licenses.

Take my favorite example of PostgreSQL – there are a ton of PostgreSQL service providers (big and small players) now and all the big ones Amazon, Microsoft Azure, Google now offer it as a service.  Most are offering PostGIS at this point too cause they can since it too is open source.  This would not be possible if PostgreSQL is closed.

So yah each has their own tweaks, but I know hey if Amazon pisses me off, Microsoft has a close enough offering I can move to.

Thanks,

Regina

Again as a project lead I’m not quite sure what it is we got for going through incubation other than more eyeballs perhaps. 

On Mon, Apr 23, 2018 at 14:50 Regina Obe <lr at pcorp.us <mailto:lr at pcorp.us> > wrote:

Tim,

Well put and sorry for letting my sanctimony get in the way of my helpfulness.

Overall I think it was a good discussion and brought out some useful approaches that all projects can benefit from.

It's good to see so much interest on gitlab (as aired here and also on postgis irc).

Thanks,

Regina

From: Tim Sutton [mailto: <mailto:tim at kartoza.com> tim at kartoza.com] 
Sent: Monday, April 23, 2018 5:13 PM
To: Vicky Vergara < <mailto:vicky at georepublic.de> vicky at georepublic.de>
Cc: Regina Obe < <mailto:lr at pcorp.us> lr at pcorp.us>; osgeo-board List < <mailto:board at lists.osgeo.org> board at lists.osgeo.org>; Sandro Santilli < <mailto:strk at kbt.io> strk at kbt.io>

Subject: Re: [Board] Travis-CI & OSGeo

Hi Regina and friends

Regarding using a discrete GitHub org, I agree this is good advice for Even. GIS.ORG <http://GIS.ORG>  has its own org in GitHub and we also benefit from exclusive use of the 5 travis jobs, autonomy over how we manage our org, and other good things.. Even: I know that Matthias Kuhn also squeezed a lot of mileage out of travis for QGIS, so you might want to chat to him and ask for any tips and tricks he might have to share. 

QGIS, by the way, is going to move to GitLab for a complex of reasons. Such moves are highly disruptive and should not be recommended lightly to a project. Especially in Even’s case who, if you follow hist activities, has only just moved GDAL over to GitHub and who is, I am sure, 100% not interested in doing the whole thing over again before another 10 years have passed :-P

Reflecting on the more ‘flaming arrow' parts of the discussion, I do think  it is good to take on board the general sentiment of the thread: if people write for help we should try to focus on solving their problems, not derail the conversation by airing views on their ‘bad’ technology choices. Even for one is, I am sure, well aware of what open source is, the benefits to humanity it offers etc. …… and also the utility he can get from a well run, stable and richly functional service like GitHub that can help him share his great work to the world.

Regards

Tim

On 23 Apr 2018, at 20:52, Vicky Vergara <vicky at georepublic.de <mailto:vicky at georepublic.de> > wrote:

Hi all,

I agree with Regina,

In my own experience with pgRouting as Community project:

We have pgRouting as a "GitHub organization" and have.

it has 3 sub-projects repositories:

pgrouting  -- very active using travis  (notice the lowercase r, so pgRouting its not only the thing used in postgreSQL)

pgRoutinglayers -- not using travis

osm2pgrouting -- somehow active using travis

in addition to that we also have:

Website

Workshop -- somehow active using travis

And a series of forks that we use to make contirbutions to other projects for example:

osgeo 

Forked from OSGeo/osgeo 

Also we have stalled ideas in other repositories under pgRouting

In total we have 15 repositories

And we can manage with the 5 concurrent jobs 

A testing build of pgRouting has 6 jobs that 5 take around 8 to 9 minutes and 1 takes less than 3 minutes.

We are making use of being open and using the "free" advantages of github.

And we are not consuming OSGeo valuable resources that can be used in other key things.

VIcky

On Mon, Apr 23, 2018 at 12:36 PM, Regina Obe <lr at pcorp.us <mailto:lr at pcorp.us> > wrote:

Apology for the flaming arrows.  Howard and I love to throw arrows at each other.

Oh where do I start here?

>From an idealist standpoint.  Yes I am crazy enough to think I can make a difference and provide something complementary to travis and appveyor.

I do not think they satisfy all needs without going thru some crazy hoops.

As strk mention gitlab is an alternative , though you may not be interested in it.

>From a pragmatic standpoint I think "why are we sanctioning spending $5000/yr in the name of OSGeo projects on something that people need to be under OSGeo github to take advantage of?".

First of all if this is something GDAL needs and Proj needs, then it should come out of GDAL's/ Proj's budget :)

So if Even were to say hay GDAL needs $5000/year for this thing, then by all means yes we should give it to him and he should put it in his budget.

GDAL deserves that much per year.  Same with Proj.

But don't make it a  "OSGeo Projects" need this.  

I don't need it I don't want it, and I don't see the point of having an OSGeo GitHub org that doesn't even reflect all our projects.

It still stands that this would be a non-issue if you guys didn't create a skeletal OSGeo github group where GDAL and proj.4 are the projects eating up all the workers.

If you each had your own project group on github you'd each have 5 workers end of story.  You wouldn't even need to waste funding on this. :)

You'd also have more room to breathe as you can create many github subprojects as QGIS has done and not crowd everyone else out :)

 <https://github.com/QGIS> https://github.com/QGIS

When people go to github to look for GDAL or proj.4 they could care less you are an OSGeo project.  They already know YOU and are looking for YOU.

If someone goes to  <https://github.com/OSGEO> https://github.com/OSGEO they think - so these are all the projects OSGeo has – NOOOO.  It's NEGATIVE advertising.

So both my idealistic and pragmatic sides are disappointed by this movement to grow the github OSGeo Org for no benefit.

Thanks,

Regina

From: Howard Butler [mailto: <mailto:howard at hobu.co> howard at hobu.co] 
Sent: Monday, April 23, 2018 1:04 PM
To: Regina Obe < <mailto:lr at pcorp.us> lr at pcorp.us>
Cc: Tim Sutton < <mailto:tim at kartoza.com> tim at kartoza.com>; Jeffrey Johnson < <mailto:ortelius at gmail.com> ortelius at gmail.com>; osgeo-board List < <mailto:board at lists.osgeo.org> board at lists.osgeo.org>; Sandro Santilli < <mailto:strk at kbt.io> strk at kbt.io>
Subject: Re: [Board] Travis-CI & OSGeo

On Apr 23, 2018, at 9:50 AM, Regina Obe <lr at pcorp.us <mailto:lr at pcorp.us> > wrote:

My issue is not that Even shouldn't be given the freedom to manage his project the way he wants.  Of course he should.

Yes it is. You're arguing that you can spend $5000 on something more worthy than supporting GDAL with the project infrastructure it wants to maintain under OSGeo's umbrella.

The point is that 

1)      He is limited because he is under the OSGEO Project infrastructure on Github.  If he were on his own project space, like PostGIS or QGIS (or Geos used to be), he wouldn't be limited by the 5 worker limit. I fail to see what benefit this Org is doing us when several of aour key projects aren't even on it (e.g. QGIS, GRASS, PostGIS) and even if they are what is the point, people should be lured to the osgeo website, not github.

This is a really good point. Why should projects that are to be on their own for infrastructure and support bother with putting anything under an OSGeo umbrella at all?

Or in a less snarky tone, OSGeo needs to decide if supporting projects with infrastructure is part of its mission. Member projects have no budgetary power to put resources into infrastructure capabilities that work for them of their choice, and there's a SAC beast that must be fed with money and take on new projects to continue to have relevance. Many projects actively avoid SAC and OSGeo resources because it is lesser quality infrastructure. It is lesser quality infrastructure because it is really damn hard to be all things to everybody. Add the fact that SAC is almost entirely unrecognized volunteer effort, and it is very difficult to succeed long term with any kind of staying power. 

GitHub, Travis, and AppVeyor are products. They cost money. They are specialized tools. They work really well. They have organizations behind them. They didn't exist in 2006 when OSGeo was formed, so we started down the path of building our own. If we started in 2018, I'm unconvinced we would have built our own.

3)      I think with $5000, that's almost the size of the osgeo budget for hardware.  I think of all the good we could do with $5000/yr and something that could help all projects not just things hosted on GitHub.

Infrastructure is *so* much more than a piece of hardware in a subsidized data center that graciously hosts us.

Like building up our own CI infrastructure that would test more than just Ubuntu.

You can use Docker with Travis to test whatever flavor you want.

And what about AppVeryor.  How much are you going to have to pay for that?  Is it under the same core limitations or will you have to shell out an additional $5000/yr for that?

Yes, I would hope so. Sandro is very vocal about his disdain for Windows, so I'm sure he will complain, but you've made good business supporting windows prisoners, so maybe you won't. 

Is OSGeo a software foundation that supports projects that make software, or is it an advocacy organization for users looking to get leverage off of free and open source software? The balance has been wildly tipped toward the latter the past 5 years... The board needs to clearly signal OSGeo's relationship to its member projects in this regard. I hope the board can ignore the flaming arrows Regina and I are shooting at each other in a battle that will never end and make a decision about OSGeo's relationship to its member projects and their infrastructure. If the relationship is "you must use SAC stuff", then the board needs to dump significantly more resources into burnishing that infrastructure to an outcome that might still be unsatisfactory. If the relationship is "use whatever you want and pay for it yourself", projects with the wherewithal to do so are going to do it outside of an organization that frankly isn't providing them with anything. If the relationship is "incubated projects get XXX dollars of infrastructure funding to spend on tools of its choosing", maybe this acts as both an incentive to incubate and a ring to reach for.

What we have now, where the answer is "volunteer your volunteers' time to build infrastructure within SAC" is not the solution.

Howard

_______________________________________________
Board mailing list
Board at lists.osgeo.org <mailto:Board at lists.osgeo.org> 
https://lists.osgeo.org/mailman/listinfo/board

-- 

Georepublic UG (haftungsbeschränkt)
Salzmannstraße 44, 
81739 München, Germany

Vicky Vergara
Operations Research

eMail: vicky at georepublic.de <http://georepublic.de/> 
Web: https://georepublic.info <https://georepublic.info/> 

Tel: +49 (089) 4161 7698-1
Fax: +49 (089) 4161 7698-9

Commercial register: Amtsgericht München, HRB 181428
CEO: Daniel Kastl

_______________________________________________
Board mailing list
Board at lists.osgeo.org <mailto:Board at lists.osgeo.org> 
https://lists.osgeo.org/mailman/listinfo/board

—

Tim Sutton

Co-founder: Kartoza

Project chair:  <http://QGIS.org> QGIS.org

Visit  <http://kartoza.com/> http://kartoza.com to find out about open source:

Desktop GIS programming services

Geospatial web development

GIS Training

Consulting Services

Skype: timlinux 

IRC: timlinux on #qgis at  <http://freenode.net> freenode.net

_______________________________________________
Board mailing list
Board at lists.osgeo.org <mailto:Board at lists.osgeo.org> 
https://lists.osgeo.org/mailman/listinfo/board

_______________________________________________
Board mailing list
Board at lists.osgeo.org <mailto:Board at lists.osgeo.org> 
https://lists.osgeo.org/mailman/listinfo/board

-- 

Jachym Cepicky
e-mail: jachym.cepicky gmail com
URL: http://les-ejk.cz
GPG: http://les-ejk.cz/pgp/JachymCepicky.pgp

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/board/attachments/20180430/9eaad17a/attachment.htm>