[SAC] QGIS servers

[Alex Mandel]

On 09/24/2013 07:10 AM, Richard Duivenvoorde wrote:
> Hi List,
> 
> last year the QGIS virtual machine has been heavily used for:
> - our issuetracking system (redmine)
> - wiki (redmine)
> - building documentation (sphinx)
> - bulding website (sphinx based)
> - building nightlies
> etc etc
> - our jenkins server is run by somebody else
> 
> Currently disk is full (this will be handled today, hopefully), but in
> general we suffered from limited resources on that server.
> Eg, hanging redmine processes, made our site unavailable etc etc
> (not talking about outages from last days, that is just the 2.0 downloads)

I'll point out, we have increased the disk before. It's simply been a
matter of waiting for QGIS to ask. Turns out we have ~200G disk, 50% of
our RAM on osgeo4 unused. 60% of the QGIS disk is actually for builds,
so if we move that off, there shouldn't be a disk crisis for years.

> Ideally all this different beasts would be separate processes OR machines:
> - one java one for jenkins et al.

Falls in line with a build server that has been discussed recently with
postgis team et al (M. Smith offered one)

> - one ruby one for redmine et al

I'd like to see more work done to optimize this setup and keep it up to
date. I see a value here because it keeps some expertise on alternates
to trac available and there is much discussion of git that could be
shared. I briefly looked into it and passenger does have some parameters
similar to mod_wsgi Daemon mode and we haven't tried them yet. Note we
are also increasing RAM on QGIS this week, I think this will help keep
it from crashing as often.

> - one python one for sphinx (building/serving docs and sites 2 times a day)

Many other projects are also using sphinx: OSGeoLive, Mapserver, etc....
A whole machine just for building 2x a days seems a little much. OSGeo
Live builds once an hour when getting close to release, though are docs
are smaller and slightly fewer languages. Seems like plenty of overlap
with existing infrastructure.

> - one dev one (for building/nightlies etc)
> 

See comments about build server above. I think other projects could
benefit from a shared service.

> We have been investigating 'Dockers' to ease this separation, and make
https://www.docker.io/ ah very similar to vagrant and other VM creation
tools. Copying a whole VM in and out is not hard.

> it easier for (future) QGIS devs to create a local development
> environment. Nice thing about docker (if I understand correctly), is
> that it runs beautifully on on a big server, and can be tweaked and
> configured from all sides.
> I also investigated the costs of running a rented server (eg at hetzner:
> 2x2Tb disk, 32 gb RAM, i7-8core etc for 49 euro).
> 
1 year of hosting on this = purchase price of such a machine.

> Question to OSGeo SAC:
> - what is the position/idea of OSGeo in this?
> 
> Should a (pretty demanding?) question like this being served by OSGeo
> server(s). Or should the OSGeo hardware be reserved for smaller/starting
> projects.
> 
> Or practical:
> 
> Can OSGeo provide us either
> one big bare-metal machine, so we could try/do the Docker scenario
> (preferred option!!)
> OR
> provide us with enough virtual machines so we could partition different
> beasts better
> OR
> should we try to fund a hetzner server for QGIS (actually: "should OSGeo
> try compete with commercial services like Hetzner for this kind of
> questions").
> 
> Please let us know what you think so we can proceed our quest for world
> domination ;-)
> 

I actually think the current OSGeo machines have the capacity to handle
all of this and the discussion over build machines might actually pan
out if a couple more people (perhaps from qgis) volunteer to help
maintain build services. I'm going to propose once the backups are moved
to the new Backup machine, that we use the freed up resources for a
general building machine.

Thanks,
Alex