[Qgis-developer] hub.qgis.org dead slow?

[Alex Mandel]

On 12/13/2012 11:53 AM, Alessandro Pasotti wrote:
> 2012/12/13 Alex Mandel <tech_dev at wildintellect.com>
> 
>> On 12/13/2012 07:29 AM, Jürgen E. Fischer wrote:
>>> Hi Andreas,
>>>
>>> On Thu, 13. Dec 2012 at 10:48:07 +0100, Andreas Neumann wrote:
>>>> I think we could easily pay from the existing QGIS funds for a dedicated
>>>> build/test server. At
>>>>
>> http://www.hetzner.de/en/hosting/produktmatrix/rootserver-produktmatrix-ex
>>>> (where I also have my private server and the GIS server of the City of
>>>> Uster) you can get dedicated root servers with lots of memory and good
>>>> CPUs for about 70-90 Euros per month. It is a very reliable company and
>>>> runs on renewable energy. Adds up to 840 to 1080 Euros per year - in my
>>>> opinion a good investment if we can get better performance for our core
>>>> services.
>>>
>>> The builds work fine and don't cause the issues (and nicing them didn't
>> help a
>>> bit) - the only problem with that, is that the diskspace runs out -
>> something
>>> was added lately that occupies space.
>>>
>>> I think it's the documentation site. But Alex is about to organize more
>> space.
>>> So that's should be shortly solved.
>>>
>>> But I'm just monitoring the stuff and try to find out what's wrong,
>> although I
>>> didn't setup any of the webservices.  So it's hard to tell what's
>> necessary and
>>> whats not and sometimes if something behaves normal or goes bezerk.
>>>
>>> There's now a script in place that restarts apache, whenever is stops
>>> responding - with a report of what was going on at the time.
>>>
>>> It's currently about cron.daily time (or was when I started this) -
>> there were
>>> four planets jobs running feedjack_update.py and apparently stressing
>>> postgresql with concurrent queries (SELECTs, INSERTs DELETEs & UPDATEs).
>>   I
>>> supposed these should be guarded to not run in parallel.  And there was
>> also an
>>> unniced backup of redmine running.
>>>
>>> There are probably more cron jobs that have that problem.  Some cron
>> jobs and
>>> webservices also don't/didn't rotate their logs (and produces/d huge ever
>>> growing logs - which apparently nobody looks at anyway).
>>>
>>> BTW The nightly builds are niced and should only run if there's nothing
>> else
>>> wanting the CPU.  See the red stuff in
>>> http://webextra.osgeo.osuosl.org/munin/osgeo.org/qgis.osgeo.org-cpu.html
>> .
>>>
>>> I for one don't want to move the build stuff to another server - I would
>> even
>>> have one (also from Hetzner ;), if I wanted to do that.  Please move
>> everything
>>> else ;)
>>>
>>> BTW can we alternatively get more CPU on our VM?
>>>
>>>
>>> Jürgen
>>>
>>
>> Yes, I'm going to request +10GB and +1 Cores, hopefully this week.
>>
>> As for the plugin repo, I think we should look at mirror/caching options
>> for the plugin repo files that are served up rather than a whole
>> separate server. Some sort of cdn service would make plugin install
>> quite quick for many people around the world and free up some resources
>> on our server. Of course we want to find a service that refreshes or is
>> easy for us to push updates quickly. Alternate to that, since several of
>> us have fast servers around with space, I could setup a mirrorbrain
>> which auto refers people to geographic closest copies of plugins (I ran
>> this for osgeo live for a while).
>>
>>
> In that case we would loose the download counter.
> 
> Are you sure that serving plugin packages is eating a significant
> percentage of resources?
> 
> 

Well, yes and no, we could look for a way that lets us keep the download
counter (fyi mirrorbrain hits the main server before delegating so you
still get hit counts).

No, I was approaching the issue from the perspective of if the plugins
download is the most critical thing to not interrupt we should find a
way to increase it's level of service no matter what the other stuff is
doing.

Thanks,
Alex