[SAC] [OSGeo] #1982: osgeo6 disk full: 100% /var

[OSGeo]

#1982: osgeo6 disk full: 100% /var
---------------------------+--------------------
 Reporter:  neteler        |       Owner:  sac@…
     Type:  defect         |      Status:  new
 Priority:  blocker        |   Milestone:
Component:  Systems Admin  |  Resolution:
 Keywords:  osgeo6         |
---------------------------+--------------------

Comment (by neteler):

 Replying to [comment:3 wildintellect]:
 > So clearly the log files contribute, but I'm not sure that's the root
 cause. I think the mailman was starting to throw errors and that cascaded
 it filling it's own logs, which then caused more errors. Anyone know if
 there's a way to limit how big a mailman log will get before it cycles,
 rather than waiting for logrotate?

 AFAIK it is postfix, not mailman writing those two log files.

 > Can someone try to extract the head of the logs to figure out what
 caused the initial issue?

 Most messages are like this:

 {{{
 Aug 14 14:01:15 osgeo6 postfix/smtp[9601]: 3FFB36332EB7: host
 aspmx.l.google.com[74.125.28.26] said: 450-4.2.1 The user you are trying
 to contact is receiving mail at a rate that 450-4.2.1 prevents additional
 messages from being delivered. Please resend your 450-4.2.1 message at a
 later time. If the user is able to receive mail at that 450-4.2.1 time,
 your message will be delivered. For more information, please 450-4.2.1
 visit 450 4.2.1  https://support.google.com/mail/?p=ReceivingRate
 d11si5262351pln.414 - gsmtp (in reply to RCPT TO command)
 }}}

 Apparently we are hammering some other servers too much. At time it is a
 kind of endless loop with those servers (i.e. for some recipients).

 For now, I have added some postfix throttling:
 {{{
 /etc/postfix/main.cf

 # throttle, see https://trac.osgeo.org/osgeo/ticket/1982
 smtp_destination_concurrency_limit = 2
 smtp_destination_rate_delay = 1s
 smtp_extra_recipient_limit = 10
 }}}

 Maybe that will help to calm down google and dtag mail servers. We can
 comment out these lines once the mail queue is closer to empty. At time:

 {{{
 mailq | grep mailman-bounces at lists.osgeo.org | wc -l
 111064
 }}}

 which is 30,000 less than 11hs ago.

 > I'll check on the munin email setup, I'm not sure what it's set to do
 when it hits a limit.

 ok thx

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/1982#comment:4>
OSGeo <http://www.osgeo.org/>
OSGeo committee and general foundation issue tracker.