rsyslogd dying during heavy load

There was a case where rsyslogd (rsyslog5-5.8.12-7) was terminating when it got too many incoming logs to process and the central syslog server was not responding in timely manner.

A few extra config settings to the existing queues have helped to withstand the load:
$ActionQueueFileName logfile
$ActionQueueSize 100000
$ActionQueueSaveOnShutdown on
$ActionQueueResumeInterval 10
$ActionQueueDiscardMark 97500
$ActionQueueHighWaterMark 80000
$ActionQueueCheckpointInterval 100
$ActionQueueMaxDiskSpace 2g
$ActionQueueTimeoutEnqueue 0
$ActionQueueDiscardSeverity 0
$ActionQueueType LinkedList
$ActionQueueResumeRetryCount -1

How to fix rpmdb/yum issues

During my last yum update I ran out of memory and update has been aborted leaving rpmdb in inconsistent state.

When I tried to run it again I've got following error message:
Found 6 pre-existing rpmdb problem(s), 'yum check'
cronie-1.4.4-16.el6_8.2.x86_64 is a duplicate with cronie-1.4.4-15.el6_7.1.x86_64
cronie-anacron-1.4.4-16.el6_8.2.x86_64 is a duplicate with cronie-anacron-1.4.4-15.el6_7.1.x86_64
[..]

Quick fix:
package-cleanup --cleandupes

or:
package-cleanup --dupes
yum-complete-transaction


If you're lucky it should remove old package versions.

package-cleanup can also be executed with --problems flag.

Another cool feature of package-cleanup command is the ability to remove old kernel versions automagically:
package-cleanup --oldkernels

fail2ban not working on EL6/Centos 6

I've noticed that fail2ban do not ban every host/IP which it should after a while.

First I checked if my regexps are correct:
fail2ban-regex /var/log/maillog /etc/fail2ban/filter.d/postfix-auth.conf --print-all-matched

Fortunately I've seen a lot of hits but most of them were not banned.
It appears that fail2ban is having issues monitoring log files.
Quick fix is to change the file monitoring backend from auto to polling.
Python inotify(pyinotify) or gamin may not work properly - you can read more about it here: https://github.com/fail2ban/fail2ban/issues/44

Edit your jail.local file, at the top add the following:
[DEFAULT]
backend = polling

Restart fail2ban and check its log file to see if it bans every IP it should.

using fail2ban to protect postfix/smtpd service

Recently I have found a lot of "lost connection after" entries in /var/log/maillog file:
lost connection after AUTH from unknown[IP.address]
lost connection after CONNECT 
lost connection after RCPT
lost connection after STARTTLS  
lost connection after UNKNOWN

It's possibly some kind of botnet trying to deliver spam using my mail server.
It won't work but it's still nice to get rid of such clients on the firewall level.

First you need to create a rule for fail2ban - create /etc/fail2ban/filter.d/postfix-auth.conf and put the following config:
# Fail2ban postfix-auth filter
[INCLUDES]
before = common.conf

[Definition]
_daemon = postfix/smtpd
failregex = ^%(__prefix_line)slost connection after .*\[<HOST>\]$
ignoreregex =

Next edit /etc/fail2ban/jail.conf and add postfix-auth service at the end of file:
[postfix-auth]
enabled  = true
port     = smtp,ssmtp
filter   = postfix-auth
action   = iptables[name=SMTP-auth, port=smtp, protocol=tcp]
logpath  = /var/log/maillog
maxretry = 2
bantime = 36000
findtime = 300

Finally restart fail2ban service and check /var/log/messages or iptables to see if your new rule works fine:
service fail2ban restart
grep Ban /var/log/messages
iptables -nvL

/var/log/maillog logpath is for Centos/Redhat.
For other distros make sure to point out proper mail.log file.