Monday, September 12, 2016

rsyslogd dying during heavy load

There was a case where rsyslogd (rsyslog5-5.8.12-7) was terminating when it got too many incoming logs to process and the central syslog server was not responding in timely manner.

A few extra config settings to the existing queues have helped to withstand the load:
$ActionQueueFileName logfile
$ActionQueueSize 100000
$ActionQueueSaveOnShutdown on
$ActionQueueResumeInterval 10
$ActionQueueDiscardMark 97500
$ActionQueueHighWaterMark 80000
$ActionQueueCheckpointInterval 100
$ActionQueueMaxDiskSpace 2g

$ActionQueueTimeoutEnqueue 0
$ActionQueueDiscardSeverity 0
$ActionQueueType LinkedList
$ActionQueueResumeRetryCount -1

Saturday, September 3, 2016

How to fix rpmdb/yum issues

During my last yum update I ran out of memory and update has been aborted leaving rpmdb in inconsistent state.

When I tried to run it again I've got following error message:
Found 6 pre-existing rpmdb problem(s), 'yum check'
cronie-1.4.4-16.el6_8.2.x86_64 is a duplicate with cronie-1.4.4-15.el6_7.1.x86_64
cronie-anacron-1.4.4-16.el6_8.2.x86_64 is a duplicate with cronie-anacron-1.4.4-15.el6_7.1.x86_64

Quick fix:
package-cleanup --cleandupes

package-cleanup --dupes

If you're lucky it should remove old package versions.

package-cleanup can also be executed with --problems flag.

Another cool feature of package-cleanup command is the ability to remove old kernel versions automagically:
package-cleanup --oldkernels