Securing ssh server from automated bot hacks

SSH server running on some quite popular hosting is exposed to many automated attacks these days.
There are more and more bots scanning the whole Internet and especially targeting popular hosting/VPS providers.

1. Changing the default sshd port number

In contrast to other well known services (ftp/mail/www) ssh server does not need to listen on the default port. It's being used by a fewer people, sometimes only you and changing the default port is yet another step in increasing your server security.
I have been using fail2ban software for some time. It scans the logs for failed login attempts and bans the attacker's IP if he fails to login too many times.
However, after a few days my firewall was full of banned IPs (good day - 5/10 IPs, bad day - more than 20).
From now on my ssh server is always running on a non-standard IP.
The number of failed login attempts has greatly decreased and the automated attacks are no longer spamming my logs.

Some useful settings from /etc/ssh/sshd_config file:
[root@server ~]# egrep "^Port|^PermitRootLogin|^MaxAuth"  /etc/ssh/sshd_config
Port 4321
PermitRootLogin no
MaxAuthTries 3

2. Logging incoming connections

Even with ssh daemon running on a different port you may find it useful to log all the connection attempts to it. You can catch it via following iptables rule:
iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 4321 -j LOG --log-prefix "iptables: "
Which should be inserted before accepting the connection on ssh port.
Next if you're using rsyslogd you can filter all the messages starting with "iptables :" and put them into a separate file:
[root@server ~]# cat /etc/rsyslog.d/iptables.conf
:msg, startswith, "iptables: " -/var/log/iptables.log
& ~
[root@server ~]# service rsyslog restart

[root@server ~]# cat /var/log/iptables.log
Dec 16 18:39:41 server kernel: iptables: IN=eth0 OUT= MAC=00:1c:14:01:30:de:00:16:83:76:07:29:08:00 SRC=10.23.189.14 DST=20.40.50.101 LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=31528 DF PROTO=TCP SPT=44074 DPT=4321 WINDOW=29200 RES=0x00 SYN URGP=0

Configuring kdump in RHEL

kdump may be very useful for you if you're experiencing kernel crashes and want to analyze what is really happening there (or you may pass this part to support).
First install kdump package:
# yum -y install kexec-tools
Then modify /etc/grub.conf and add "crashkernel=128M@16M" after "root=LABEL=/" for current kernel.

It's necessary to reboot the box to make the new settings active:
# /sbin/init 6

After the reboot make sure that kdump service is enabled and running:
# chkconfig kdump on
# service kdump start

You should also test if your setup is fine. Open console and enter the following command to trigger the dump:
echo "c" > /proc/sysrq-trigger

How to recover (reset) root password on Linux

Reset password before reboot

First of all, it's a good practice to reset root's password before you're going to reboot the host after doing some changes or if system was running a long time without reboot.
If it fail to boot you might be asked to provide root password for fsck or even if you try to boot in single mode.

Reset root password in single user mode

If you're lucky, and system is using default settings you may try to boot up in single user mode (hit ESC when you see grub menu, choose the kernel you want to boot, hit 'e' and add 'single' at the end of the line which starts with 'kernel...' word).

You should get a root shell (without asking you for password).

Then type:

passwd

and reboot the box.

Reset root password using init=/bin/bash

You may find your system more secure and configured to ask for password even in single user mode. In that case reboot the box and add init=/bin/bash to the line starting with 'kernel...' word.

System will boot up and launch a bash session for you.

Now, mount the root's filesystem in rw mode:

mount -o remount,rw /

and reset root's password:

passwd

Finally reboot the box and you should be able to login using new password.

How to setup single user mode to ask for password

If you would like to protect your system from allowing bad guys to logon in single user mode without password it may be easily fixed.

in RHEL5/CentOS5:

Edit /etc/inittab and add the following entry at the end of file:

~~:S:wait:/sbin/sulogin

in RHEL6/CentOS6:

Edit /etc/sysconfig/init file and change the SINGLE variable to:

SINGLE=/sbin/sulogin