Wednesday, June 26, 2013

How to add timestamp to vmstat and iostat output

vmstat does not have an option to log timestamps which may cause its logs to be difficult to interpret later.
The workaround is to use awk to generate timestamp and print vmstat output plus data:
# vmstat 1 | awk '{now=strftime("%Y-%m-%d %T "); print now $0}'
2013-06-26 21:47:41 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
2013-06-26 21:47:41  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
2013-06-26 21:47:41  1  0      0 385944  71768 517152    0    0    96    28 1018  118  3  5 89  3  0
2013-06-26 21:47:42  0  0      0 385944  71768 517152    0    0     0     0 1008   65  0  1 99  0  0

2013-06-26 21:47:43  0  0      0 385944  71768 517152    0    0     0     0 1023   73  0  0 100  0  0

Fortunately iostat on Linux have -t switch which adds timestamp for every output.
However if you're using iostat on system where -t switch is not available you may use the same trick we did with vmstat.

Wednesday, June 19, 2013

Howto reset powerpath error counters

If you had issues with I/O paths and they have been fixed you might want to reset error counters to keep a clean view.
There are two ways to achieve this:

  1. error counters will be automatically cleared during machine reboot
  2. by running powermt restore command


powermt restore will check I/O paths and update their current status, additionally it will clear Q-IOs and error counters.

# powermt display
Symmetrix logical device count=0
CLARiiON logical device count=1
Hitachi logical device count=0
Invista logical device count=0
HP xp logical device count=0
Ess logical device count=0
HP HSx logical device count=0
======================================================================
--- Host Bus Adapters ---  ------ I/O Paths -----  ------ Stats ------
###  HW Path               Summary   Total   Dead  IO/Sec Q-IOs Errors
======================================================================
   1 qla2xxx               optimal       2      0       -     0     77
   2 qla2xxx               optimal       2      0       -     0      0

# powermt restore
# powermt display
Symmetrix logical device count=0
CLARiiON logical device count=1
Hitachi logical device count=0
Invista logical device count=0
HP xp logical device count=0
Ess logical device count=0
HP HSx logical device count=0
======================================================================
--- Host Bus Adapters ---  ------ I/O Paths -----  ------ Stats ------
###  HW Path               Summary   Total   Dead  IO/Sec Q-IOs Errors
======================================================================
   1 qla2xxx               optimal       2      0       -     0      0
   2 qla2xxx               optimal       2      0       -     0      0

Wednesday, June 5, 2013

NIS server setup on RHEL/CentOS

NIS SERVER SETUP:

1. Install required packages:
# yum install ypserv portmap

2. Edit /etc/sysconfig/network and add static NIS ports:
YPSERV_ARGS="-p 841"
YPPASSWDD_ARGS="-p 842"
YPXFRD_ARGS="-p 843"

To verify currently used ports run:
# rpcinfo -p localhost

3. Open required ports in firewall. Edit /etc/sysconfig/iptables and add:
-A RH-Firewall-1-INPUT -p udp --dport 111 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp --dport 111 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp --dport 841 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp --dport 841 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp --dport 842 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp --dport 842 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp --dport 843 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp --dport 843 -j ACCEPT
# service iptables restart

4. Setup NIS domain name - edit /etc/sysconfig/network and add:
NISDOMAIN="internal"

5. Initialize NIS maps:
/usr/lib/yp/ypinit -m

6. Secure NIS - permit only trusted networks. You may allow only specified subnets using iptables (i.e. by adding -s 192.168.0.0/24 to the portmap and NIS ports) or by editing /var/yp/securenets and including permitted networks in following format:
255.255.255.0 192.168.0.0

7. Start services and include them in rc3.d scripts:
# service portmap start
# service ypserv start
# chkconfig portmap on
# chkconfig ypserv on
You may include yppasswdd and ypxfrd as well if you need them.

UPDATING NIS MAPS:

# cd /var/yp; make


NIS CLIENT SETUP:

Execute:
# authconfig --update --nisdomain=internal --nisserver=nis.example.com --enablenis

Or:
# authconfig-tui

Finally make sure that ypbind is running and it's added to rc3.d services:
# service ypbind start
# chkconfig ypbind on

VIEWING NIS MAPS:

On the client:
# getent <mapname>
# getent passwd

On any allowed host:
# ypcat -d internal -h nis.example.com <mapname>
# ypcat -d internal -h nis.example.com passwd

Sunday, June 2, 2013

Dropped rx packets in ifconfig

Default settings for NICs is good for most cases however there are times when you need to do some performance tuning.
When you start to observe increasing drops of RX packets it means that your system cannot process incoming packets fast enough. You can verify on your monitoring system to correlate this issue with increased network traffic at the same time.
# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:22:19:50:ea:76
          inet addr:192.168.x.x  Bcast:192.168.x.x  Mask:255.255.xxx.xxx
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:3208932 errors:0 dropped:19188 overruns:0 frame:0
          TX packets:1543138 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000

First verify current NIC settings:
# ethtool -g eth0
Ring parameters for eth0:
Pre-set maximums:
RX:             1020
RX Mini:        0
RX Jumbo:       4080
TX:             255
Current hardware settings:
RX:             255
RX Mini:        0
RX Jumbo:       0
TX:             255

Increasing ring buffer for rx should fix this issue:
# ethtool -G eth0 rx 512

To have this settings persistent make sure you add this command to /etc/rc.local script.