Narrowing down processes using loads of disk I/O

iostat shows you the reads and writes

# iostat -m
Linux 2.6.28-11-generic (rant-e-minor) 12/04/2009 _i686_ (2 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
9.20 0.01 2.76 1.10 0.00 87.02

Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sda 8.26 0.19 0.03 3443 642
sda1 8.25 0.19 0.03 3442 642
sda2 0.00 0.00 0.00 0 0
sda5 0.00 0.00 0.00 0 0


Enable block dump to see which processes writes data, and then tail the kern.log or equivalent:

# echo 1 > /proc/sys/vm/block_dump
# tail -f /var/log/kern.log
Dec 4 14:09:33 rant-e-minor kernel: [18343.000899] kjournald(772): WRITE block 237973000 on sda1
Dec 4 14:09:33 rant-e-minor kernel: [18343.000902] kjournald(772): WRITE block 237973008 on sda1
Dec 4 14:09:33 rant-e-minor kernel: [18343.000905] kjournald(772): WRITE block 237973016 on sda1
Dec 4 14:09:33 rant-e-minor kernel: [18343.001365] kjournald(772): WRITE block 237973024 on sda1
Dec 4 14:09:37 rant-e-minor kernel: [18346.074288] wpa_supplicant(7871): dirtied inode 8742018 (wpa_supplicant.log) on sda1
Dec 4 14:09:37 rant-e-minor kernel: [18346.074297] wpa_supplicant(7871): dirtied inode 8742018 (wpa_supplicant.log) on sda1
Dec 4 14:09:37 rant-e-minor kernel: [18346.074299] wpa_supplicant(7871): dirtied inode 8742018 (wpa_supplicant.log) on sda1

Don't forget to 'echo 0 > /proc/sys/vm/block_dump'

Killing disgusting zombie processes

I had a dead rsync that refused to die even with kill -9. After some google research, I found that 'ps -l' can show parent process of a zombie.

First some background; A zombie is basically a dead process, and they reason they exist is so that the parent process can retrieve the zombie's exit status and such. If you kill the parent process, the zombie will become a child process of PID 1 (init), and init is always waiting for children to die, so that they don't remain zombies. Note, zombies consumes almost no resources.

# ps ax |grep rsync
4337 pts/3 Z 0:00 [rsync]
# kill -9 4337
# ps ax |grep rsync
4337 pts/3 Z 0:00 [rsync]
# ps -l
F S UID PID PPID C PRI NI ADDR SZ WCHAN TTY TIME CMD
4 S 0 2678 2654 0 75 0 - 1016 wait pts/3 00:00:00 bash
0 T 0 4335 2678 0 75 0 - 904 finish pts/3 00:00:00 nfs2local.sh
0 Z 0 4337 4335 0 75 0 - 0 exit pts/3 00:00:00 rsync

# kill 4335
# ps ax |grep rsync
#

Voila!

VLAN tagging in Debian GNU/Linux

$ sudo apt-get install vlan

edit /etc/network/interfaces:

auto vlan150
iface vlan150 inet static
address 172.0.0.150
netmask 255.255.255.0
vlan_raw_device eth0

As root, bring the interface up:
# ifup vlan150

You can also use vconfig and ifconfig:

# vconfig add eth0 150
Added VLAN with VID == 150 to IF -:eth0:-
# ifconfig eth0.150

Now you could try tcpdumping on it for example:

# tcpdump -i eth0 vlan 150

Remove passphrase from an openssl certificate

This bugs me so much, and yet I can't seem to remember it.

openssl rsa -in file1.key -out file2.key

Configuring CARP on Debian GNU/Linux

Two machines will share on virtual IP for failover/redundancy purposes.

The shared IP will be 192.168.162.30.
Machine 1: 192.168.162.150 master
Machine 2: 192.168.162.151 backup

Required packages: ucarp iputils-arping

##### machine1
## See manual of ucarp for more information. -v = virtual id,
## -P preempt master, -k = skew (priority if you like), etc..

# /etc/network/interfaces
iface eth0 inet static
address 192.168.162.150
netmask 255.255.255.0
network 192.168.162.0
broadcast 192.168.162.255
gateway 192.168.162.254
dns-nameservers 192.168.162.25 192.168.162.26
up ucarp -i eth0 -s 192.168.162.150 -v 150 -p secretPassword -a 192.168.162.30 \
--upscript=/etc/ucarp/vip-150.up.sh --downscript=/etc/ucarp/vip-150.down.sh \
-P -z -k 10 --daemonize
down pkill ucarp

# vip-150.up.sh
#!/bin/sh
exec 2> /dev/null

/sbin/ip addr add 192.168.162.30/24 dev "$1"
start-stop-daemon --start --pidfile /var/run/ucarp-arping.192.168.162.30 --make-pidfile --background --exec /usr/sbin/arping -- -q -U 192.168.162.30


# vip-150.down.sh
#!/bin/sh
exec 2> /dev/null

/sbin/ip addr del 192.168.162.30/24 dev "$1"
start-stop-daemon --stop --pidfile /var/run/ucarp-arping.192.168.162.30 --exec /usr/sbin/arping
rm /var/run/ucarp-arping.192.168.162.30

Now you do the same on the backup host, and of course change .150 to .151 in the example above :)

If you ping the shared IP, and bring down the masters eth0, you'll see that the backup will take over the shared IP within a second or so. you can easily verify with arp!

APACHE2 + mod_jk

# workers.properties, using one tomcat.

workers.tomcat_home=/usr/local/tomcat
workers.java_home=/usr/java/j2sdk
ps=/
worker.list=tomcat1
worker.tomcat1.port=8009
worker.tomcat1.host=localhost
worker.tomcat1.type=ajp13
worker.tomcat1.lbfactor=100

# Apache conf

JkWorkersFile /etc/apache2/workers.properties
JkLogFile /var/log/apache2/mod_jk.log
JkLogLevel errors
JkMount /*.jsp tomcat1
JkMount /*.pack tomcat1
JkMount /*.do tomcat1
JkAutoAlias /usr/local/tomcat/conf/Catalina/localhost


you might need JkMounts in the VirtualHost directive too.

NFS kernel-server

# apt-get install nfs-kernel-server

If you like me use a firewall between the networks, you need to configure NFS to use pre-defined ports (or atleast should cause it makes your life more easy), as opposed to having portmapper deciding dynamically.

Edit /etc/default/nfs-kernel-server (only showing what I have changed, I choose to beef up RCNFSDCOUNT from 8 to 32 as I have 40 machines mounting the same export):
RPCNFSDCOUNT=32
RPCMOUNTDOPTS="--port 4002"

Edit /etc/default/nfs-common:
STATDOPTS="--port 4000 --outgoing-port 4001"

Restart both nfs-kernel-server and nfs-common. Note that all clients need to have the same ports setup.

Open:
TCP ports 111, 2049, 4000 & 4002.
UDP ports 111, 794, 2049, 4000 & 4002.

Check the nfs-server from a client with rpcinfo:

# rpcinfo -p program vers proto port
100000 2 tcp 111 portmapper
391002 2 tcp 705 sgi_fam
100000 2 udp 111 portmapper
100024 1 udp 4000 status
100024 1 tcp 4000 status
100003 2 udp 2049 nfs
100003 3 udp 2049 nfs
100003 4 udp 2049 nfs
100021 1 udp 58452 nlockmgr
100021 3 udp 58452 nlockmgr
100021 4 udp 58452 nlockmgr
100003 2 tcp 2049 nfs
100003 3 tcp 2049 nfs
100003 4 tcp 2049 nfs
100021 1 tcp 38677 nlockmgr
100021 3 tcp 38677 nlockmgr
100021 4 tcp 38677 nlockmgr
100005 1 udp 4002 mountd
100005 1 tcp 4002 mountd
100005 2 udp 4002 mountd
100005 2 tcp 4002 mountd
100005 3 udp 4002 mountd
100005 3 tcp 4002 mountd

Now just setup some exports in /etc/exports and run exportfs and mount it, don't forget to add the shares to fstab for mounts upon boot!

Bonding on debian

apt-get install ifenslave-2.6

Then add following to /etc/network/interfaces:
auto bond0
iface bond0 inet static
address 10.12.12.163
netmask 255.255.255.0
network 10.12.12.0
gateway 10.12.12.254
slaves eth0 eth1
bond_mode active-backup
bond_miimon 100
bond_downdelay 200
bond_updelay 200

Now just bring the interface up!


NOTE:
For Etch, you need to add the following lines to /etc/modprobe.d/arch/i386

alias bond0 bonding
options bonding mode=1 miimon=100 downdelay=200 updelay=200

Don't forget to run update-modules & bring up the interface!

mdadm + lvm2

1) Create partitions on the disks.

2) Create the md0 device with proper raid level and the disks
# mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2 /dev/sdb1 /dev/sdc1

3) Create physical volumes
# pvcreate md0

4) Create the volume
# vgcreate myvolume /dev/md0

5) Display the volume, note the PE size
# vgdisplay lvm-raid
Free PE / Size 119234 / 465.76 GB

6) make a file system on it
# lvcreate -l 119234 myvolume -n myraidname

7) create the filesystem on the raid volume
# mkfs.ext3 /dev/myvolume/myraidname

9) Add the raid device to mdadm.conf, so it's recognized next time you boot
mdadm -Es | grep md0 >>/etc/mdadm.conf


From http://en.wikipedia.org/wiki/Mdadm

View the status of a multi disk array.
# mdadm --detail /dev/md0

View the status of all multi disk arrays.
# cat /proc/mdstat

Using TPM in Linux

Nothing for me to add, found a really good guide:

https://www.grounation.org/index.php?post/2008/07/04/8-how-to-use-a-tpm-with-linux

Using stunnel to connect to gmail imaps/pop3s

# Useful for clients/daemons that can't imaps/pop3s.
# Axigen mail server can't migrate imaps mailboxes for example.

# stunnel.conf
sslVersion = SSLv3

socket = l:TCP_NODELAY=1
socket = r:TCP_NODELAY=1

debug = 0
output = /var/log/stunnel4/stunnel.log

client = yes

[pop3s]
accept = 110
connect = pop.gmail.com:995

[imaps]
accept = 143
connect = imap.gmail.com:993

Generate .CSV file of MySQL results

SELECT foo, bar INTO OUTFILE '/some/file.csv' FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\n' FROM t1 WHERE ...

iptables NAT port forward

# The port forwarding
iptables -t nat -A PREROUTING -i ethX -p tcp -m tcp --dport 12345 \
-j DNAT --to-destination 10.10.10.10:12345

# The masquerade
iptables -t nat -A POSTROUTING -o ethX -j MASQUERADE

# Don't forget to turn on ip_forwarding:
sysctl -w net.ipv4.conf.ethX.forwarding=1

Self-signed Apache style SSL Certificate

# Generate the key
openssl genrsa -des3 -out myhost.com.key 1024

# Generate the Certificate Signing Request
openssl req -new -key myhost.com.key -out myhost.com.csr

# Generate a Self-Signed SSL Certificate
openssl x509 -req -days 365 -in myhost.com.csr -signkey myhost.com.key -out myhost.com.crt

create a ramdisk for fast read/write access

ramfs grows dynamically, tmpfs doesn't. tmpfs uses the swap if you exceed the size specified, while ramfs doesn't. For an application like varnish, you can set a fixed size of the cache, so it _should_ not be a problem.

# example of mount
mount -t ramfs none /tmp/varnish -o size=1024m

# for fstab
cache /tmp/varnish ramfs defaults 0 0