Narrowing down processes using loads of disk I/O

iostat shows you the reads and writes

# iostat -m
Linux 2.6.28-11-generic (rant-e-minor) 12/04/2009 _i686_ (2 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
9.20 0.01 2.76 1.10 0.00 87.02

Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sda 8.26 0.19 0.03 3443 642
sda1 8.25 0.19 0.03 3442 642
sda2 0.00 0.00 0.00 0 0
sda5 0.00 0.00 0.00 0 0


Enable block dump to see which processes writes data, and then tail the kern.log or equivalent:

# echo 1 > /proc/sys/vm/block_dump
# tail -f /var/log/kern.log
Dec 4 14:09:33 rant-e-minor kernel: [18343.000899] kjournald(772): WRITE block 237973000 on sda1
Dec 4 14:09:33 rant-e-minor kernel: [18343.000902] kjournald(772): WRITE block 237973008 on sda1
Dec 4 14:09:33 rant-e-minor kernel: [18343.000905] kjournald(772): WRITE block 237973016 on sda1
Dec 4 14:09:33 rant-e-minor kernel: [18343.001365] kjournald(772): WRITE block 237973024 on sda1
Dec 4 14:09:37 rant-e-minor kernel: [18346.074288] wpa_supplicant(7871): dirtied inode 8742018 (wpa_supplicant.log) on sda1
Dec 4 14:09:37 rant-e-minor kernel: [18346.074297] wpa_supplicant(7871): dirtied inode 8742018 (wpa_supplicant.log) on sda1
Dec 4 14:09:37 rant-e-minor kernel: [18346.074299] wpa_supplicant(7871): dirtied inode 8742018 (wpa_supplicant.log) on sda1

Don't forget to 'echo 0 > /proc/sys/vm/block_dump'