An easy way to identify what process is generating your IO Wait load is to enable block I/O debugging. This is done by setting /proc/sys/vm/block_dump to a non zero value like:
echo 1 > /proc/sys/vm/block_dump
This will cause messages like the following to start appearing in dmesg:
bash(6856): dirtied inode 19446664 (ld-2.5.so) on md1
Using the following one-liner will produce a summary output of the dmesg entries:
dmesg | egrep "READ|WRITE|dirtied" | egrep -o '([a-zA-Z]*)' | sort | uniq -c | sort -rn | head 354 md 324 export 288 kjournald 53 irqbalance 45 pdflush 14 portmap 14 bash 10 egrep 10 crond 8 ncftpput
Once you are finished you should disable block I/O debugging by setting /proc/sys/vm/block_dump to a zero value like:
echo 0 > /proc/sys/vm/block_dump
Another cool method with a perl script: http://www.xaprb.com/blog/2009/08/23/how-to-find-per-process-io-statistics-on-linux/