The above system appears to be normal to me!
I can see that you said this in at first when you opened this case:
"It appears to be I/O that is causing the problem"
How did you come to this conclusion at the first place?
Maybe if you showed the iostat output during the test will help making sure that there is something wrong with the I/O! vmstat will help as well (I/O waits!)
from the vmstat -v (0 pending disk I/Os blocked with no pbuf) command and lvmo -a (global_blocked_io_count = 0) seems to me that the disks buffers are being handled nicely! there is no blocked io on any of them!
I remember the first ouput you posted above (for the lvmo -a command) showing global_blocked_io_count = 368314 which indicates that there were some of the IOs blocked at some stage!
I think you are using two different servers for your testing isn't it? or you might rebooted the system and reset the statistics by that!
any way, let us return to the point where you said "It appears to be I/O that is causing the problem" and lets inditify whether this is the case and we should know what is causing this by that time! (I guess as you said its the sockets connections going high)
I would like to get the output of the following during the test:
vmstat 5
iostat -A 1 5
pstat -a | grep aios | wc -l
vmstat -v (before and after the test)
PS. sorry if this seems to be too much but performance tuning takes most of the administrators time!
Regards,
Khalid