[Xen-users] Re: very odd iowait problem

[Bill Davidsen <davidsen@xxxxxxx>]

Miles Fidelman wrote:
> Hi Folks,
>
> I'm experiencing a very odd, daily, high-load situation - that seems 
> to localize in my disk stack.  I direct this to the xen-users, 
> linux-raid and linux-ha lists as I expect there's a pretty high degree 
> of experience on these lists with complicated disk driver stacks.
> I recently virtualized a production system, and have been slowly 
> wringing out the bugs that have shown up.  This seems to be the last 
> one, and it's a doozie.
> Basic setup:  Two identical machines except for the DomUs they're 
> running.
> Two machines, slightly older Pentium 4 processors, 4meg RAM each 
> (max), 2 CPUs each, 4 SATA Drives each.
> Debian Lenny Install for Dom0 and DomUs (2.6.26-2-xen-686)
>
> Disk setup on each:
> - 4 partitions on each drive
> - 3 RAID-1s set up across the 4 drives (4 drives in each - yes it's 
> silly, but easy) - for Dom0 /boot / swap
> - 1 RAID-6 set up across the 4 drives - set up as a LVM PV - underlies 
> all my DomUs
> note: all the RAIDs are set up with internal metadata, chunk size of 
> 131072KB - per advice here - works like a charm
> - pairs of LVs - / and swap per VM
> - each LV is linked with it's counterpart on the other machine, using 
> DRBD
> - LVs are specified as drbd: devices in DomU .cfg files
> - LVs are mounted with noatime option inside production DomU - makes a 
> big difference
> A few DomUs - currently started and stopped either via links in 
> /etc/xen/auto or manually - I've temporarily turned off heartbeat and 
> pacemaker until I get the underlying stuff stable.
> ------
> now to the problem:
>
> for several days in a row, at 2:05am, iowait on the production DomU 
> went from averaging 10% or to 100% (I've been running vmstat 1 in a 
> window and watching the iowait column)
> the past two days, this has happened at 2:26am instead of 2:05
>
> rebooting the VM fixes the problem, though it has occured again within 
> 20 minutes of the reboot, and then another reboot fixes the problem 
> until 2am the next day
> killing a bunch of processes also fixed things, but at that point so 
> little was running that I just rebooted the DomU - unfortunately, one 
> night it looked like lwresd was eating up resources, the next night it 
> was something else.
> ------
> ok... so I'm thinking there'a cron job that's doing something that 
> eats up all my i/o - I traced a couple of other issues back to cron 
> jobs - I can't seem to find either a cron job that runs around this 
> time, or anything in my logs
> so, now I set up a bunch of things to watch what's going - copies of 
> atop running in Dom0 on both servers, and in the production DomU 
> (note: I caught a couple of more bugs by running top in a window, and 
> seeing what was frozen in the window, after the machine crashed)
> ok - so I'm up at 2am for the 4th day in a row (along with a couple of 
> proposals I'm writing during the day, and a couple of fires with my 
> kids' computers - I've discovered that Mozy is perhaps the world's 
> worst backup service - it's impossible to restore things) - anyway.... 
> 2:26 rolls around, the iowait goes to 100%, and I start looking using 
> ps, and iostat, and lsof and such to try to locate whatever process is 
> locking up my DomU, when I notice:
> --- on one server, atop is showing one drive (/dev/sdb) maxing out at 
> 98% busy - sort of suggestive of a drive failure, and something that 
> would certainly ripple through all the layers of RAID, LVM, DRBD to 
> slow down everything on top of it (which is everything)
> Now this is pretty weird - given the way my system is set up, I'd 
> expect a dying disk  that to show up as very high iowaits, but....
> - it's a relatively new drive
> - I've been running smartd, and smartctl doesn't yield any results 
> suggesting a drive problem
> - the problem goes away when I reboot the DomU
>
> One more symptom:  I migrated the DomU to my other server, and there's 
> still a correlation between seeing the 98% busy on /dev/sdb, and 
> seeing iowait of 100% on the DomU - even though we're now talking a 
> disk on one machine dragging down a VM on the other machine.  
> (Presumeably it's impacting DRBD replication.)
> So....
> - on the one hand, the 98% busy on /dev/sdb is rippling up through md, 
> lvm, drbd, dom0 - and causing 100% iowait in the production DomU - 
> which is to be expected in a raided, drbd'd environment - a low level 
> delay ripples all the way up
> - on the other hand, it's only effecting the one DomU, and it's not 
> effecting the Dom0 on that machine
> - there seems to be something going on at 2:25am, give or take a few, 
> that kicks everything into the high iowait state (but I can't find a 
> job running at that time - though I guess someone could be hitting me 
> with some spam that's kicking amavisd or clam into a high-resource mode)
> All of which leads to two questions:
> - if it's a disk going bad, why does this manifest nightly, at roughly 
> the same time, and effect only one DomU?
> - if it's something in the DomU, by what mechanism is that rippling 
> all they way down to a component of a raid array, hidden below several 
> layers of stuff that's supposed to isolate virtual volumes from hardware?
> The only thought that occurs to me is that perhaps there's a bad 
> record or block on that one drive, that only gets exercised when on 
> particular process runs.
> - is that a possibility?
> - if yes, why isn't drbd or md or something catching it and fixing it 
> (or adding the block to the bad block table)?
> - any suggestions on diagnostic or drive rebuilding steps to take 
> next?  (includings that I can do before staying up until 2am tomorrow!)
> If it weren't hitting me, I'd be intrigued by this one.  
> Unfortunately, it IS hitting me, and I'm getting tireder and crankier 
> by the minute, hour, and day.  And it's now 4:26am.  Sigh...
> Thanks very much for any ideas or suggestions.
Get some sleep, for one.

I would install and enable process accounting, turn it on at midnight 
and let it run until morning (unless you feel like staying up to 
reboot). That's at a low enough level that I would expect it to have 
information as to what's running, at least.
--
Bill Davidsen <davidsen@xxxxxxx>
 "We can't solve today's problems by using the same thinking we
  used in creating them." - Einstein


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users