Re: [Xen-users] Cannot start domains after FC6->F7 upgrade

[Gerry Reno <greno@xxxxxxxxxxx>]

Nico Kadel-Garcia wrote:
> Gerry Reno wrote:
> > Yes, the situation for us is that I upgraded our servers by d/l the 
> > fedora-release* rpms, installing them and then doing a 'yum -y 
> > upgrade'.  And guess what?  Worked great.  Except I was not aware 
> > that the new libata drivers did not properly support our old 
> > highpoint ATA controllers.  So at first boot, instant crash.  So play 
> > around with this for a while and then start opening bugs on the F7 
> > kernels.  I had had some problems getting the 2.6.20 series kernels 
> > booting on the highpoint controllers but that was with the old IDE 
> > drivers and some parameters in the kernel had changed and changed the 
> > tolerance w/regard to our bus timing.  So we were getting 'unknown 
> > bus timing' error there.  I worked with Sergei and Chuck and we were 
> > able to solve that issue on FC6 with a BIOS tweak on our hardware.  
> > Actually overclocking the bus did the trick.  But with F7 (which I 
> > was expecting to just fix all of this), things got even worse.  The 
> > new libata drivers failed badly w/regard to the highpoint 
> > controllers.  So the only option was to boot F7 using the old FC6 
> > kernels.  This worked, or so I thought, until I began restoring the 
> > system to operational status by bringing up all the Xen guests.  
> > Kaboom!  No way could I get any of the domains to start.  So now we 
> > are really stuck.  I'm assuming that Alan and Sergei are probably 
> > trying to get libata fixed but Alan didn't seem too optimistic that 
> > this would happen soon.  I really do not want to try a bare metal 
> > restore of the server back to FC6.  So now I'm trying to figure out 
> > what other options might be in the picture.  I'm thinking along the 
> > lines of maybe seeing if VMWare could run the xen images.  I already 
> > tried some things using qemu, but the networking is way too slow.  
> > Like 5x to 10x slower than Xen.  So maybe I need to go the other way 
> > and look at openvz and just toss some processes into separate ve's 
> > and do that until things get straightened out with libata.
> Ouch. Dude, you've just spent at least a whole work day of your 
> valuable time with this, right? 
Nico, I wish it was only "a whole work day".  It's more like several days.
> Is your time worth switching to a more robustly supported ATA card, 
> like a 3Ware or Adaptec? I know the Highpoint's are very inexpensive, 
> but their "Linux support" really hasn't been reliable enough to use 
> them for boot drives.
That's not true.  We have been using the highpoint controllers for at 
least five years onboard on various M/B's as well as pci cards and they 
have been very stable.  At least until Fedora's latest kernel changes 
starting with the 2.6.20 series.  I've had dozens of kernels running 
over the years on these affected servers without any problems until lately.
> I sympathize, I really do: I've had similar issues happen with a 
> thousand servers where the "kernel team" had so customized their 
> kernel that we couldn't reliably update to a new OS and new kernel, 
> and I had to babysit it into operating with new hardware. (Hint: any 
> kernel developer who can't give you a diff between the distributed 
> kernel they started with and their new kernel, and says "we can 
> backport anything we need from the new kernels!", should be fired 
> immediately.)
For us, Fedora has created the 'perfect storm'.  We read release notes 
but didn't see any mention of lack of support for highpoint so we 
prepared all our filesystems with LABELS and then did the upgrade.  Once 
we couldn't boot the F7 kernels and had to drop back to F6 kernel then 
we run into the problem of the ABI incompatible changes between Xen 
3.0.x and 3.1.x.  Both of these problems at the same time just killed 
us.  As far as libata I think the kernel team should have left the old 
IDE drivers in the kernel along with libata and provided a command line 
switch that would let the user switch back to the old drivers if they 
had any major problems with the new drivers.  That would have provided 
many of the non-working libata cases a temporary workaround until the 
kernel team could solve these issues.


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users