Re: [Xen-devel] pv_ops routing problem?

[Michael D Labriola <mlabriol@xxxxxxxx>]

xen-devel-bounces@xxxxxxxxxxxxxxxxxxx wrote on 04/28/2010 09:21:50 AM:

*snip*>
> > > > On 04/27/2010 02:22 PM, Michael D Labriola wrote:
> > > > > Ok, I've got a problem.  My current setup is as follows:
> > > > >
> > > > > I have a pv_ops dom0 (xen/master from a few weeks ago) running 3 

> > > domUs. 
> > > > > Each domU has an eth0 interface and one or more VLAN interfaces.
> > > > >
> > > > > vm1, vm2, and vm3 are all domUs running with the same pv_ops 
> kernel 
> > as 
> > > 
> > > > > dom0.  vm1's eth0.107 interface routes through vm3 to get to 
vm2's 
> 
> > > > > eth0.1020 interface.
> > > > >
> > > > > dom0:
> > > > >   eth0:     192.168.1.20/24
> > > > >
> > > > > vm1:
> > > > >   eth0:     192.168.1.10/24
> > > > >   eth0.107: 10.7.1.1/16
> > > > >   gateway:  10.7.0.1
> > > > >
> > > > > vm2:
> > > > >   eth0:      192.168.1.11/24
> > > > >   eth0.1020: 10.20.1.1/16
> > > > >   gateway:   10.20.0.1
> > > > >
> > > > > vm3:
> > > > >   eth0:      192.168.1.12/24
> > > > >   eth0.107:  10.7.0.1/16
> > > > >   eth0.1020: 10.20.0.1/16
> > > > >
> > > > >
> > > > > >From vm1, I can ping the gateway (10.7.0.1) and vm3's 1020 
> > interface 
> > > > > (10.20.0.1), but get no response if I try to ping vm2's 
10.20.1.1.
> > > > >
> > > > > vm3 can ping both 10.7.1.1 and 10.20.1.1.
> > > > >
> > > > > Should be a pretty simple packet forwarding example, and it used 

> to 
> > > work 
> > > > > using the old xen-linux 2.6.18 kernel we were using on our 
> deployed 
> > > > > systems.  But now it's broken...
> > > > >
> > > > > I have verified that the exact same setup running on 3 physical 
> PCs 
> > > > > bare-metal works w/out any problems.  I had the problem 
initially 
> on 
> > 
> > > Xen 
> > > > > 3.4.2, but just upgraded to 4.0.0 and have the exact same 
> symptoms.
> > > > > 
> > > > 
> > > > Have you tried turning off tx checksum offload?  (ethtool -K <dev> 

> tx 
> > > off)
> > > > 
> > > > Though for me that just fixes network access between dom0 and 
domU;
> > > > external traffic normally works fine.
> > > 
> > > Jeremy, I haven't tried that yet.  I'll try it this morning, 
although 
> I 
> > > don't really expect that to fix the problem.  I'm also going to pull 

> > from 
> > > the xen/master branch again to see if that makes a difference.  My 
> last 
> > > 2.6.31.12 kernel was pulled from your tree a few weeks ago.
> > > 
> > > By the way, the packet forwarding is definitely happening on my 
router 
> 
> > > domU.  If I look at vm3's eth0 w/ wireshark, I can see the VLAN 
> > > encapsulated ICMP echo request coming in on the 107 VLAN and I can 
see 
> 
> > > that vm3 is forwarding it out over its 1020 VLAN interface. However, 

> if 
> > I 
> > > look at my dom0's vif3.0 interface with wireshark I only see the 
VLAN 
> > 107 
> > > traffic...  no sign of the 1020 forwarded packet anywhere.  It's as 
if 
> 
> > > forwarded packets aren't finding their way onto dom0's vif3.0.  I've 

> > never 
> > > seen this before, and was under the impression that dom0's vif 
> > interfaces 
> > > _WERE_ the domU's interfaces just with a different name... 
> > 
> > Should the tx checksum offload only be disabled on dom0?  And does it 
> have 
> > to get disabled before xend starts?  I've tried disabling it on just 
> dom0 
> > and also on domUs with no difference.
> > 
> > Also, I updated to the current xen/master and it behaves the same.
> > 
> > I just tried adding 107 and 1020 VLAN interfaces to my dom0 (10.7.0.2 
> and 
> > 10.20.0.2) and it turns out that dom0 also cannot ping the VLAN 
> interfaces 
> > on any of the domUs...  not sure if that's a separate issue or if it's 

> all 
> > related...
> 
> Once I lowered and re-raised dom0's VLAN interfaces dom0 could ping all 
> the domU's VLAN interfaces just fine.  Not sure why I had to do that, 
> though...
> 
> Still can't ping from vm1 to vm2 using v3 as the default gateway, 
though.
> 
> I've also tried this now on xen/stable with the exact same results.
> 
> -Mike
> 
> PS - Jeremy, when I pulled from you this morning xen/stable was actually 

> one commit ahead of xen/stable-2.6.32.x... not sure if that was 
> intentional.  ;-)

Can anyone confirm or deny that routing between VLAN interfaces on domUs 
hosted on a xen/master dom0 is indeed broken?  I'm not sure what else to 
do to troubleshoot this and it's behaving like this on all my hardware.

---
Michael D Labriola
Electric Boat
mlabriol@xxxxxxxx
401-848-8871 (desk)
401-848-8513 (lab)
401-316-9844 (cell)



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel