[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-users] firewall xenU woes (help would be appreciated)



Hi Adam,

Adam Tworkowski wrote:
> Hi,
> 
> I seem to have made some progress.  I add the ifconfig statements for
> the bridges in /etc/network/interfaces and they come up automatically.
> 
Yes, but your dummies are not being added to the bridges with your config.

> I discovered a typo in my xen firewall config -- I was using the plural
> "vifs" instead of "vif".
> 
> ifconfig is now reporting all bridges and vifs is per Marcus' example.
> route looks similar is well.  
> 
> brctl reports multiple vifs bound to the related bridges only when I
> remove the "backend=fw" statemnent from the vif parameter in the domU1
> xen config.  

Is your firewall kernel configured for network backend?

> 
> Should I be attempting to use the "backend" parameter?

I have used the dummy+bridge system in dom0 because when using
the backend from the Firewall to domUs a new vif with the next
sequential number is created at the firewall. I found that
Shorewall didn't like this when starting & stopping domUs,
as it will class the new interfaces as undefined.
If, however, you attach the firewall to a dummy+bridge dmz
exported from dom0 you can achieve much the same thing just
by not assigning an ip at the dom0 end.
Then, each domU can attach itself to the bridge as required
without creating new interfaces, and dom0 is isolated from
the dmz.
You either need to use the dummy+bridge method OR the network
backend to achieve your design. Not both.

DUMMY + BRIDGE METHOD:
             +------------------------+
             |   +---------------+    |
 Internet -------+ Firewall domU +--------- Intranet
             |   +------+--------+    |
             |          |             |
             |          |   DMZ       |
             |    +-----+------+      |
             |    |     :      |      |
             | +--+--+  :   +--+--+   |
             | |domU1|  :   |domU2|   |
             | +-----+  :   +-----+   |
             |          :             |
             |    +-----+------+      |
             |    |   dom0     |      |
             +----+------------+------+


NETWORK BACKEND METHOD:
             +------------------------+
             | physical machine, dom0 |
             |   +---------------+    |
 Internet -------+ Firewall domU +--------- Intranet
             |   +------+--------+    |
             |          |             |
             |          |   DMZ       |
             |    +-----+------+      |
             |    |            |      |
             | +--+--+      +--+--+   |
             | |domU1|      |domU2|   |
             | +-----+      +-----+   |
             +------------------------+

        I'd have to think about how to automatically add
        new vifs to a bridge using this method, which
        may help my shorewall problem, but it will be a
        while before I visit this issue again as I prefer
        the first method.

> 
> I still cannot "see" the firewall from domu1.  
> 
> Should the vifs show up in with arp?  The fw (domU) reports its first
> hop in arp but does not report domU1.
> 
> Looking at your brctl show ... I suspect I have an issue here:
> 
> br1           vif1.2
>               vif3.0
> br2           vif1.2
> br3           vif1.3
> xen-br0               dummy0
>               vif1.0
> 
> (note that I omitted the br ids /stp info.)

Right, so I'd guess that your firewall has something weird going on here.
- You have two vif1.2 interfaces?
- The only bridge that will work is xen-br0. br1-3 have no interface attached.
- Try starting your domU on xen-br0 rather than br1 and see if you've
        got everything working. You can always move it to another bridge
        later.

> 
> I see that your bridges all have dummy interfaces reported as attached.
> As well you have multiples vifs attached to your xen-br0. Any ideas?

The dummy interface provides a network device for the bridge to attach to.
When a new domU is created, a vif is attached to the bridge specified in
the domU config.


Marcus.

> 
> I will have to throw another NIC in this machine so I can actually get
> to this machine and provide copy and past logs, etc.
> 
> Thanks
> 
> Adam
> 
> On Mon, 2005-08-29 at 18:15 +1000, Marcus Brown wrote:
> 
>>Hi Adam,
>>I've added some of my configs and thoughts below:
>>
>>Adam Tworkowski wrote:
>>
>>>Hi Marcus,
>>>
>>>I have the dummy module loaded on dom0 and several dummy interfaces
>>>which show up with ifconfig.  They are configured as follows:
>>>
>>>auto dummy0
>>>iface dummy0 inet static
>>>     address 192.168.254.1
>>>     netmask 255.255.255.248
>>>     network 192.168.254.0
>>>     broadcast 192.168.254.7
>>>     gateway 192.168.254.6
>>>     post-up brctl addbr xen-br0 || true
>>>     post-up brctl addif xen-br0 dummy0 || true
>>>     post-up brctl delif xen-br0 dummy0 || true
>>>
>>>Additional bridges configured in same manner.
>>>
>>>Without intervention dom0 boots with 4 dummy interfaces
>>>(dummy0,dummy1,etc.) and xen-br0.  This is apparent from ifconfig.
>>>The additional bridges, however, do not load w/o intervention.  I have
>>>to use the brctl command as per the above /etc/network/interfaces.
>>>
>>>In /etc/xen/scripts/network I have the bridges mapped to the dummy
>>>devices as so:
>>>
>>>bridge=${bridge:-xen-br0}
>>>netdev=${netdev:-dummy0}
>>>bridge=${bridge:-xen-br1}
>>>netdev=${netdev:-dummy1}
>>>...
>>>
>>
>>As xen brings up xen-br0 automatically, I haven't used post hooks for
>>dummy0:
>>/etc/network/interfaces includes:
>>      auto dummy0
>>      iface dummy0 inet static
>>              address         192.168.254.1
>>              netmask         255.255.255.248
>>              network         192.168.254.0
>>              broadcast       192.168.254.7
>>              gateway         192.168.254.6
>>              dns-nameservers 192.168.254.6
>>
>>      auto dummy1
>>      iface dummy1 inet static
>>              address         192.168.254.9
>>              netmask         255.255.255.248
>>              network         192.168.254.8
>>              broadcast       192.168.254.15
>>              post-up         brctl addbr br1 || true
>>              post-up         brctl addif br1 dummy1 || true
>>              post-up         ifconfig br1 192.168.254.9/29
>>              post-down       brctl delif br1 dummy1
>>
>>      auto dummy2
>>      iface dummy2 inet static
>>              address         192.168.254.17
>>              netmask         255.255.255.248
>>              network         192.168.254.16
>>              broadcast       192.168.254.23
>>              post-up         brctl addbr br2 || true
>>              post-up         brctl addif br2 dummy2 || true
>>              post-up         ifconfig br2 192.168.254.17/29
>>              post-down       brctl delif br2 dummy2
>>
>>      ... and so on
>>
>>/etc/xen/scripts/network includes:
>>      bridge=${bridge:-xen-br0}
>>      netdev=${netdev:-dummy0}
>>      antispoof=${antispoof:-yes}
>>
>>I'm not sure if multiple bridge & netdev statements work here,
>>but nice if it does.
>>
>>BTW:
>>The post hook configuring the ip of the bridge is wrong, as this
>>method only works if the bridge already has an ip.
>>It should read:
>>        post-up ifconfig br2 192.168.254.17 netmask 255.255.255.248
>>Also, I don't think it's necessary to assign ip to the dummy, but
>>I'll look into that later.
>>
>>
>>>It is worth noting that the routing table shows something like the
>>>following (having only brought up xen-br0 and xen-br1):
>>>
>>>192.168.254.16 0.0.0.0       255.255.255.248 dummy2
>>>192.168.254.24 0.0.0.0       255.255.255.248 dummy3
>>>192.168.254.0  0.0.0.0       255.255.255.248 xen-br0
>>>192.168.254.8  0.0.0.0       255.255.255.248 xen-br1
>>>0.0.0.0        192.168.254.6 0.0.0.0         xen-br0
>>>
>>>Does this look sound?
>>>
>>
>>yep,
>># route -n
>>Kernel IP routing table
>>Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
>>192.168.254.64  0.0.0.0         255.255.255.248 U     0      0        0 dummy6
>>192.168.254.72  0.0.0.0         255.255.255.248 U     0      0        0 dummy7
>>192.168.254.16  0.0.0.0         255.255.255.248 U     0      0        0 dummy2
>>192.168.254.24  0.0.0.0         255.255.255.248 U     0      0        0 dummy3
>>192.168.254.0   0.0.0.0         255.255.255.248 U     0      0        0 
>>xen-br0
>>192.168.254.8   0.0.0.0         255.255.255.248 U     0      0        0 dummy1
>>192.168.254.96  0.0.0.0         255.255.255.240 U     0      0        0 dummy9
>>192.168.254.80  0.0.0.0         255.255.255.240 U     0      0        0 dummy8
>>192.168.254.48  0.0.0.0         255.255.255.240 U     0      0        0 dummy5
>>192.168.254.32  0.0.0.0         255.255.255.240 U     0      0        0 dummy4
>>192.168.253.0   0.0.0.0         255.255.255.0   U     0      0        0 
>>dummy10
>>192.168.254.0   0.0.0.0         255.255.255.0   U     0      0        0 br1
>>192.168.254.0   0.0.0.0         255.255.255.0   U     0      0        0 br2
>>192.168.254.0   0.0.0.0         255.255.255.0   U     0      0        0 br3
>>192.168.254.0   0.0.0.0         255.255.255.0   U     0      0        0 br4
>>192.168.254.0   0.0.0.0         255.255.255.0   U     0      0        0 br5
>>0.0.0.0         192.168.254.6   0.0.0.0         UG    0      0        0 
>>xen-br0
>>
>># ifconfig | egrep "Link|addr"
>>br1       Link encap:Ethernet  HWaddr 62:3F:39:21:9D:DB
>>          inet addr:192.168.254.9  Bcast:192.168.254.255  Mask:255.255.255.0
>>br2       Link encap:Ethernet  HWaddr 82:D1:F6:2A:55:B6
>>          inet addr:192.168.254.17  Bcast:192.168.254.255  Mask:255.255.255.0
>>br3       Link encap:Ethernet  HWaddr AE:DB:D9:29:16:B8
>>          inet addr:192.168.254.25  Bcast:192.168.254.255  Mask:255.255.255.0
>>br4       Link encap:Ethernet  HWaddr 62:50:E4:EF:72:A1
>>          inet addr:192.168.254.33  Bcast:192.168.254.255  Mask:255.255.255.0
>>br5       Link encap:Ethernet  HWaddr 9E:4A:BC:41:68:A8
>>          inet addr:192.168.254.49  Bcast:192.168.254.255  Mask:255.255.255.0
>>dummy0    Link encap:Ethernet  HWaddr FA:99:8D:02:7D:FE
>>          inet addr:192.168.254.1  Bcast:192.168.254.7  Mask:255.255.255.248
>>dummy1    Link encap:Ethernet  HWaddr 62:3F:39:21:9D:DB
>>          inet addr:192.168.254.9  Bcast:192.168.254.15  Mask:255.255.255.248
>>dummy2    Link encap:Ethernet  HWaddr 82:D1:F6:2A:55:B6
>>          inet addr:192.168.254.17  Bcast:192.168.254.23  Mask:255.255.255.248
>>dummy3    Link encap:Ethernet  HWaddr AE:DB:D9:29:16:B8
>>          inet addr:192.168.254.25  Bcast:192.168.254.31  Mask:255.255.255.248
>>dummy4    Link encap:Ethernet  HWaddr 62:50:E4:EF:72:A1
>>          inet addr:192.168.254.33  Bcast:192.168.254.47  Mask:255.255.255.240
>>dummy5    Link encap:Ethernet  HWaddr 9E:4A:BC:41:68:A8
>>          inet addr:192.168.254.49  Bcast:192.168.254.63  Mask:255.255.255.240
>>dummy6    Link encap:Ethernet  HWaddr 06:EB:9E:DF:25:2F
>>          inet addr:192.168.254.65  Bcast:192.168.254.71  Mask:255.255.255.248
>>dummy7    Link encap:Ethernet  HWaddr 6E:92:58:A4:1D:C2
>>          inet addr:192.168.254.73  Bcast:192.168.254.79  Mask:255.255.255.248
>>dummy8    Link encap:Ethernet  HWaddr 32:4E:B6:5D:1B:12
>>          inet addr:192.168.254.81  Bcast:192.168.254.95  Mask:255.255.255.240
>>dummy9    Link encap:Ethernet  HWaddr 06:1E:0C:F4:5C:09
>>          inet addr:192.168.254.97  Bcast:192.168.254.111  
>> Mask:255.255.255.240
>>dummy10   Link encap:Ethernet  HWaddr E6:5B:DC:A9:F2:54
>>          inet addr:192.168.253.1  Bcast:192.168.253.255  Mask:255.255.255.0
>>lo        Link encap:Local Loopback
>>          inet addr:127.0.0.1  Mask:255.0.0.0
>>vif1.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
>>vif1.1    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
>>vif1.2    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
>>vif1.3    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
>>vif1.4    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
>>vif1.5    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
>>vif1.6    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
>>vif1.7    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
>>vif1.8    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
>>vif1.9    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
>>vif1.10   Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
>>vif2.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
>>vif4.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
>>vif5.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
>>xen-br0   Link encap:Ethernet  HWaddr FA:99:8D:02:7D:FE
>>          inet addr:192.168.254.1  Bcast:192.168.254.7  Mask:255.255.255.255
>>
>># brctl show
>>bridge name     bridge id               STP enabled     interfaces
>>br10            8000.e65bdca9f254       no              dummy10
>>                                                        vif1.0
>>br1             8000.623f39219ddb       no              dummy1
>>                                                        vif1.2
>>br2             8000.82d1f62a55b6       no              dummy2
>>                                                        vif1.3
>>br3             8000.aedbd92916b8       no              dummy3
>>                                                        vif1.4
>>br4             8000.6250e4ef72a1       no              dummy4
>>                                                        vif1.5
>>br5             8000.9e4abc4168a8       no              dummy5
>>                                                        vif1.6
>>                                                        vif2.0
>>                                                        vif4.0
>>br6             8000.06eb9edf252f       no              dummy6
>>                                                        vif1.7
>>br7             8000.6e9258a41dc2       no              dummy7
>>                                                        vif1.8
>>br8             8000.324eb65d1b12       no              dummy8
>>                                                        vif1.9
>>br9             8000.061e0cf45c09       no              dummy9
>>                                                        vif1.10
>>xen-br0         8000.fa998d027dfe       no              dummy0
>>                                                        vif1.1
>>                                                        vif5.0
>>
>>
>>
>>
>>>Also note that /proc/net/dev only shows network activity for dummy0 and
>>>xen-br0 -- none for xen-br1 even though I have pinged the associated
>>>address and attempt to have domU1 use xen-br1 (vif = 'backend=fw,
>>>brige=xen-br1' ].
>>>
>>
>>fw doesn't need to export the interface AFAIK,  as domU1 can have the
>>bridge exported to it directly from dom0.
>>eg.
>>
>>/etc/xen/Firewall includes:
>>      name = "Firewall"
>>      pci = ['00,0b,0', '00,0a,0' ]
>>      nics=11
>>      vif = [ 'mac=aa:00:00:00:22:01, bridge=br10',
>>              'mac=aa:00:00:25:40:01, bridge=xen-br0',
>>              'mac=aa:00:00:25:40:09, bridge=br1',
>>              'mac=aa:00:00:25:40:17, bridge=br2',
>>              'mac=aa:00:00:25:40:25, bridge=br3',
>>              'mac=aa:00:00:25:40:33, bridge=br4',
>>              'mac=aa:00:00:25:40:49, bridge=br5',
>>              'mac=aa:00:00:25:40:45, bridge=br6',
>>              'mac=aa:00:00:25:40:73, bridge=br7',
>>              'mac=aa:00:00:25:40:81, bridge=br8',
>>              'mac=aa:00:00:25:40:97, bridge=br9' ]
>>
>>In the Firewall, the PCI NICs become eth0/1, and the bridges are eth2...
>>
>>An example domU:
>>/etc/xen/Mail includes:
>>      name = "MailServer"
>>      vif = [ 'mac=AA:00:00:25:40:51, bridge=br5' ]
>>
>>You can tell I've gone overboard with bridges. I'll be reducing this
>>number with the latest design, but they're great to have lying around
>>for testing :) . That many interfaces causes shorewall to think for
>>30-60 secs when recreating iptables rules !!!
>>
>>Marcus.
>>
>>
>>>What else do you need from me to help troubleshoot?  
>>>
>>>Adam 
>>>
>>>On Mon, 2005-08-29 at 02:24 -0400, Marcus Brown wrote:
>>>
>>>
>>>>Hi Adam,
>>>>
>>>>Perhaps you could try loading the dummy modules in dom0 instead?
>>>>Then attach your dom0 bridges to the dummy interfaces by replacing
>>>>the netdev argument for "eth0" in /etc/xen/scripts/network with "dummy",
>>>>and create auto scripts for the dummy interfaces that create bridges.
>>>>
>>>>Sorry to be so brief ... just caught me before going to bed.
>>>>   (Trying to work on a quote :) )
>>>>If you get stuck perhaps you could provide more detail?
>>>>
>>>>Regards,
>>>>
>>>>Marcus.
>>>>
>>>>
>>>>Adam Tworkowski wrote:
>>>>
>>>>
>>>>
>>>>>Hi,
>>>>>
>>>>>Could someone please help me out with networking my firewall on xenU
>>>>>configuration.  I have combed the list archives for posts on similar
>>>>>configurations and getting bridging working properly but I am stuck and
>>>>>generally confused.  I am trying to achieve the scenario below (ascii
>>>>>borrowed from previous thread).
>>>>>
>>>>>             +------------------------+
>>>>>             | physical machine, dom0 |
>>>>>             |   +---------------+    |
>>>>>-- Internet -------+ Firewall domU +--------- Intranet
>>>>>             |   +------+--------+    |
>>>>>             |          |             |
>>>>>             |          |   DMZ       |
>>>>>             |    +-----+------+      |
>>>>>             |    |            |      |
>>>>>             | +--+--+      +--+--+   |
>>>>>             | |domU1|      |domU2|   |
>>>>>             | +-----+      +-----+   |
>>>>>             +------------------------+
>>>>>
>>>>>So far:
>>>>>
>>>>>- physical ethernet device (will be two later) hidden from dom0 (no issue) 
>>>>>- domU (fw) using ethernet device -- has access to internet on eth0 (no 
>>>>>issue)
>>>>>
>>>>>- domU1 and domU2 can only see domU when using bridge=xen-br0 although it 
>>>>>appears that only on of the two can be active as if both are pinging domU 
>>>>>there is much
>>>>>packet.  Using vif = ['mac=xx.xx..., bridge=xen-br0'"].
>>>>>
>>>>>Should domU1 and domU2 should be using xen-br0 at all or should this only 
>>>>>be for domU and the former be using xen-br1 and xen-br2 respectively.
>>>>>
>>>>>- I have attempted to have domU1 use vif = ['backend=fw'] without success
>>>>>
>>>>>- I have also attempted to created xen-br1,xen-br2 and have the non-fw 
>>>>>domUs use these: vif = ['bridge=xen-br1']
>>>>>
>>>>>- Having searched the list, I have attempted to create dummy0, dummy1,etc. 
>>>>>on the domU as suggested.  
>>>>>
>>>>>
>>>>>I am hoping that someone can lay out what they have done to get this 
>>>>>working.  My suspicion is that my bridging is not working correctly on 
>>>>>dom0.
>>>>>
>>>>>Question:  what devices should be showing from 'ifconfig' on respective 
>>>>>nodes (dom0, domU, domU1...)?  
>>>>>
>>>>>xen0 shows multiple vifs, xen-br0-4.
>>>>>xenU shows eth0 - should it show vifs?
>>>>>xenU1,2 shows eth0.  Should it show more? vifs?  
>>>>>
>>>>>Maybe someone can share the output of their ifconfigs to compare.
>>>>>
>>>>>All my instance are running on Debian Sarge.
>>>>>
>>>>>Thanks in advance!
>>>>>

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.