OpenVZ VLAN

From Gentoo Linux Wiki

Virtualization TOC

Virtualization Paravirtualization Xen DomU creation Running Xen Troubleshooting Links and resources KVM QEmu VirtualBox VMware VMware Server VMware Workstation Gentoo guest install MS VirtualPC addons Container-based virtualization LXC OpenVZ VLAN

[edit] What you can get from this HOWTO?

You can have multiple VLANs attached to OpenVZ installation. All VLANs can be attached to one one more NICs and all VLANs can be accessible on any VE.

Main motivaion for the HOWTO was lack of clear documentation on OpenVZ wiki.

[edit] Theory

The theory is simple, but the procedure should be done in the following order:

• setup network in HN - without VLANs, bridges, nothing. I've started from scratch and used one network interface for management (it is eth0), and second for VLANs - it's called trunk (in example it is eth1).
• add VLAN to eth1 interface in HN. New interface will be created (probably eth1.[number-of-the-vlan], but this depends on implementation/distro),
• add veth devices to VE,
• put some defaults to /proc filesystem (this was tha magic I didn't know before I've started)
• create bridge in HN and bridge veth interface with newly created interface (probably eth1.[number-of-the-vlan]),
• put the stuff all together to start at boot time.

Each of steps could be achieved in several different ways, this would be best to get several most polished versions :-)

[edit] Two (or many) VLANs on one interface in one VE

Ok, now the real example. Let's start with two nic (eth0, eth1), two VLANs (20,33) and one VE (110). First, you should create two bridges, which names can correspond to number of the VLAN it is created for (but of course the number could be the name of your girl or dog):

ln -s /etc/init.d/net.lo /etc/init.d/net.br20
ln -s /etc/init.d/net.lo /etc/init.d/net.br33

Then setup vlans on the interface - if you're on Gentoo just edit your /etc/conf.d/net file:

# vlans:
vlans_eth1="20 33"
config_eth1=( "null" )
vconfig_eth1=( "set_name_type DEV_PLUS_VID_NO_PAD" )
config_eth1_20=( "null" )
config_eth1_33=( "null" )
# And this is your management network:
config_eth0=("126.66.64.151 netmask 255.255.254.0")
routes_eth0=("default via 126.66.65.249")
# now bridges for VLANs (bridges should NOT have ip address in HN; they could have but then routing can go hell)
bridge_br20="eth1.20"
bridge_br33="eth1.33"

Then you need to make sure all bridges start at boot before vz starts. You should change to:

RC_NET_STRICT_CHECKING="yes"

in /etc/conf.d/rc; this will make sure that all rc configured network interfaces are up before other rc services depending on net (like /etc/init.d/vz) will get started.

Alternative: If you got troubles with bridge and vz startup sequence, instead of using standard services you can use /etc/conf.d/local.start:

/etc/init.d/net.br20 start
/etc/init.d/net.br33 start
sleep 2
/etc/init.d/vz start

OK! now when you reboot you should have eth0, eth1, eth1.20 and eth1.33, as well as br20 with eth1.20 bridged, and br33 with eth1.33 bridged. Enter ifconfig -a and brctl show to check this. All those should still be without any configuration. Next thing is to add virtual ethernet (veth) interfaces to your existing VE:

vzctl set 110 --netif_add eth0 --save
vzctl set 110 --netif_add eth1 --save

Now, in your HN system there are two new interfaces which are named veth110.0 and veth110.1 . They reflect veth interfaces in VE and I guess you may think about them this way.

Yeah, here comes the tricky part. You need to put some stuff on before a VE starts. I'll show you how to do this automagically on each VE start, VZ reboot, system reboot etc. First, create a script in file /etc/vz/conf/vnetwork.sh:

add_interfaces_110(){
   brctl addif br20 veth110.0
   if [ $? -ne 0 ]; then echo "veth110.0 added to br20."; else echo "Error adding veth110.0 to br20"; fi
   brctl addif br33 veth110.1
   if [ $? -ne 0 ]; then echo "veth110.1 added to br33."; else echo "Error adding veth110.1 to br33"; fi
   ifconfig veth110.0 0
   ifconfig veth110.1 0
}
set_interfaces() {
ls -1 /proc/sys/net/ipv4/conf/ |
while read x
do
   echo "1" >  /proc/sys/net/ipv4/conf/$x/proxy_arp
   echo "1" >  /proc/sys/net/ipv4/conf/$x/forwarding
done
}
status() {
ls -1 /proc/sys/net/ipv4/conf/ |
   while read x
   do
   echo "$x proxy_arp set to:" & cat /proc/sys/net/ipv4/conf/$x/proxy_arp
   echo "$x forwarding set to:" & cat /proc/sys/net/ipv4/conf/$x/forwarding
   done
   }
add_interfaces_110
set_interfaces

Yes, this stinks, but works anyway :-] What is done here:

• add_interfaces_110() bridges veth interfaces with proper bridges. Bridges are already include vlan interfaces, remember that? You should do the same function (or rewrite the stuff) for every single VE you add VLANs to.
• ifconfig veth110.0 0 - I don't really know what this does, this was a curse. It really adds zero config to interface, but makes is visible through /proc filesystem. Nightmare.
• set_interfaces() sets /proc values which are in place at last,
• status() is obsolete but I've left it if someone would need to mess around.
• add_interfaces_110 set_interfaces run functions.

ps1 The:

echo "1" >  /proc/sys/net/ipv4/conf/$x/forwarding

is not actually needed if you already have

net.ipv4.ip_forward = 1

is /etc/sysctl.conf as recommended by offical openvz documentation.

ps2 I doubt that

echo "1" >  /proc/sys/net/ipv4/conf/$x/proxy_arp

is actually needed, at least I've never used it and everything works fine without proxy arp. I even have:

net.ipv4.conf.default.proxy_arp = 0

in /etc/sysctl.conf ,so I'd be glad to know why is it better to enable proxy_arp.

OK, but when or how to launch this script? there is a file /etc/vz/dists/gentoo.conf whis is being used if and only if template of the VE is Gentoo (check your /etc/vz/conf/110.conf). If you got other distro inside VE, edit your distro script file in the /etc/vz/dists/ dir. So lets edit this file and add the line:

POST_CREATE=postcreate.sh

Ok, now edit /etc/vz/dists/scripts/postcreate.sh file and add another simple function:

function bridge2hn()
{
   /etc/vz/conf/vnetwork.sh
}

...and on the and of the file:

bridge2hn

This file is being sourced several times on VE start. It's true, it throws some warnings/errors, but neverthless, it always work.

And finally, go to the VE and configure network interfaces like usual Gentoo machine (VE doesn't even know its on a trunk!).

[edit] Aternative setup to /etc/vz/conf/vnetwork.sh

The setup with /etc/vz/conf/vnetwork.sh is quite cumbersome and there is a better way to setup this using a /etc/vz/vznet.conf config file specifically made for setting up VE networking on VE startup. With this setup it does not matter if you assign several vlan interfaces or normal interfaces to one VE.

#!/bin/bash
EXTERNAL_SCRIPT="/etc/vz/bin/vbridge.add"

This script is run for every ifname (separated by ;) defined in NETIF="" in /etc/vz/conf/${VEID}.conf Where ifname has the following spec:

ifname=<ifname>,mac=<mac>,host_ifname=<host_ifname>,host_mac=<host_mac>

and in our case would look like this deined in /etc/vz/conf/110.conf

NETIF="ifname=eth0,mac=<mac1>,host_ifname=veth${VEID}.0,host_mac=<host_mac1>;
       ifname=eth1,mac=<mac2>,host_ifname=veth${VEID}.1,host_mac=<host_mac2>"

The $EXTERNAL_SCRIPT accepts host_ifname as it's argument which you can use to add to bridge. Sample $EXTERNAL_SCRIPT is:

#!/bin/bash
if [ $# != 3 ] || [ "$1" != "init" ] || [ "$2" != "veth" ]; then
       echo "ERROR: Bad arguments"
fi
host_ifname=$3
CONFIGFILE=/etc/vz/conf/$VEID.conf
. $CONFIGFILE
if [ ! -n "$NETIF" ]; then
        echo "ERROR: According to $CONFIGFILE VE$VEID has no NETIF configured."
        exit 1
fi
if [ ! -n "$BRIDGEDEV" ]; then
        echo "ERROR: According to $CONFIGFILE VE$VEID has no BRIDGEDEV mapping not defined."
        exit 1
fi
for bridge_and_host_ifname in $(echo $BRIDGEDEV | tr -s ';' ' '); do
        bridge=$(echo $bridge_and_host_ifname | awk -F':' '{print $1}')
        if [ "X${host_ifname}" == "X$(echo $bridge_and_host_ifname | awk -F':' '{print $2}')" ]; then
                [ "X${bridge}" == "X" ] && exit 0
                ifconfig $host_ifname 0
                if [ $? != 0 ]; then
                        echo "ERROR: Failed to setup interface $host_ifname"
                        exit 1
                fi
                brctl addif $bridge $host_ifname
                if [ $? != 0 ]; then
                        echo "ERROR: Adding interface $host_ifname to $bridge bridge failed"
                        exit 1
                fi
                exit 0
        fi
done
echo "ERROR: host_ifname $host_ifname does not appear in BRIDGEDEV conf"
exit 1

The above script uses the BRIDGEDEV customization variable in /etc/vz/conf/${VEID}.conf

BRIDGEDEV="<bridge0>:veth${VEID}.0;<bridge1>:veth${VEID}.1"

in our case it is:

BRIDGEDEV="br20:veth${VEID}.0;br23:veth${VEID}.1"

As you can see with this setup it does not matter if you use vlan or normal lan interfaces, the configuration will be the same, except etc /etc/conf.d/net of course.

Also if you are using vzctl-3.0.23 you can now bind VE interfaces to specific bridge with --netif_add

vzctl set 110 --netif_add eth0,<mac>,host_ifname=veth${VEID}.0,<host_mac>,br20
vzctl set 110 --netif_add eth1,<mac>,host_ifname=veth${VEID}.1,<host_mac>,br23

or with NETIF="" in /etc/vz/conf/${VEID}.conf

 NETIF="ifname=eth0,mac=<mac1>,host_ifname=veth${VEID}.0,host_mac=<host_mac1>,br20;
       ifname=eth1,mac=<mac2>,host_ifname=veth${VEID}.1,host_mac=<host_mac2>,br23"

Now with the bridge binding the $EXTERNAL_SCRIPT can be simplified and there is not need for aditional custom BRIDGEDEV configuration variable. I'll post the updated $EXTERNAL_SCRIPT for vzctl-3.0.23 as soon as I'll actually upgrade to this version :)

EDIT: and here is a modified script for vzctl-3.0.23

 #!/bin/bash
 if [ $# != 3 ] || [ "$1" != "init" ] || [ "$2" != "veth" ]; then
         echo "ERROR: Bad arguments"
 fi
 host_ifname=$3
 CONFIGFILE=/etc/vz/conf/$VEID.conf
 . $CONFIGFILE
 if [ ! -n "$NETIF" ]; then
         echo "ERROR: According to $CONFIGFILE VE$VEID has no NETIF configured."
         exit 1
 fi
 for if_config in $(echo $NETIF | tr -s ';' ' '); do
         [ "X${host_ifname}" != "X$(echo $if_config | sed -e 's/^/,/' -e 's/$/,/' -e 's/^.*,host_ifname=\([^,]*\),.*$/\1/')" ] && continue
         bridge=$(echo $if_config | sed -e 's/^/,/' -e 's/$/,/' | sed -n -e 's/^.*,bridge=\([^,]*\),.*$/\1/p')
         if [ "X" == "X${bridge}" ]; then
                 echo "ERROR: According to $CONFIGFILE VE$VEID has not specifed bridge name for $host_ifname in NETIF."
                 exit 1
         fi
         ifconfig $host_ifname 0
         if [ $? != 0 ]; then
                 echo "ERROR: Failed to setup interface $host_ifname"
                 exit 1
         fi
         brctl addif $bridge $host_ifname
         if [ $? != 0 ]; then
                 echo "ERROR: Adding interface $host_ifname to $bridge bridge failed"
                 exit 1
         fi
         exit 0
 done
 echo "ERROR: VE$VEID has not configured HN host_ifname $host_ifname in NETIF in $CONFIGFILE"
 exit 1

Also if you are using vzctl-3.0.23 and the last version of script you can now bind VE interfaces to specific bridge with --netif_add

vzctl set 110 --netif_add eth0,<mac>,host_ifname=veth${VEID}.0,<host_mac>,bridge=br20
vzctl set 110 --netif_add eth1,<mac>,host_ifname=veth${VEID}.1,<host_mac>,bridge=br23

or with NETIF="" in /etc/vz/conf/${VEID}.conf

 NETIF="ifname=eth0,mac=<mac1>,host_ifname=veth${VEID}.0,host_mac=<host_mac1>,bridge=br20;
       ifname=eth1,mac=<mac2>,host_ifname=veth${VEID}.1,host_mac=<host_mac2>,bridge=br23"

[edit] Two (or many) VLANs on one interface in two (or many) VEs

Configuring second and next VEs goes exactly the same, but you don't need to mess with those last files except you need separate add_interfaces_xxx function for each VE.

Ok, so - thats it. If somebody got stuck with this quick procedure, let me know (on discussion page?). I've got 8 machines on this HE, 9 VLANs and all goes smoothly without any troubles :-) EDIT: alternate version, yay! great!