Subversion hosting, CVS hosting, Trac hosting, Bugzilla hosting and software collaboration Providing hosted Subversion, CVS, Trac and Bugzilla repositories

April 18, 2008

Routing Between Virtual Machines on Separate Physical Servers

Filed under: Operating Systems — Tags: , — Greg Larkin @ 5:30 pm

Hi everyone,

A while back, I was setting up some virtual machines, each with a public and host-only network interface. I like using the host-only interface for high-performance VM-to-VM communications and then to expose the WAN IP to the Internet for public access.

I soon ran into a problem, though. What happens if a VM on one physical server needs to make an internal connection to one on another physical server? The first VM has to use the public IP address of the second VM, but I still want the connection to be essentially private and not route the packets outside of the firewall-protected LAN.

Here’s a diagram showing the VMware Server configuration with virtual machines and host-only networks (click to enlarge):

Network Diagram in VMware Server Environment

For the following examples, let’s use these IP addresses for the physical servers and virtual machines:

Hostname WAN IP LAN IP Gateway IP

Notice that the LAN IPs are repeated for VMs on different physical hosts. That’s because the VMs are connecting to a host-only network on each physical machine and are independent of each other. Therefore, they can use the same IP numbers without conflicting.

Ok, so assume VM1 is a public-facing web server, and VM4 is a database server with no external services exposed except for ssh. Since there’s no way for VM1 to use VM4′s LAN IP address to connect to the database – the IPs are the same – VM1 has to use VM4′s WAN IP.

Let’s now trace where a packet as it originates from VM1 bound for VM4. This table shows the path that the packet takes, along with the source address as it passes through each hop.

Hop # Hostname Source Addr Dest Addr
1 VM1
2 PH1
3 GW1

Woops – the packet has made it out to the gateway router (GW1)! That’s not what we want, because the packets exchanged between any VM should stay on the inside of the firewall. The same problem occurs if any VM on one physical machine connects to a VM on the other physical machine.

Luckily, this easily fixed by adding some static routes to each physical machine to help guide the packets where they need to go. Instead of allowing packets bound for a VM to use the default route and get sent to the gateway router, a static route redirects them to the next hop that we specify.

On RHEL4, the /etc/sysconfig/static-routes file controls any required extra routes. On PH1, the file will look like:

any host gw
any host gw
any host gw

The file on PH2 looks like:
any host gw
any host gw
any host gw

This simply means “when sending a packet to host X (e.g., first send it to host Y (e.g., because it has the information about how to get there”.

Once the static routes are installed, the packet trace now looks like:

Hop # Hostname Source Addr Dest Addr
1 VM1
2 PH1
3 PH2
4 VM4

Of course, this can be extended across many virtual machines and physical machines, and it always helps to trace the packet and the network routes at each hop to determine where it’s going next. The traceroute command is your friend!

Call me - Greg Larkin: error

March 17, 2008

Setting Up Outbound Connections on a VMware Host-Only Network

Filed under: Operating Systems — Tags: , — Greg Larkin @ 9:23 am

Hi everyone,

I was recently reminded of a problem I ran into when I first set up the service on VMware Server. The VMware technology has the concept of virtual networks, including a host-only network. The host-only network enables several VMs on the same host to communicate on their own private Ethernet switch. This is a great way to simulate a real-world, production environment.

However, what if you want resources on the host-only network, such as private servers without routable public IP addresses, to be able to make outbound connections to the outside world? That’s where it gets a bit tricky! The service assigns a FreeBSD jail to each client, and these servers each have an IP address on the host-only network. They need to make connections to the public Internet, so after some digging around, I found a solution.

The first thing to do is assign a host-only network gateway address to your VMware Server host. In my case, the host-only network is addressed as The physical server’s gateway address is therefore

Each VM has 2 NICs defined, one with a routable Internet address and one with a host-only address, perhaps A FreeBSD jail running inside a VM will have a host-only IP address aliased to the 2nd NIC, such as Somehow, a packet originating in the jail must pass out through the VM, then the physical host to the destination and back again.

The FreeBSD VM can easily send its packets out to the host-only address on the physical host by using this directive in its /etc/rc.conf file:


Since the jail IP addresses are aliased to the host-only NIC in the VM, packets originating from a jail will also use as their default router.

At the physical host level, in order for packets to pass from its host-only interface to its external interface, it must be configured as a router. That’s done by adding the following directive to the /etc/sysctl.conf file (on RHEL 4 and other flavors of Linux):

net.ipv4.ip_forward = 1

So far, so good. Now here’s where the craziness, errr magic, happens! The following firewall script is added to /etc/rc.local:

IF_PUB1=eth0              # Public Ethernet interface of VMware Server host
IP_PUB1=AAA.BBB.CCC.11    # Public IP of VMware Server host
NET_PRV1=   # VMware Server host-only network
$IPT -F -t nat
$IPT -F -t mangle
$IPT -F -t filter
$IPT -t nat -A POSTROUTING -s $NET_PRV1 -o $IF_PUB1 -j SNAT –to $IP_PUB1

The most important bit of this script is the last line. The rest of it defines some variables and cleans up the firewall rules to a known state. Since a hardware firewall is doing all of the heavy lifting in front of this server, the iptables software firewall is going to serve simply as a source address packet mangler. Hmm, “packet mangling” – that sounds nasty! But it’s actually a good thing here, because it ensures that packets get from point A to B and back again.

We also need to look at the routing table to figure out how source address mangling will change how the packet behaves:

Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
AAA.BBB.CCC.12  AAA.BBB.CCC.11 UGH       0 0          0 eth0
AAA.BBB.CCC.8 U         0 0          0 eth0   U         0 0          0 vmnet1         AAA.BBB.CCC.9         UG        0 0          0 eth0

Normally, if a packet originates from the vmnet1 interface (VMware host-only network) with a source address of and bound for (, it will be handled by the default route and sent out the eth0 interface. That’s all well and good, but when Google tries to reply, a router somewhere along the way sees a source address of and drops the packet.

Enter POSTROUTING and SNAT! Here is the expanded iptables command from our script above:

/sbin/iptables -t nat -A POSTROUTING -s -o eth0 -j SNAT –to AAA.BBB.CCC.11

Ok, let’s break it down:

  1. Add a rule to the nat table (“-t nat”)
  2. Append the rule to the POSTROUTING chain (“-A POSTROUTING”) – i.e. apply rule after deciding which route will handle the packet
  3. Process the packet when its source address originates on the host-only network (“-s”) and it’s bound for the eth0 interface (“-o eth0”) – normally a bad thing!
  4. Jump to the SNAT target for source address modification (“-j SNAT”)
  5. Change the source address to AAA.BBB.CCC.11 and send the packet on its merry way (“--to AAA.BBB.CCC.11”)

This means that when the packet reaches the Google server, it contains the valid source address of AAA.BBB.CCC.11 that maps to our VMware Server host. Reply packets flow back to the host, and then iptables remaps the destination address from AAA.BBB.CCC.11 to the proper host-only network originating address, according to the stored connection information.

Iptables is an incredibly flexible tool that performs many useful packet modification tasks, as well as firewall functions. If you have any favorites uses for it, feel free to post comments and feedback!

Call me - Greg Larkin: error

Powered by WordPress