Hi everyone,

A while back, I was setting up some virtual machines, each with a public and host-only network interface. I like using the host-only interface for high-performance VM-to-VM communications and then to expose the WAN IP to the Internet for public access.

I soon ran into a problem, though. What happens if a VM on one physical server needs to make an internal connection to one on another physical server? The first VM has to use the public IP address of the second VM, but I still want the connection to be essentially private and not route the packets outside of the firewall-protected LAN.

Here’s a diagram showing the VMware Server configuration with virtual machines and host-only networks (click to enlarge):

For the following examples, let’s use these IP addresses for the physical servers and virtual machines:

Hostname	WAN IP	LAN IP	Gateway IP
GW1	207.46.193.241	N/A	N/A
PH1	207.46.193.242	172.16.80.1	207.46.193.241
VM1	207.46.193.243	172.16.80.2	172.16.80.1
VM2	207.46.193.244	172.16.80.3	172.16.80.1
VM3	207.46.193.245	172.16.80.4	172.16.80.1
PH2	207.46.193.246	172.16.80.1	207.46.193.241
VM4	207.46.193.247	172.16.80.2	172.16.80.1
VM5	207.46.193.248	172.16.80.3	172.16.80.1
VM6	207.46.193.249	172.16.80.4	172.16.80.1

Notice that the LAN IPs are repeated for VMs on different physical hosts. That’s because the VMs are connecting to a host-only network on each physical machine and are independent of each other. Therefore, they can use the same IP numbers without conflicting.

Ok, so assume VM1 is a public-facing web server, and VM4 is a database server with no external services exposed except for ssh. Since there’s no way for VM1 to use VM4′s LAN IP address to connect to the database – the IPs are the same – VM1 has to use VM4′s WAN IP.

Let’s now trace where a packet as it originates from VM1 bound for VM4. This table shows the path that the packet takes, along with the source address as it passes through each hop.

Hop #	Hostname	Source Addr	Dest Addr
1	VM1	172.16.80.2	207.46.193.247
2	PH1	207.46.193.242	207.46.193.247
3	GW1	207.46.193.242	207.46.193.247

Woops – the packet has made it out to the gateway router (GW1)! That’s not what we want, because the packets exchanged between any VM should stay on the inside of the firewall. The same problem occurs if any VM on one physical machine connects to a VM on the other physical machine.

Luckily, this easily fixed by adding some static routes to each physical machine to help guide the packets where they need to go. Instead of allowing packets bound for a VM to use the default route and get sent to the gateway router, a static route redirects them to the next hop that we specify.

On RHEL4, the /etc/sysconfig/static-routes file controls any required extra routes. On PH1, the file will look like:

any host 207.46.193.247 gw 207.46.193.246
any host 207.46.193.248 gw 207.46.193.246
any host 207.46.193.249 gw 207.46.193.246

The file on PH2 looks like:

any host 207.46.193.243 gw 207.46.193.242
any host 207.46.193.244 gw 207.46.193.242
any host 207.46.193.245 gw 207.46.193.242

This simply means “when sending a packet to host X (e.g. 207.46.193.245), first send it to host Y (e.g. 207.46.193.242), because it has the information about how to get there”.

Once the static routes are installed, the packet trace now looks like:

Hop #	Hostname	Source Addr	Dest Addr
1	VM1	172.16.80.2	207.46.193.247
2	PH1	207.46.193.242	207.46.193.247
3	PH2	207.46.193.242	207.46.193.247
4	VM4	207.46.193.242	207.46.193.247

Of course, this can be extended across many virtual machines and physical machines, and it always helps to trace the packet and the network routes at each hop to determine where it’s going next. The traceroute command is your friend!

Keep in touch,
Greg
SourceHosting.net, LLC