Subversion hosting, CVS hosting, Trac hosting, Bugzilla hosting and software collaboration Providing hosted Subversion, CVS, Trac and Bugzilla repositories

April 18, 2008

Routing Between Virtual Machines on Separate Physical Servers

Filed under: Operating Systems — Tags: , — Greg Larkin @ 5:30 pm

Hi everyone,

A while back, I was setting up some virtual machines, each with a public and host-only network interface. I like using the host-only interface for high-performance VM-to-VM communications and then to expose the WAN IP to the Internet for public access.

I soon ran into a problem, though. What happens if a VM on one physical server needs to make an internal connection to one on another physical server? The first VM has to use the public IP address of the second VM, but I still want the connection to be essentially private and not route the packets outside of the firewall-protected LAN.

Here’s a diagram showing the VMware Server configuration with virtual machines and host-only networks (click to enlarge):

Network Diagram in VMware Server Environment

For the following examples, let’s use these IP addresses for the physical servers and virtual machines:

Hostname WAN IP LAN IP Gateway IP

Notice that the LAN IPs are repeated for VMs on different physical hosts. That’s because the VMs are connecting to a host-only network on each physical machine and are independent of each other. Therefore, they can use the same IP numbers without conflicting.

Ok, so assume VM1 is a public-facing web server, and VM4 is a database server with no external services exposed except for ssh. Since there’s no way for VM1 to use VM4′s LAN IP address to connect to the database – the IPs are the same – VM1 has to use VM4′s WAN IP.

Let’s now trace where a packet as it originates from VM1 bound for VM4. This table shows the path that the packet takes, along with the source address as it passes through each hop.

Hop # Hostname Source Addr Dest Addr
1 VM1
2 PH1
3 GW1

Woops – the packet has made it out to the gateway router (GW1)! That’s not what we want, because the packets exchanged between any VM should stay on the inside of the firewall. The same problem occurs if any VM on one physical machine connects to a VM on the other physical machine.

Luckily, this easily fixed by adding some static routes to each physical machine to help guide the packets where they need to go. Instead of allowing packets bound for a VM to use the default route and get sent to the gateway router, a static route redirects them to the next hop that we specify.

On RHEL4, the /etc/sysconfig/static-routes file controls any required extra routes. On PH1, the file will look like:

any host gw
any host gw
any host gw

The file on PH2 looks like:
any host gw
any host gw
any host gw

This simply means “when sending a packet to host X (e.g., first send it to host Y (e.g., because it has the information about how to get there”.

Once the static routes are installed, the packet trace now looks like:

Hop # Hostname Source Addr Dest Addr
1 VM1
2 PH1
3 PH2
4 VM4

Of course, this can be extended across many virtual machines and physical machines, and it always helps to trace the packet and the network routes at each hop to determine where it’s going next. The traceroute command is your friend!

Bookmark and Share

Keep in touch,
Greg, LLC

No Comments

No comments yet.

RSS feed for comments on this post.

Sorry, the comment form is closed at this time.

Call me - Greg Larkin: error

Powered by WordPress