Hi everyone,

I updated the ports tree and repackaged the FreeBSD 7.0 VMware virtual machine that turned out to be corrupted in my earlier release. I apologize for anyone who was inconvenienced, and I still don’t know how part of one of the disks was corrupted, since the source image was fine.

Anyway, you can find the new download here: VMware 7.0 virtual machine with ZFS enabled

If you run into any problems, please contact me.

Keep in touch,
Greg

Hi everyone,

In a previous post, I asked for assistance about getting a FreeBSD 7.0 virtual machine running on VMware Workstation on MS Windows Vista. Well, it turns out I didn’t need any help after all - the torrent file was corrupted!

Somehow, one of the 2Gb slices of the VM’s boot disk was truncated to ~1.4Gb. I don’t know how it happened - perhaps during the ZIP compression. It’s very odd, because the rest of the slices were fine.

Anyway, I’m preparing a new torrent, and I’ll post the link when it’s ready.

Keep in touch,
Greg

Hi everyone,

A while back, I was setting up some virtual machines, each with a public and host-only network interface. I like using the host-only interface for high-performance VM-to-VM communications and then to expose the WAN IP to the Internet for public access.

I soon ran into a problem, though. What happens if a VM on one physical server needs to make an internal connection to one on another physical server? The first VM has to use the public IP address of the second VM, but I still want the connection to be essentially private and not route the packets outside of the firewall-protected LAN.

Here’s a diagram showing the VMware Server configuration with virtual machines and host-only networks (click to enlarge):

For the following examples, let’s use these IP addresses for the physical servers and virtual machines:

Notice that the LAN IPs are repeated for VMs on different physical hosts. That’s because the VMs are connecting to a host-only network on each physical machine and are independent of each other. Therefore, they can use the same IP numbers without conflicting.

Ok, so assume VM1 is a public-facing web server, and VM4 is a database server with no external services exposed except for ssh. Since there’s no way for VM1 to use VM4’s LAN IP address to connect to the database - the IPs are the same - VM1 has to use VM4’s WAN IP.

Let’s now trace where a packet as it originates from VM1 bound for VM4. This table shows the path that the packet takes, along with the source address as it passes through each hop.

Woops - the packet has made it out to the gateway router (GW1)! That’s not what we want, because the packets exchanged between any VM should stay on the inside of the firewall. The same problem occurs if any VM on one physical machine connects to a VM on the other physical machine.

Luckily, this easily fixed by adding some static routes to each physical machine to help guide the packets where they need to go. Instead of allowing packets bound for a VM to use the default route and get sent to the gateway router, a static route redirects them to the next hop that we specify.

On RHEL4, the /etc/sysconfig/static-routes file controls any required extra routes. On PH1, the file will look like:

any host 207.46.193.247 gw 207.46.193.246
any host 207.46.193.248 gw 207.46.193.246
any host 207.46.193.249 gw 207.46.193.246

The file on PH2 looks like:

any host 207.46.193.243 gw 207.46.193.242
any host 207.46.193.244 gw 207.46.193.242
any host 207.46.193.245 gw 207.46.193.242

This simply means “when sending a packet to host X (e.g. 207.46.193.245), first send it to host Y (e.g. 207.46.193.242), because it has the information about how to get there”.

Once the static routes are installed, the packet trace now looks like:

Of course, this can be extended across many virtual machines and physical machines, and it always helps to trace the packet and the network routes at each hop to determine where it’s going next. The traceroute command is your friend!

Keep in touch,
Greg

Hi everyone,

In a previous post, I postulated a solution for rotating the Apache log files produced by the VMware MUI application. Unfortunately, the MUI version of Apache doesn’t work well with logrotate, as I discovered over the weekend.

After receiving an alert on the cell phone in the middle of the night that the MUI was down, I discovered this in the /var/log/vmware-mui/error_log file:

[Sun Apr 13 04:02:03 2008] [error] Could not destroy the stale named pipe directory: /var/run/vmware//httpd/21157\n
[Sun Apr 13 04:02:03 2008] [error] VMWARE PANIC: \nNOT_IMPLEMENTED F(4023):707\n
[Sun Apr 13 04:02:03 2008] [error] Panic: Could not allocate temporary context.\n

Ok, perhaps there’s a better solution than logrotate and one that I’m already using with Apache in all of the SourceHosting.net virtual machines. A while back, I discovered the httplog tool, which builds on the premise of the standard rotatelogs tool that’s part of the Apache distribution. Since it doesn’t restart Apache when it rotates its log files, I suspected it might work well with the MUI.

Since the MUI is running on a RHEL4 system, I created an RPM of httplog instead of installing directly from the source distro. After many years working with Linux and FreeBSD, I much prefer everything to be installed from an RPM or a port, just for sanity’s sake and managing future upgrades.

The next step is to integrate the httplog command into the MUI’s httpd.conf file, found in /usr/lib/vmware-mui/apache/conf. If you’d like to update your file, download this patch file, and run the following commands (after installing httplog):

rm -f /etc/logrotate.d/httpd.vmware
/etc/rc.d/init.d/httpd.vmware stop
patch -b --verbose  /usr/lib/vmware-mui/apache/conf/httpd.conf < httpd.conf.httplog-patch
/etc/rc.d/init.d/httpd.vmware start

After starting up httpd.vmware, your /var/log/vmware-mui directory should look something like this:

total 16
lrwxrwxrwx  1 root root   36 Apr 14 17:12 error_log -> /var/log/vmware-mui/200815-error_log
-rw-r--r--  1 root root  186 Apr 14 17:12 200815-error_log
lrwxrwxrwx  1 root root   41 Apr 14 17:12 ssl_engine_log -> /var/log/vmware-mui/200815-ssl_engine_log
lrwxrwxrwx  1 root root   42 Apr 14 17:16 ssl_request_log -> /var/log/vmware-mui/200815-ssl_request_log
lrwxrwxrwx  1 root root   37 Apr 14 17:16 access_log -> /var/log/vmware-mui/200815-access_log
-rw-r--r--  1 root root  396 Apr 14 17:31 200815-ssl_request_log
-rw-r--r--  1 root root 3820 Apr 14 17:31 200815-ssl_engine_log
-rw-r--r--  1 root root  328 Apr 14 17:31 200815-access_log

I’ve got my httplog process configured to encode the week number into the Apache log filenames, rotate them each week and compress the rotated logs. If you want to automatically remove compressed logs after a certain number of weeks, a simple cron job accomplishes that.

httplog accepts many different filename configurations, so man httplog for more information or post questions here.

Keep in touch,
Greg

Hi everyone,

In a future post to follow very shortly, I’ll discuss a change to the VMware MUI application that relies on the httplog tool to perform Apache log rotation.

The first thing I had to do was get httplog installed on the RHEL4 system here. I quickly discovered that the source distribution wasn’t too good at detecting the proper version of zlib and configuring the Makefile accordingly. If you don’t have exactly zlib v1.1.3, the configure script assumes that you don’t have it at all. If you have a later version of zlib (like RHEL4 does), you have to do some hand-editing of the source code to build the tool.

I found an httplog SRPM online, but the .spec file wasn’t quite up to snuff. I added some dependencies and tweaked the configure script a bit. If you just want to install the tool, download the httplog RPM, or you can make additional tweaks with the httplog SRPM.

Keep in touch,
Greg

Hi everyone,

A couple of weeks ago, I made a FreeBSD 7.0 + ZFS VMware image torrent available on the SourceHosting.net BitTorrent Tracker. A reader of this blog contacted me a couple of days ago about a boot problem he’s having with the image, and I was wondering if anyone out there can reproduce it and/or provide a solution? Here are the particulars:

• Host OS: Microsoft Windows Vista
• VMware Workstation version 6.0.3 build – 80004
• FreeBSD 7.0 + ZFS VMware image (torrent)

Initially, it appears that the VMware image ZIP did not unpack correctly, because this error is displayed:

However, checking the directory where the ZIP file was unpacked shows that the file does exist:

I tried to debug the problem from here since I don’t have Vista installed anywhere, but nothing has worked yet. He sees the same problem with VMware Player, too, so I wonder if it’s something to do with Vista permissions or some other security setting.

If someone has an idea what the problem could be, let me know.

Keep in touch,
Greg

Hi everyone,

I recently upgraded the production servers to VMware Server 1.0.5 and also upgraded the VMware MUI package. The MUI (Web-based Management Interface) is useful when you need to restart a VM, reallocate VM memory and perform other maintenance tasks, but you don’t have access to the VMware Server Console or VirtualCenter.

The MUI is driven by Apache 1.3.31, and as such, it generates the familiar log files:

/var/log/vmware-mui/access_log
/var/log/vmware-mui/error_log
/var/log/vmware-mui/ssl_engine_log
/var/log/vmware-mui/ssl_request_log

However, after a while, the log directory tends to fill up:

# ls -larS
total 78988
drwxr-xr-x  14 root root     4096 Apr  7 11:58 ..
drwxr-xr-x   2 root root     4096 May  9  2007 .
-rw-r--r--   1 root root    53985 Mar 24 09:20 error_log
-rw-r--r--   1 root root  8280230 Apr  7 12:59 access_log
-rw-r--r--   1 root root  9955524 Apr  7 12:59 ssl_request_log
-rw-r--r--   1 root root 62473978 Apr  7 12:59 ssl_engine_log

Ok, it’s only 78Mb so far, but why wait until the logs fill up the disk? Since the VMware Server host is running RHEL4, it came pre-installed with logrotate, and an existing configuration for the standard Apache log rotation can be easily adapted for the VMware Server MUI. Just place the following in /etc/logrotate.d/httpd.vmware:

/var/log/vmware-mui/*log {
    missingok
    notifempty
    sharedscripts
    postrotate
        /bin/kill -HUP `cat /var/run/httpd.vmware.pid 2>/dev/null` 2> /dev/null || true
    endscript
}

The default settings in /etc/logrotate.conf also take effect during rotation and you can enable log file compression and length of retention in there.

Keep in touch,
Greg

Hi everyone,

Since FreeBSD 7.0 was recently released, I figured it’s a good time to put a VMware image together and have a look. Probably one of the most interesting new enhancements is the addition of the ZFS filesystem. I can’t wait to try that out, and it should make managing disk space a lot easier in the context of the SourceHosting.net service.

You can find the zipped image on the SourceHosting.net BitTorrent tracker. Some notes about the image:

• The VM has been configured with 768Mb of memory. You can reduce it to 512Mb, but the ZFS documentation says “Me want more memory!“
• The root password is “password”
• ZFS is enabled by default
• The /usr/ports filesystem is located in a ZFS pool
• The Ethernet interface is bridged to the host and uses DHCP

I tested this VM in VMware Server and VMware Player, and it seems to work fine. If you have any questions or problems, feel free to post comments here.

Keep in touch,
Greg

Hi everyone,

I was recently reminded of a problem I ran into when I first set up the SourceHosting.net service on VMware Server. The VMware technology has the concept of virtual networks, including a host-only network. The host-only network enables several VMs on the same host to communicate on their own private Ethernet switch. This is a great way to simulate a real-world, production environment.

However, what if you want resources on the host-only network, such as private servers without routable public IP addresses, to be able to make outbound connections to the outside world? That’s where it gets a bit tricky! The SourceHosting.net service assigns a FreeBSD jail to each client, and these servers each have an IP address on the host-only network. They need to make connections to the public Internet, so after some digging around, I found a solution.

The first thing to do is assign a host-only network gateway address to your VMware Server host. In my case, the host-only network is addressed as 172.16.80.0/255.255.240.0. The physical server’s gateway address is therefore 172.16.80.1.

Each VM has 2 NICs defined, one with a routable Internet address and one with a host-only address, perhaps 172.16.80.2. A FreeBSD jail running inside a VM will have a host-only IP address aliased to the 2nd NIC, such as 172.16.80.55. Somehow, a packet originating in the jail must pass out through the VM, then the physical host to the destination and back again.

The FreeBSD VM can easily send its packets out to the host-only address on the physical host by using this directive in its /etc/rc.conf file:

Since the jail IP addresses are aliased to the host-only NIC in the VM, packets originating from a jail will also use 172.16.80.1 as their default router.

At the physical host level, in order for packets to pass from its host-only interface to its external interface, it must be configured as a router. That’s done by adding the following directive to the /etc/sysctl.conf file (on RHEL 4 and other flavors of Linux):

So far, so good. Now here’s where the craziness, errr magic, happens! The following firewall script is added to /etc/rc.local:

The most important bit of this script is the last line. The rest of it defines some variables and cleans up the firewall rules to a known state. Since a hardware firewall is doing all of the heavy lifting in front of this server, the iptables software firewall is going to serve simply as a source address packet mangler. Hmm, “packet mangling” - that sounds nasty! But it’s actually a good thing here, because it ensures that packets get from point A to B and back again.

We also need to look at the routing table to figure out how source address mangling will change how the packet behaves:

Normally, if a packet originates from the vmnet1 interface (VMware host-only network) with a source address of 172.16.80.55 and bound for www.google.com (74.125.47.103), it will be handled by the default route and sent out the eth0 interface. That’s all well and good, but when Google tries to reply, a router somewhere along the way sees a source address of 172.16.80.55 and drops the packet.

Enter POSTROUTING and SNAT! Here is the expanded iptables command from our script above:

Ok, let’s break it down:

1. Add a rule to the nat table (”-t nat“)
2. Append the rule to the POSTROUTING chain (”-A POSTROUTING“) - i.e. apply rule after deciding which route will handle the packet
3. Process the packet when its source address originates on the host-only network (”-s 172.16.80.0/20“) and it’s bound for the eth0 interface (”-o eth0“) - normally a bad thing!
4. Jump to the SNAT target for source address modification (”-j SNAT“)
5. Change the source address to AAA.BBB.CCC.11 and send the packet on its merry way (”--to AAA.BBB.CCC.11“)

This means that when the packet reaches the Google server, it contains the valid source address of AAA.BBB.CCC.11 that maps to our VMware Server host. Reply packets flow back to the host, and then iptables remaps the destination address from AAA.BBB.CCC.11 to the proper host-only network originating address, according to the stored connection information.

Iptables is an incredibly flexible tool that performs many useful packet modification tasks, as well as firewall functions. If you have any favorites uses for it, feel free to post comments and feedback!

Keep in touch,
Greg

Hi everyone,

I’ve got FreeBSD 7.0 downloaded and running as a VMware Server virtual machine so I can start checking it out and get a feel for how it works. Once ZFS is deemed stable for production, I’ll plan to migrate to the 7.x series.

The first thing I generally do after setting up a new VM is install VMware Tools. Wait - scratch that. The first thing I have to do before installing VMware Tools is install Perl, since it’s not part of the FreeBSD base system. That’s easy enough:

Ok, on to the VMware Tools installer! After starting the vmware-install.pl script, I answered a bunch of questions (used all default responses), and finally was met with this somewhat odd message:

Wait a minute - I’m pretty sure I’m installing in a virtual machine here! Initially, I pored through the installation Perl script and discovered that it executes vmware-checkvm to determine if it’s running inside a VM or not. Ok, let’s try that by hand:

That’s a problem! vmware-checkvm is a statically-linked binary, and to get it working, the FreeBSD compat6x port needs to be installed. The compat6x port installs a variety of libraries that were found in FreeBSD 6.x but have had their versions bumped for FreeBSD 7.0.

Hmm, still no joy after doing that - vmware-checkvm is still core dumping. After more Googling and nosing about in the VMware Tools installation script, I learned that VMware expects the libc.so.6 library installed by compat6x to be in /lib. However, all compat{3,4,5,6}x ports install their libraries in /usr/local/lib/compat to avoid messing with the base system.

A simple symbolic link gets libc.so.6 in place for VMware:

ln -s /usr/local/lib/compat/libc.so.6 /lib

And now success, right? Wrong!

I’m not logged in over the network, so why am I getting this message? It turns out that tcsh (root shell in FreeBSD) is setting the REMOTEHOST environment variable, even though I’m logged in at the VM’s console.

The solution is a simple command:

Start up the VMware Tool installer one more time, and now it works!

Maybe the newest version of VMware Server supports FreeBSD 7.0 natively, so I’ll have to test that out soon. Details will be posted here when I do!

Keep in touch,
Greg