Hi everyone,

A couple of weeks ago, I made a FreeBSD 7.0 + ZFS VMware image torrent available on the SourceHosting.net BitTorrent Tracker. A reader of this blog contacted me a couple of days ago about a boot problem he’s having with the image, and I was wondering if anyone out there can reproduce it and/or provide a solution? Here are the particulars:

• Host OS: Microsoft Windows Vista
• VMware Workstation version 6.0.3 build – 80004
• FreeBSD 7.0 + ZFS VMware image (torrent)

Initially, it appears that the VMware image ZIP did not unpack correctly, because this error is displayed:

However, checking the directory where the ZIP file was unpacked shows that the file does exist:

I tried to debug the problem from here since I don’t have Vista installed anywhere, but nothing has worked yet. He sees the same problem with VMware Player, too, so I wonder if it’s something to do with Vista permissions or some other security setting.

If someone has an idea what the problem could be, let me know.

Hi everyone,

I recently upgraded the production servers to VMware Server 1.0.5 and also upgraded the VMware MUI package. The MUI (Web-based Management Interface) is useful when you need to restart a VM, reallocate VM memory and perform other maintenance tasks, but you don’t have access to the VMware Server Console or VirtualCenter.

The MUI is driven by Apache 1.3.31, and as such, it generates the familiar log files:

/var/log/vmware-mui/access_log
/var/log/vmware-mui/error_log
/var/log/vmware-mui/ssl_engine_log
/var/log/vmware-mui/ssl_request_log

However, after a while, the log directory tends to fill up:

# ls -larS
total 78988
drwxr-xr-x  14 root root     4096 Apr  7 11:58 ..
drwxr-xr-x   2 root root     4096 May  9  2007 .
-rw-r--r--   1 root root    53985 Mar 24 09:20 error_log
-rw-r--r--   1 root root  8280230 Apr  7 12:59 access_log
-rw-r--r--   1 root root  9955524 Apr  7 12:59 ssl_request_log
-rw-r--r--   1 root root 62473978 Apr  7 12:59 ssl_engine_log

Ok, it’s only 78Mb so far, but why wait until the logs fill up the disk? Since the VMware Server host is running RHEL4, it came pre-installed with logrotate, and an existing configuration for the standard Apache log rotation can be easily adapted for the VMware Server MUI. Just place the following in /etc/logrotate.d/httpd.vmware:

/var/log/vmware-mui/*log {
    missingok
    notifempty
    sharedscripts
    postrotate
        /bin/kill -HUP `cat /var/run/httpd.vmware.pid 2>/dev/null` 2> /dev/null || true
    endscript
}

The default settings in /etc/logrotate.conf also take effect during rotation and you can enable log file compression and length of retention in there.

Hi everyone,

Since FreeBSD 7.0 was recently released, I figured it’s a good time to put a VMware image together and have a look. Probably one of the most interesting new enhancements is the addition of the ZFS filesystem. I can’t wait to try that out, and it should make managing disk space a lot easier in the context of the SourceHosting.net service.

You can find the zipped image on the SourceHosting.net BitTorrent tracker. Some notes about the image:

• The VM has been configured with 768Mb of memory. You can reduce it to 512Mb, but the ZFS documentation says “Me want more memory!“
• The root password is “password”
• ZFS is enabled by default
• The /usr/ports filesystem is located in a ZFS pool
• The Ethernet interface is bridged to the host and uses DHCP

I tested this VM in VMware Server and VMware Player, and it seems to work fine. If you have any questions or problems, feel free to post comments here.

Hi everyone,

I was recently reminded of a problem I ran into when I first set up the SourceHosting.net service on VMware Server. The VMware technology has the concept of virtual networks, including a host-only network. The host-only network enables several VMs on the same host to communicate on their own private Ethernet switch. This is a great way to simulate a real-world, production environment.

However, what if you want resources on the host-only network, such as private servers without routable public IP addresses, to be able to make outbound connections to the outside world? That’s where it gets a bit tricky! The SourceHosting.net service assigns a FreeBSD jail to each client, and these servers each have an IP address on the host-only network. They need to make connections to the public Internet, so after some digging around, I found a solution.

The first thing to do is assign a host-only network gateway address to your VMware Server host. In my case, the host-only network is addressed as 172.16.80.0/255.255.240.0. The physical server’s gateway address is therefore 172.16.80.1.

Each VM has 2 NICs defined, one with a routable Internet address and one with a host-only address, perhaps 172.16.80.2. A FreeBSD jail running inside a VM will have a host-only IP address aliased to the 2nd NIC, such as 172.16.80.55. Somehow, a packet originating in the jail must pass out through the VM, then the physical host to the destination and back again.

The FreeBSD VM can easily send its packets out to the host-only address on the physical host by using this directive in its /etc/rc.conf file:

Since the jail IP addresses are aliased to the host-only NIC in the VM, packets originating from a jail will also use 172.16.80.1 as their default router.

At the physical host level, in order for packets to pass from its host-only interface to its external interface, it must be configured as a router. That’s done by adding the following directive to the /etc/sysctl.conf file (on RHEL 4 and other flavors of Linux):

So far, so good. Now here’s where the craziness, errr magic, happens! The following firewall script is added to /etc/rc.local:

The most important bit of this script is the last line. The rest of it defines some variables and cleans up the firewall rules to a known state. Since a hardware firewall is doing all of the heavy lifting in front of this server, the iptables software firewall is going to serve simply as a source address packet mangler. Hmm, “packet mangling” – that sounds nasty! But it’s actually a good thing here, because it ensures that packets get from point A to B and back again.

We also need to look at the routing table to figure out how source address mangling will change how the packet behaves:

Normally, if a packet originates from the vmnet1 interface (VMware host-only network) with a source address of 172.16.80.55 and bound for www.google.com (74.125.47.103), it will be handled by the default route and sent out the eth0 interface. That’s all well and good, but when Google tries to reply, a router somewhere along the way sees a source address of 172.16.80.55 and drops the packet.

Enter POSTROUTING and SNAT! Here is the expanded iptables command from our script above:

Ok, let’s break it down:

1. Add a rule to the nat table (“-t nat”)
2. Append the rule to the POSTROUTING chain (“-A POSTROUTING”) – i.e. apply rule after deciding which route will handle the packet
3. Process the packet when its source address originates on the host-only network (“-s 172.16.80.0/20”) and it’s bound for the eth0 interface (“-o eth0”) – normally a bad thing!
4. Jump to the SNAT target for source address modification (“-j SNAT”)
5. Change the source address to AAA.BBB.CCC.11 and send the packet on its merry way (“--to AAA.BBB.CCC.11”)

This means that when the packet reaches the Google server, it contains the valid source address of AAA.BBB.CCC.11 that maps to our VMware Server host. Reply packets flow back to the host, and then iptables remaps the destination address from AAA.BBB.CCC.11 to the proper host-only network originating address, according to the stored connection information.

Iptables is an incredibly flexible tool that performs many useful packet modification tasks, as well as firewall functions. If you have any favorites uses for it, feel free to post comments and feedback!

Hi everyone,

I’ve got FreeBSD 7.0 downloaded and running as a VMware Server virtual machine so I can start checking it out and get a feel for how it works. Once ZFS is deemed stable for production, I’ll plan to migrate to the 7.x series.

The first thing I generally do after setting up a new VM is install VMware Tools. Wait – scratch that. The first thing I have to do before installing VMware Tools is install Perl, since it’s not part of the FreeBSD base system. That’s easy enough:

Ok, on to the VMware Tools installer! After starting the vmware-install.pl script, I answered a bunch of questions (used all default responses), and finally was met with this somewhat odd message:

Wait a minute – I’m pretty sure I’m installing in a virtual machine here! Initially, I pored through the installation Perl script and discovered that it executes vmware-checkvm to determine if it’s running inside a VM or not. Ok, let’s try that by hand:

That’s a problem! vmware-checkvm is a statically-linked binary, and to get it working, the FreeBSD compat6x port needs to be installed. The compat6x port installs a variety of libraries that were found in FreeBSD 6.x but have had their versions bumped for FreeBSD 7.0.

Hmm, still no joy after doing that – vmware-checkvm is still core dumping. After more Googling and nosing about in the VMware Tools installation script, I learned that VMware expects the libc.so.6 library installed by compat6x to be in /lib. However, all compat{3,4,5,6}x ports install their libraries in /usr/local/lib/compat to avoid messing with the base system.

A simple symbolic link gets libc.so.6 in place for VMware:

ln -s /usr/local/lib/compat/libc.so.6 /lib

And now success, right? Wrong!

I’m not logged in over the network, so why am I getting this message? It turns out that tcsh (root shell in FreeBSD) is setting the REMOTEHOST environment variable, even though I’m logged in at the VM’s console.

The solution is a simple command:

Start up the VMware Tool installer one more time, and now it works!

Maybe the newest version of VMware Server supports FreeBSD 7.0 natively, so I’ll have to test that out soon. Details will be posted here when I do!

Hi everyone,

Here’s a little trick I’ve been using for quick deployment of additional virtual disk space to the VMware VMs that comprise the SourceHosting.net service. The VMware Server installation includes the vmware-vdiskmanager tool for creating, renaming, expanding and generally messing about with virtual hard drives.

However, I don’t like running this tool to create a new 50Gb virtual disk in the middle of the day because it just slams the disk I/O channel. To get around the problem, I’ve created several disks of different sizes during off-hours and compressed them down for easy storage. Then when I need to provision a disk, I expand it, rename it and hook it to the virtual machine in Virtual Center:

So here are some compressed disk images for you (SCSI format):

• 10Gb (8391 byte download)
• 20Gb (16415 byte download)
• 50Gb (40373 byte download)
• 100Gb (80373 byte download)

Once downloaded, extract them as follows:

CAUTION: The resulting extracted files will be the actual size represented in the filename. They compress down so well because they are mostly empty space until they are hooked to a VM and a filesystem is created.

After I extract the files, I typically rename the virtual disk to something more meaningful, like the name of the mount point in my VM. This way, I can easily tell which virtual disk is used for what without consulting the VM config file. The disk rename command looks like this:

Simple!

Hi everyone,

I installed a bunch of security updates, new kernels, etc. to the VMware Server hosts and virtual machines tonight. One problem that has been dogging me for a while now is that one (and only one) of the VMs boots in a strange state and reports Ethernet watchdog timeouts on its host-only interface. The issue here is that it’s then unable to communicate with its name server, NFS server and other LAN resources.

After some different Google searches, I found a solution provided by Antonio Lorusso. I’ve made his changes to the VMware Server file in question, and I’ll keep an eye on the VMs as they auto-start after the next host server reboot. I expect that the timeouts will be a thing of the past. Thank you, Antonio!

Hi everyone,

As I was setting up the new FreeBSD 6.2 VM with Apache 2.2 preinstalled in it, I ran into a problem. The VM’s virtual Ethernet adapter is configured for DHCP, and the VMware software acts as a DHCP server to hand out IP addresses as VMs are started.

The problem is that Apache likes to know the IP address of the machine it’s running on, and in some cases, it won’t start unless it does. Luckily, the DHCP client in FreeBSD, and likely other operating systems, calls separate hook scripts (if they exist) before and after configuring the adapter with a dynamic address.

In my case, I wanted to automatically create a ServerName directive for Apache with the dynamic address. The default Apache 2.2 installation on FreeBSD provides a user configuration file include directory at /usr/local/etc/apache22/Includes. As it starts up, Apache loads any file placed in that directory and named with the “.conf” suffix. I think we’ve got a solution!

It’s also necessary to tweak the contents of the /etc/hosts file so that the VM can resolve its own hostname. I wrote a DHCP client exit hook script that configures both Apache and the /etc/hosts file. This script should be easily adaptable to other operating systems, as it uses the standard Bourne shell syntax.

Simply download the script, unzip the contents into /etc, and let me know what you think!

Hi everyone,

As I got further into configuring my sample VM, I decided it would make sense to provide a few different VM flavors for folks who don’t want to configure Apache, PHP, and other ports to get the PHP frameworks installed and working.

I’ve set up a BitTorrent tracker at http://torrents.sourcehosting.net:10692/, and I’ll place the various VMs there from now on. The installation process is as follows:

• Download and install a BitTorrent client, if you don’t already have one. I recommend Azureus.
• Grab the UsrPorts torrent.
• Grab a FreeBSD 6.2 Basic Installation torrent, such as the Bare Bones VM, or the VM pre-configured with Apache. Check the main tracker site for other options, and view the torrent info page to find out what ports have been preinstalled.
• Once the UsrPorts.zip file has been fetched by your BitTorrent client, extract the contents to a new directory.
• Extract your selected FreeBSD 6.2 VM .zip file to the same directory and start up the VM with VMware Player or VMware Server.

The UsrPorts.zip file contains the /usr/ports filesystem that’s mounted in the VM. Since this filesystem is typically kept up to date by running portsnap while the VM is running, I’ve provided an initial snapshot for you, but it’s not necessary to keep re-downloading when I provide a new FreeBSD VM that you’d like to try out.

I hope this system works better for folks, and please consider seeding the torrents as you download the .zip files.

Hello everyone,

Now that we’ve got some example virtual machines to play around with, let’s move on to the next step and get one running.

I’m using VMware Player on Windows XP, and the VM files have been extracted to My Documents\My Virtual Machines. The folder contents look like this:

Digging into the VM installation directory proper, you see all of the files that make up your virtual machine:

The .vmx file is a text file that contains all of the virtual machine configuration options. You can change the VM memory allocation, add/remove devices, etc. just by editing that file. The .vmdk files are the virtual machine’s disks and store its state across reboots.

If your VM ever runs out of disk space and you still have room on your host operating system, you can create additional vmdk files and attach them to the VM. That sure is easier than cracking open a server case and physically installing new drives!

Next, double-click on the FreeBSD 6.2 Basic Installation.vmx file, and VMware Player starts up and boots the VM:

After the usual messages, the virtual machine has booted, and you see the familiar UNIX login prompt:

Next time, we’ll dive into installing the various PHP frameworks under FreeBSD. The FreeBSD ports system is designed for ease of use, and you’ll see how simple it is to get a software package and all of its dependencies installed with one command.