Hi everyone,

I recently learned of the ElasticDrive application that implements a disk device backed by Amazon S3 and other kinds of storage services. To determine whether I could easily deploy an infinitely-scalable disk device on my FreeBSD servers, I downloaded ElasticDrive and proceeded to install it on a FreeBSD 7.0 virtual machine.

There are a few prerequisites, such as FUSE and Python setuptools, but those are easily installed with the FreeBSD ports system. If I can get ElasticDrive to work, it will be handy to package it as a port for one-command installation.
In the mean time, I installed it manually and edited the /etc/elasticdrive.cfg to point to my Amazon S3 account. I reduced the size of the S3 backing store to a 100Mb disk:

[drives]
#Path to fuse is the path to the device mount folder. NOT your fuse libraries
fuseblock|/mnt/elasticdrive="s3://mySecret:mySecretToo@aws.amazon.com/?bucket=myBucket&
stripesize=65536&blocksize=4096&blocks=25600&sizebyblocks=1"

After I fired up the tool, I saw my ElasticDrive file:

fbsd70# ls -al /mnt/elasticdrive
total 102403
drwxr-xr-x  2 root  wheel          0 Dec 31  1969 .
drwxr-xr-x  4 root  wheel        512 May  7 13:36 ..
-rw-r--r--  1 root  wheel  104857600 Dec 31  1969 ed0
-r--r--r--  1 root  wheel        391 Dec 31  1969 stats
fbsd70# du -sh /mnt/elasticdrive
100M    /mnt/elasticdrive

Cool – this looks promising!

FreeBSD uses the mdconfig command to create a file-backed loop device, similar to the Linux losetup command. This is too easy:

fbsd70# mdconfig -a -t vnode -f /mnt/elasticdrive/ed0 -u 0
fbsd70# ls -al /dev/md0*
crw-r-----  1 root  operator    0, 101 May  8 12:32 /dev/md0
crw-r-----  1 root  operator    0, 106 May  8 12:32 /dev/md0a
crw-r-----  1 root  operator    0, 107 May  8 12:32 /dev/md0c
fbsd70# fdisk /dev/md0
******* Working on device /dev/md0 *******
parameters extracted from in-core disklabel are:
cylinders=12 heads=255 sectors/track=63 (16065 blks/cyl)

parameters to be used for BIOS calculations are:
cylinders=12 heads=255 sectors/track=63 (16065 blks/cyl)

fdisk: invalid fdisk partition table found
Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
start 63, size 192717 (94 Meg), flag 80 (active)
beg: cyl 0/ head 1/ sector 1;
end: cyl 11/ head 254/ sector 63
The data for partition 2 is:
<UNUSED>
The data for partition 3 is:
<UNUSED>
The data for partition 4 is:
<UNUSED>
fbsd70#

Ok, let’s create a filesystem and test it out:

fbsd70# newfs /dev/md0a
/dev/md0a: 100.0MB (204784 sectors) block size 16384, fragment size 2048
using 4 cylinder groups of 25.00MB, 1600 blks, 3200 inodes.
super-block backups (for fsck -b #) at:
160, 51360, 102560, 153760
cg 0: bad magic number

Woops – what’s this now?! Looking at the newfs source code, this error is generated when data read back from the disk doesn’t match data written to the disk.

I’ve tried to figure out what the problem is using truss and other techniques, and I’ve contacted ElasticDrive to see if they have any ideas. So far, nothing has worked.

If anyone else has an idea how to troubleshoot this, let me know.