Subversion hosting, CVS hosting, Trac hosting, Bugzilla hosting and software collaboration Providing hosted Subversion, CVS, Trac and Bugzilla repositories

July 17, 2009

Avoid Monotonous Tasks With WWW::Mechanize

Filed under: Software Development — Tags: , — Greg Larkin @ 2:59 pm
Scheutz Mechanical Calculator

The Scheutz Mechanical Calculator

Hi everyone,

A client recently asked to create a large number of users in a custom installation of phpDeadlock that he uses to manage access to his Subversion repository here. Since phpDeadlock doesn’t have a user management API, this sounded like a monotonous, error-prone task.  After entering 10-15 users, the chances of misspelling a name or email address or clicking the wrong button in the UI will likely increase dramatically.

I briefly investigated adding records directly to the backend MySQL database, but since various other actions are fired by user creation, including email notifications to the new user and the system administrator, using the application’s UI was the safest choice.

I had run across the WWW::Mechanize Perl module a while back, but hadn’t used it yet. I knew it could be used to automate interaction with a web site, and after reading through the rich list of methods, I promptly began hacking a script together to parse a list of email addresses supplied by the client and use it to drive the user creation UI.

The module implements a headless web browser, including cookie jar, history, form submission and other behaviors that you would expect, except it doesn’t parse and execute Javascript. That was a non-issue for me.

phpDeadlock has an administrator login prompt that requests a password before any privileged pages are accessed. Logging in to the application was a no-brainer in WWW::Mechanize:

use WWW::Mechanize;
use String::Random;
use Text::Capitalize;

my $mech = WWW::Mechanize->new();

That was too easy! All further page requests will be authenticated. Next, the script enters a loop to process email addresses presented on STDIN, one per line, and populate the required fields:

for my $email (<STDIN>) {
    chomp $email;
    my @fields = split /\@/, $email;
    my $finitial = substr($fields[0], 0, 1);
    my $lname = substr($fields[0], 1);
    my $pw = new String::Random->randpattern("ssssssss");

        fields => {
            firstname => capitalize($finitial),
            lastname => capitalize($lname),
            email => $email,
            username => $fields[0],
            password => $pw,
            password2 => $pw

    print "Added new user for $finitial $lname\n";

If the user’s full name had been supplied, I could have easily parsed that and set the firstname and lastname fields appropriately.

The possibilities for using this module are endless, and I encourage you to try it out, especially when you’re dreading a repetitive web application UI task.  Make sure to check the list of other Perl modules that are based on WWW::Mechanize, too!

Call me - Greg Larkin: error

July 16, 2009

Improving PHP Application Performance

Filed under: Software Development — Tags: , — Greg Larkin @ 9:47 am
Photo by: Mark McArdle

Photo by: Mark McArdle

Hi everyone,

I came across a helpful list of PHP optimization tips, and I’d like to share it with you here:

In addition, using the XDebug profiler for collecting performance data and KCacheGrind for viewing it is highly recommended to keep your PHP applications running smoothly!

Call me - Greg Larkin: error

June 19, 2009

Reducing I/O Priority on RedHat Enterprise Linux V4.0

Filed under: Operating Systems — Tags: , , — Greg Larkin @ 2:45 pm

Hi everyone,

I have some servers with RHEL4 installed on them, and I’ve noticed a problem every time I start processes that saturate the I/O channels, such as VMware’s vmware-vdiskmanager. This tool performs various operations on VMware virtual disk files, and when creating a new one, the load average on the server tends to spike into the double digits. As you can imagine, this negatively affects virtual machines running at the same time!

After some searching, I found the ionice tool that looked like a perfect solution to the problem. Unfortunately, it doesn’t run on the 2.6.9 vintage kernel supplied with RHEL4. Back to the drawing board!

After more searching, I came across a forum thread and a link to an ionice replacement for RHEL4, written in Perl. I downloaded it and tried it out, and it appears to work as advertised. My heavy I/O operations take longer now (fine), and the load average stays within acceptable limits (great!).

The script has some hard-coded values, and it can be easily tweaked as needed. Thanks to Greg Bell at ServEdge for writing it!

Call me - Greg Larkin: error

February 20, 2008

Pssst… Hey Kid, Wanna Free Hard Drive?

Filed under: Misc — Tags: , — Greg Larkin @ 8:17 pm

Hi everyone,

Here’s a little trick I’ve been using for quick deployment of additional virtual disk space to the VMware VMs that comprise the service. The VMware Server installation includes the vmware-vdiskmanager tool for creating, renaming, expanding and generally messing about with virtual hard drives.

However, I don’t like running this tool to create a new 50Gb virtual disk in the middle of the day because it just slams the disk I/O channel. To get around the problem, I’ve created several disks of different sizes during off-hours and compressed them down for easy storage. Then when I need to provision a disk, I expand it, rename it and hook it to the virtual machine in Virtual Center:

VirtualCenter Add Hardware Wizard

So here are some compressed disk images for you (SCSI format):

  • 10Gb (8391 byte download)
  • 20Gb (16415 byte download)
  • 50Gb (40373 byte download)
  • 100Gb (80373 byte download)

Once downloaded, extract them as follows:

nice -19 bzcat xxxGb.tar.bz2 | tar xvfB -

CAUTION: The resulting extracted files will be the actual size represented in the filename. They compress down so well because they are mostly empty space until they are hooked to a VM and a filesystem is created.

After I extract the files, I typically rename the virtual disk to something more meaningful, like the name of the mount point in my VM. This way, I can easily tell which virtual disk is used for what without consulting the VM config file. The disk rename command looks like this:

# vmware-vdiskmanager -n 10GbDisk.vmdk UsrSrc.vmdk
Using log file /tmp/vmware-root/vdiskmanager.log
Renaming completed successfully.
# ls *.vmdk
UsrSrc-f001.vmdk  UsrSrc-f003.vmdk  UsrSrc-f005.vmdk  UsrSrc.vmdk
UsrSrc-f002.vmdk  UsrSrc-f004.vmdk  UsrSrc-f006.vmdk


Call me - Greg Larkin: error

Powered by WordPress