Subversion hosting, CVS hosting, Trac hosting, Bugzilla hosting and software collaboration Providing hosted Subversion, CVS, Trac and Bugzilla repositories

July 17, 2009

Avoid Monotonous Tasks With WWW::Mechanize

Filed under: Software Development — Tags: , — Greg Larkin @ 2:59 pm
Scheutz Mechanical Calculator

The Scheutz Mechanical Calculator

Hi everyone,

A client recently asked to create a large number of users in a custom installation of phpDeadlock that he uses to manage access to his Subversion repository here. Since phpDeadlock doesn’t have a user management API, this sounded like a monotonous, error-prone task.  After entering 10-15 users, the chances of misspelling a name or email address or clicking the wrong button in the UI will likely increase dramatically.

I briefly investigated adding records directly to the backend MySQL database, but since various other actions are fired by user creation, including email notifications to the new user and the system administrator, using the application’s UI was the safest choice.

I had run across the WWW::Mechanize Perl module a while back, but hadn’t used it yet. I knew it could be used to automate interaction with a web site, and after reading through the rich list of methods, I promptly began hacking a script together to parse a list of email addresses supplied by the client and use it to drive the user creation UI.

The module implements a headless web browser, including cookie jar, history, form submission and other behaviors that you would expect, except it doesn’t parse and execute Javascript. That was a non-issue for me.

phpDeadlock has an administrator login prompt that requests a password before any privileged pages are accessed. Logging in to the application was a no-brainer in WWW::Mechanize:

use WWW::Mechanize;
use String::Random;
use Text::Capitalize;

my $mech = WWW::Mechanize->new();

That was too easy! All further page requests will be authenticated. Next, the script enters a loop to process email addresses presented on STDIN, one per line, and populate the required fields:

for my $email (<STDIN>) {
    chomp $email;
    my @fields = split /\@/, $email;
    my $finitial = substr($fields[0], 0, 1);
    my $lname = substr($fields[0], 1);
    my $pw = new String::Random->randpattern("ssssssss");

        fields => {
            firstname => capitalize($finitial),
            lastname => capitalize($lname),
            email => $email,
            username => $fields[0],
            password => $pw,
            password2 => $pw

    print "Added new user for $finitial $lname\n";

If the user’s full name had been supplied, I could have easily parsed that and set the firstname and lastname fields appropriately.

The possibilities for using this module are endless, and I encourage you to try it out, especially when you’re dreading a repetitive web application UI task.  Make sure to check the list of other Perl modules that are based on WWW::Mechanize, too!

Call me - Greg Larkin: error

July 1, 2009

Subversion Dump File Validation

Filed under: Source Code Control — Tags: , — Greg Larkin @ 11:42 am
Common result from converting a non-Subversion source code repository to a Subversion dump file

Common result from converting a complex non-Subversion source code repository to a Subversion dump file

Hi everyone,

Many new clients have existing repositories that they want to import into a Subversion repository here. If we’re lucky, the client already uses Subversion, and it’s no more difficult to import the repository than dumping and loading it.

Converting from CVS to SVN is quite easy using the cvs2svn script. We rarely run into any problems, and it has a flexible set of command line options. If, however, the client uses a different source code control system, the process can get trickier.

There are several repository converters available for download, including Polarion Importer for SVN and VSS2SVN. These tools are welcome additions to a release engineer’s bag of tricks, especially if different groups in an organization have not standardized on a single SCM system and you’re trying to convert everyone to Subversion.

As we’ve used the tools mentioned above more and more frequently, we’ve found that they work well for new clients with reasonably small and uncomplicated repositories. However, as the complexity grows with more tags, branches and merge points, the likelihood of producing a corrupted or logically-incorrect Subversion dump file increases.

This problem is illustrated by a recent repository conversion in which the new client used a tool to convert from their existing VSS repository to a Subversion dump file, preparing for their migration to Upon cursory inspection, the dump file contents looked reasonable, but some number of revisions into the loading process, svnadmin reported the following error:

<<< Started new transaction, based on original revision 33
svnadmin: File not found: transaction '30-1', path '/src/docs/ChangeLog'
    * editing path : src/docs/ChangeLog ...

After reading through the dump file, we discovered that it contained a sequence of operations on the “/src/docs/ChangeLog” file that occurred before the file had been added to the repository. We’ve also run across negative-numbered revisions and attempts to delete files from locations that don’t exist. All of these situations will abort repository loading.

These errors aren’t difficult to detect, and even fix, by parsing through the dump file and rewriting it slightly with the Perl CPAN module SVN::Dumpfile. This module extracts data from a dump file and also allows new data to be inserted into it.

In order to make it easier to test software that generates Subversion dump files, we are building on the work of SVN::Dumpfile and creating a sequence of dump file validation tests. Not even the svnadmin tests included in the Subversion distribution parse dump files, and perhaps this work will migrate into their suite as well.

If you have any suggestions or ideas for specific tests that you would like included, please let us know.

Call me - Greg Larkin: error

February 22, 2008

Super-Configurable Subversion Notifications

Filed under: Source Code Control — Tags: , , , — Greg Larkin @ 8:13 pm

Hi everyone,

Are you using the Perl SVN::Notify module to generate your Subversion commit notification emails yet? If not, go install it, and try out svnnotify. It generates very nice-looking text or HTML-formatted emails, and it has loads of options to make it do just what you want. Place as many svnnotify commands as you like in your Subversion post-commit hook script, and off you go.

Unfortunately, with a large number of developers and many different projects in your repository, your post-commit script may start to look like this:

svnnotify -r $2 -d -P "SVN Commit " -O -H HTML::ColorDiff -p $1 -t
svnnotify -r $2 -H  Mirror::Rsync -p $1 --rsync-host \
    --to /usr/local/www/htdocs --rsync-delete=yes
svnnotify -r $2 -d -P "SVN Tag Creation " -O -H HTML::ColorDiff -p $1 \
svnnotify -r $2 -d -P "SVN Branch Creation " -O -H HTML::ColorDiff -p $1 \
svnnotify -r $2 -d -P "SVN Vendor Drop " -O -H HTML::ColorDiff -p $1 \
svnnotify -r $2 -d -P "ClientA Special Version Commit " -O -H HTML::ColorDiff \
    -p $1 -x

Yikes -that’s bound to keep any Subversion administrator up nights! However, help is on the way from Perl developer John Peacock.

John has created a nice wrapper around SVN::Notify named SVN::Notify::Config. This Perl module transforms the mess of command lines above to a cleaner, easily-editable form using YAML to express the desired svnnotify options.

The equivalent post-commit hook using SVN::Notify::Config is now:

#!/usr/bin/perl -MSVN::Notify::Config=$0
 --- #YAML:1.0
   PATH: "/usr/bin:/usr/local/bin"
   handler: HTML::ColorDiff
   subject-prefix: "SVN Commit "
   handler: Mirror::Rsync
   to: /usr/local/www/htdocs
   rsync-delete: yes
   handler: HTML::ColorDiff
   subject-prefix: "SVN Tag Creation "
   handler: HTML::ColorDiff
   subject-prefix: "SVN Branch Creation "
   handler: HTML::ColorDiff
   subject-prefix: "SVN Vendor Drop "
   handler: HTML::ColorDiff
   subject-prefix: "ClientA Special Version Commit "

That’s a lot more readable, and I expect it will be easier to maintain, even though there are more lines in the file. I’m still boning up on the YAML syntax and how this module uses it, so there may be some optimizations to be made. Feel free to send corrections!

The other thing I like about using YAML this way is it lends itself well to hooking into a notification management web interface, and that makes it even easier to keep those emails flowing where they should.

Call me - Greg Larkin: error

Powered by WordPress