Subversion hosting, CVS hosting, Trac hosting, Bugzilla hosting and software collaboration Providing hosted Subversion, CVS, Trac and Bugzilla repositories

July 1, 2009

Subversion Dump File Validation

Filed under: Source Code Control — Tags: , — Greg Larkin @ 11:42 am
Common result from converting a non-Subversion source code repository to a Subversion dump file

Common result from converting a complex non-Subversion source code repository to a Subversion dump file

Hi everyone,

Many new clients have existing repositories that they want to import into a Subversion repository here. If we’re lucky, the client already uses Subversion, and it’s no more difficult to import the repository than dumping and loading it.

Converting from CVS to SVN is quite easy using the cvs2svn script. We rarely run into any problems, and it has a flexible set of command line options. If, however, the client uses a different source code control system, the process can get trickier.

There are several repository converters available for download, including Polarion Importer for SVN and VSS2SVN. These tools are welcome additions to a release engineer’s bag of tricks, especially if different groups in an organization have not standardized on a single SCM system and you’re trying to convert everyone to Subversion.

As we’ve used the tools mentioned above more and more frequently, we’ve found that they work well for new clients with reasonably small and uncomplicated repositories. However, as the complexity grows with more tags, branches and merge points, the likelihood of producing a corrupted or logically-incorrect Subversion dump file increases.

This problem is illustrated by a recent repository conversion in which the new client used a tool to convert from their existing VSS repository to a Subversion dump file, preparing for their migration to Upon cursory inspection, the dump file contents looked reasonable, but some number of revisions into the loading process, svnadmin reported the following error:

<<< Started new transaction, based on original revision 33
svnadmin: File not found: transaction '30-1', path '/src/docs/ChangeLog'
    * editing path : src/docs/ChangeLog ...

After reading through the dump file, we discovered that it contained a sequence of operations on the “/src/docs/ChangeLog” file that occurred before the file had been added to the repository. We’ve also run across negative-numbered revisions and attempts to delete files from locations that don’t exist. All of these situations will abort repository loading.

These errors aren’t difficult to detect, and even fix, by parsing through the dump file and rewriting it slightly with the Perl CPAN module SVN::Dumpfile. This module extracts data from a dump file and also allows new data to be inserted into it.

In order to make it easier to test software that generates Subversion dump files, we are building on the work of SVN::Dumpfile and creating a sequence of dump file validation tests. Not even the svnadmin tests included in the Subversion distribution parse dump files, and perhaps this work will migrate into their suite as well.

If you have any suggestions or ideas for specific tests that you would like included, please let us know.

Call me - Greg Larkin: error

November 26, 2008

Graphical Display of Subversion Repository Revision History in Trac

Filed under: Source Code Control — Tags: , , , — Greg Larkin @ 8:15 pm

Hi everyone,

Once again, I bring you another useful Subversion-related tool. This time, I recently discovered a Trac plugin named RevtreePlugin. If you want to display your Subversion branches and revisions graphically inside of Trac, this is the plugin for you!

As I’ve noted before, I’m a FreeBSD ports tree committer now, so I absolutely had to build a port for this plugin. You can find the details of the port in FreshPorts, and if you’re using FreeBSD (you are, aren’t you?), install it like so:

cd /usr/ports/www/trac-revtree && make install clean

You will be presented with some options to enable some display enhancers. Make sure to enable the mergeinfo enhancer, and you’ll see some interesting graphs if you’ve been merging a lot.

Happy Thanksgiving!

Call me - Greg Larkin: error

November 3, 2008

How To Dump a Remote Subversion Repository

Filed under: Source Code Control — Tags: — Greg Larkin @ 10:53 am

Hi folks,

I run across a lot of software utilities in my day job and as a FreeBSD ports tree committer. Recently, I wanted to generate a Subversion dump file from a repository, but I didn’t have access to the machine where it was housed. Since svnadmin dump requires direct access to the Subversion repository directory, I started looking around for another way.

After some online searching, I found rsvndump, and it works great! Here is an example use to fetch the first 10 revisions of the public TWiki Subversion repository:

[glarkin@fbsd70 ~]$ rsvndump –stop 10 > twiki.dump
* Dumped revision 0 (local 0).
* Dumped revision 1 (local 1).
* Dumped revision 2 (local 2).
* Dumped revision 3 (local 3).
* Dumped revision 4 (local 4).
* Dumped revision 5 (local 5).
* Dumped revision 6 (local 6).
* Dumped revision 7 (local 7).
* Dumped revision 8 (local 8).
* Dumped revision 9 (local 9).
* Dumped revision 10 (local 10).
[glarkin@fbsd70 ~]$ ls -l twiki.dump
-rw-r–r–  1 glarkin  glarkin  334658 Nov  3 09:43 twiki.dump

Once you have a dump file, you can easily create a local Subversion repository and use svnadmin load to populate the new repository.

Of course, I’m now honor-bound to create FreeBSD ports for useful utilities like this if they don’t already exist. If you’d like to install rsvndump on FreeBSD, just do the following:

cd /usr/ports/devel/rsvndump && make install clean

Call me - Greg Larkin: error

February 22, 2008

Super-Configurable Subversion Notifications

Filed under: Source Code Control — Tags: , , , — Greg Larkin @ 8:13 pm

Hi everyone,

Are you using the Perl SVN::Notify module to generate your Subversion commit notification emails yet? If not, go install it, and try out svnnotify. It generates very nice-looking text or HTML-formatted emails, and it has loads of options to make it do just what you want. Place as many svnnotify commands as you like in your Subversion post-commit hook script, and off you go.

Unfortunately, with a large number of developers and many different projects in your repository, your post-commit script may start to look like this:

svnnotify -r $2 -d -P "SVN Commit " -O -H HTML::ColorDiff -p $1 -t
svnnotify -r $2 -H  Mirror::Rsync -p $1 --rsync-host \
    --to /usr/local/www/htdocs --rsync-delete=yes
svnnotify -r $2 -d -P "SVN Tag Creation " -O -H HTML::ColorDiff -p $1 \
svnnotify -r $2 -d -P "SVN Branch Creation " -O -H HTML::ColorDiff -p $1 \
svnnotify -r $2 -d -P "SVN Vendor Drop " -O -H HTML::ColorDiff -p $1 \
svnnotify -r $2 -d -P "ClientA Special Version Commit " -O -H HTML::ColorDiff \
    -p $1 -x

Yikes -that’s bound to keep any Subversion administrator up nights! However, help is on the way from Perl developer John Peacock.

John has created a nice wrapper around SVN::Notify named SVN::Notify::Config. This Perl module transforms the mess of command lines above to a cleaner, easily-editable form using YAML to express the desired svnnotify options.

The equivalent post-commit hook using SVN::Notify::Config is now:

#!/usr/bin/perl -MSVN::Notify::Config=$0
 --- #YAML:1.0
   PATH: "/usr/bin:/usr/local/bin"
   handler: HTML::ColorDiff
   subject-prefix: "SVN Commit "
   handler: Mirror::Rsync
   to: /usr/local/www/htdocs
   rsync-delete: yes
   handler: HTML::ColorDiff
   subject-prefix: "SVN Tag Creation "
   handler: HTML::ColorDiff
   subject-prefix: "SVN Branch Creation "
   handler: HTML::ColorDiff
   subject-prefix: "SVN Vendor Drop "
   handler: HTML::ColorDiff
   subject-prefix: "ClientA Special Version Commit "

That’s a lot more readable, and I expect it will be easier to maintain, even though there are more lines in the file. I’m still boning up on the YAML syntax and how this module uses it, so there may be some optimizations to be made. Feel free to send corrections!

The other thing I like about using YAML this way is it lends itself well to hooking into a notification management web interface, and that makes it even easier to keep those emails flowing where they should.

Call me - Greg Larkin: error

December 14, 2007

Subversion 1.5 Is on Its Way

Filed under: Source Code Control — Tags: , , , — Greg Larkin @ 3:40 pm

Dear readers,

I participated in a CM Crossroads webinar on Wednesday that discussed new features appearing in Subversion 1.5. There are some excellent bits on the way, and the release is currently predicted for Q1 2008.

First and foremost, the new release will support Merge Tracking. This means that you will no longer have to manually keep track of which merges you’ve applied and which ones you’ve intentionally skipped. Recording this information becomes harder and harder the longer a branch lives.

The new Merge Tracking feature will also support querying of the merge history. I hope this functionality is similar to the cleartool findmerge command from IBM/Rational ClearCase. I began using CC in the early ’9os, and the findmerge command seemed like magic at the time. It was a huge time-saver when preparing complicated releases based on the work of multiple developers.

Subversion 1.5 also optionally launches a graphical resolver when a conflict is detected within a merge operation over a large number of files. This allows you to manage conflicts as they happen instead of waiting until the very end of the operation to go back and manually edit the markers within each problematic file.

WebDAV write-thru proxies is an interesting new feature for SaaS providers (i.e.!). It allows us to set up a number of repository proxies at various locations around the globe. Users point to the repository closest to them and perform read-only operations on it. When a write operation is performed, the proxy forwards the request to the master repository. Changes are mirrored back to the proxy for future read-only operations. I personally look forward to experimenting with this feature to increase performance for our world-wide user base.

One final important change also concerns service providers or those of you who operate your own repositories. If you use the FSFS repository format, it stores all revision files in one monolithic directory. The more revisions you have in your repository, the slower certain repository operations work. UNIX filesystems perform much better as trees of directories and files instead of one enormous directory with thousands of files in it. Subversion 1.5 supports a hierarchical directory tree repository format, removing the performance bottleneck as the number of revisions grows.

I hope that gives you a taste of what’s arriving in Subversion in 2008. For the latest details, keep an eye on the Subversion project status page.

Call me - Greg Larkin: error

Powered by WordPress