Common result from converting a complex non-Subversion source code repository to a Subversion dump file

Hi everyone,

Many new SourceHosting.net clients have existing repositories that they want to import into a Subversion repository here. If we’re lucky, the client already uses Subversion, and it’s no more difficult to import the repository than dumping and loading it.

Converting from CVS to SVN is quite easy using the cvs2svn script. We rarely run into any problems, and it has a flexible set of command line options. If, however, the client uses a different source code control system, the process can get trickier.

There are several repository converters available for download, including Polarion Importer for SVN and VSS2SVN. These tools are welcome additions to a release engineer’s bag of tricks, especially if different groups in an organization have not standardized on a single SCM system and you’re trying to convert everyone to Subversion.

As we’ve used the tools mentioned above more and more frequently, we’ve found that they work well for new clients with reasonably small and uncomplicated repositories. However, as the complexity grows with more tags, branches and merge points, the likelihood of producing a corrupted or logically-incorrect Subversion dump file increases.

This problem is illustrated by a recent repository conversion in which the new client used a tool to convert from their existing VSS repository to a Subversion dump file, preparing for their migration to SourceHosting.net. Upon cursory inspection, the dump file contents looked reasonable, but some number of revisions into the loading process, svnadmin reported the following error:

<<< Started new transaction, based on original revision 33
svnadmin: File not found: transaction '30-1', path '/src/docs/ChangeLog'
    * editing path : src/docs/ChangeLog ...

After reading through the dump file, we discovered that it contained a sequence of operations on the “/src/docs/ChangeLog” file that occurred before the file had been added to the repository. We’ve also run across negative-numbered revisions and attempts to delete files from locations that don’t exist. All of these situations will abort repository loading.

These errors aren’t difficult to detect, and even fix, by parsing through the dump file and rewriting it slightly with the Perl CPAN module SVN::Dumpfile. This module extracts data from a dump file and also allows new data to be inserted into it.

In order to make it easier to test software that generates Subversion dump files, we are building on the work of SVN::Dumpfile and creating a sequence of dump file validation tests. Not even the svnadmin tests included in the Subversion distribution parse dump files, and perhaps this work will migrate into their suite as well.

If you have any suggestions or ideas for specific tests that you would like included, please let us know.

Hi everyone,

Once again, I bring you another useful Subversion-related tool. This time, I recently discovered a Trac plugin named RevtreePlugin. If you want to display your Subversion branches and revisions graphically inside of Trac, this is the plugin for you!

As I’ve noted before, I’m a FreeBSD ports tree committer now, so I absolutely had to build a port for this plugin. You can find the details of the port in FreshPorts, and if you’re using FreeBSD (you are, aren’t you?), install it like so:

You will be presented with some options to enable some display enhancers. Make sure to enable the mergeinfo enhancer, and you’ll see some interesting graphs if you’ve been merging a lot.

Happy Thanksgiving!

Hi folks,

I run across a lot of software utilities in my day job and as a FreeBSD ports tree committer. Recently, I wanted to generate a Subversion dump file from a repository, but I didn’t have access to the machine where it was housed. Since svnadmin dump requires direct access to the Subversion repository directory, I started looking around for another way.

After some online searching, I found rsvndump, and it works great! Here is an example use to fetch the first 10 revisions of the public TWiki Subversion repository:

Once you have a dump file, you can easily create a local Subversion repository and use svnadmin load to populate the new repository.

Of course, I’m now honor-bound to create FreeBSD ports for useful utilities like this if they don’t already exist. If you’d like to install rsvndump on FreeBSD, just do the following:

Hi everyone,

Are you using the Perl SVN::Notify module to generate your Subversion commit notification emails yet? If not, go install it, and try out svnnotify. It generates very nice-looking text or HTML-formatted emails, and it has loads of options to make it do just what you want. Place as many svnnotify commands as you like in your Subversion post-commit hook script, and off you go.

Unfortunately, with a large number of developers and many different projects in your repository, your post-commit script may start to look like this:

svnnotify -r $2 -d -P "SVN Commit " -O -H HTML::ColorDiff -p $1 -t dev@company.com
svnnotify -r $2 -H  Mirror::Rsync -p $1 --rsync-host devserver.company.com \
    --to /usr/local/www/htdocs --rsync-delete=yes
svnnotify -r $2 -d -P "SVN Tag Creation " -O -H HTML::ColorDiff -p $1 \
    -x builder@company.com=/tags
svnnotify -r $2 -d -P "SVN Branch Creation " -O -H HTML::ColorDiff -p $1 \
    -x builder@company.com=/branches
svnnotify -r $2 -d -P "SVN Vendor Drop " -O -H HTML::ColorDiff -p $1 \
    -x vendormgr@company.com=/vendor
svnnotify -r $2 -d -P "ClientA Special Version Commit " -O -H HTML::ColorDiff \
    -p $1 -x dev@clientA.com=/branches/clientA

John has created a nice wrapper around SVN::Notify named SVN::Notify::Config. This Perl module transforms the mess of command lines above to a cleaner, easily-editable form using YAML to express the desired svnnotify options.

The equivalent post-commit hook using SVN::Notify::Config is now:

#!/usr/bin/perl -MSVN::Notify::Config=$0
 --- #YAML:1.0
 '':
   PATH: "/usr/bin:/usr/local/bin"
 '/':
   handler: HTML::ColorDiff
   with-diff:
   no-first-line:
   subject-prefix: "SVN Commit "
   to: dev@company.com
 '/':
   handler: Mirror::Rsync
   to: /usr/local/www/htdocs
   rsync-host: devserver.company.com
   rsync-delete: yes
 '/tags':
   handler: HTML::ColorDiff
   with-diff:
   no-first-line:
   subject-prefix: "SVN Tag Creation "
   to: builder@company.com
 '/branches':
   handler: HTML::ColorDiff
   with-diff:
   no-first-line:
   subject-prefix: "SVN Branch Creation "
   to: builder@company.com
 '/vendor':
   handler: HTML::ColorDiff
   with-diff:
   no-first-line:
   subject-prefix: "SVN Vendor Drop "
   to: vendormgr@company.com
 '/branches/clientA':
   handler: HTML::ColorDiff
   with-diff:
   no-first-line:
   subject-prefix: "ClientA Special Version Commit "
   to: dev@clientA.com

The other thing I like about using YAML this way is it lends itself well to hooking into a notification management web interface, and that makes it even easier to keep those emails flowing where they should.

Dear readers,

I participated in a CM Crossroads webinar on Wednesday that discussed new features appearing in Subversion 1.5. There are some excellent bits on the way, and the release is currently predicted for Q1 2008.

First and foremost, the new release will support Merge Tracking. This means that you will no longer have to manually keep track of which merges you’ve applied and which ones you’ve intentionally skipped. Recording this information becomes harder and harder the longer a branch lives.

The new Merge Tracking feature will also support querying of the merge history. I hope this functionality is similar to the cleartool findmerge command from IBM/Rational ClearCase. I began using CC in the early ’9os, and the findmerge command seemed like magic at the time. It was a huge time-saver when preparing complicated releases based on the work of multiple developers.

Subversion 1.5 also optionally launches a graphical resolver when a conflict is detected within a merge operation over a large number of files. This allows you to manage conflicts as they happen instead of waiting until the very end of the operation to go back and manually edit the markers within each problematic file.

WebDAV write-thru proxies is an interesting new feature for SaaS providers (i.e. SourceHosting.net!). It allows us to set up a number of repository proxies at various locations around the globe. Users point to the repository closest to them and perform read-only operations on it. When a write operation is performed, the proxy forwards the request to the master repository. Changes are mirrored back to the proxy for future read-only operations. I personally look forward to experimenting with this feature to increase performance for our world-wide user base.

One final important change also concerns service providers or those of you who operate your own repositories. If you use the FSFS repository format, it stores all revision files in one monolithic directory. The more revisions you have in your repository, the slower certain repository operations work. UNIX filesystems perform much better as trees of directories and files instead of one enormous directory with thousands of files in it. Subversion 1.5 supports a hierarchical directory tree repository format, removing the performance bottleneck as the number of revisions grows.

I hope that gives you a taste of what’s arriving in Subversion in 2008. For the latest details, keep an eye on the Subversion project status page.