How to migrate all URLs in svn:externals properties across a repository?

Question

We are in the process of moving our SVN repositories from one machine to another one, and with it will come a new domain name for the new repo. The problem is, that within the repository, there are lots of svn:externals references to other projects within the repository. So for example, we have projectA, which has in the svn:externals properties:

external/libraryA svn://oldserver.net/repo/libraryA
external/libraryB svn://oldserver.net/repo/libraryB

...and so on. All of the URL's reference this particular domain name, so it can be easily parsed. Having already learned my lesson, I will migrate these URLs to be "svn://localhost/", but I need to find a way to go through the repository history and rewrite all of the old URLs, so that we can still check out older revisions of these projects without having broken links.

How would I go about doing this?

Toscanini

What particular lesson is it you're referring to with the "svn://localhost/"-thing? – Amends

I meant, that one should use a generic hostname instead of a specific domain name when possible, because when the domain name changes, I end up with a mess like this. =/ – Toscanini

I don't see the need of using the svn://localhost prefix. Do you know that you can use relative URLS? See subversion.tigris.org/svn_1.5_releasenotes.html#externals – Stavanger

Answer 1

As you indicated that you still want to be able to check out older revisions, the only solution is really to "rewrite" the entire history (solution D mentioned earlier).

To do this, you should:

1) Dump the contents of the entire repository using svnadmin dump:

$ svnadmin dump /path/to/repos > original-dumpfile
* Dumped revision 0.
* Dumped revision 1.
* Dumped revision 2.
* Dumped revision 3.

2) Edit the dump file, to change the svn:externals URLs. This is the most difficult part: Assuming the repository contains binary data as well, opening the dump file in a plain text editor will most likely corrupt the dump file. I've had good experiences using a so-called "hex-editor", for instance the Freeware Hex Editor XVI32

3) Create a new repository and load the modified dumpfile into it:

$ svnadmin create newrepos
$ svnadmin load newrepos < modified-dumpfile

For more information, you might also be interested in this link:
http://svnbook.red-bean.com/en/1.1/ch05s03.html

NOTE: Subversion 1.5 actually added support for relative URLs in the svn:externals property, which can precisely prevent these sort of problems in the future:
http://subversion.tigris.org/svn_1.5_releasenotes.html#externals

Midday

This is the solution, but you really don't want to open up a -even moderately sized- svn dump file in an editor... 'sed' to the rescue! – Intersect

You should never edit an SVN dump with a text editor unless you are willing to deal with the MD5 and SHA1 hashes for the file and prop data. The manual recalculation of these hashes makes hand editing a SVN dump files of anything more than a few dozen revs a daunting job. It's far better to use one of the tools created for the job such as svndumptool. – Delciedelcina

Answer 2

I'd use SvnDumpTool for this. It has exactly what you're looking for:

svndumptool transform-prop svn:externals "(\S*) (|-r ?\d* ?)http://oldserver.net(/\S*)" "\2\3 \1" source.dumpfile source-fixed-externals.dumpfile

This fixes up each external to the subversion 1.5 format, and uses relative URLs.

So svn:externals like:

external/libraryA svn://oldserver.net/repo/libraryA

become:

 /repo/libraryA external/libraryA

using server root relative URLs.

Bulbous

I don't understand the second part (after the regex) .Please can you detail more ? Thank you . – Placebo

I added a relevant link and added more clarification to the answer. Hope this helps. – Bulbous

See this post also: https://mcmap.net/q/823176/-regex-for-svndumptool?lq=1 – Delciedelcina

For all linux newbies: Unpack svndumptool with tar --no-same-owner -xvzf svndumptool-0.6.1.tar.gz, cd into the new dir, install it with ./setup.py install and then don't forget to add .py to the command line in the answer... – Down

Answer 3

I had to relocate 12 working copies across 9 users and 4 deployments. It was a simple change, replacing a domain with an IP, i.e. thing.domain.net -> 192.168.0.1

Expecting svn relocate to behave as described (traverse nested externals) I wrote a simple DOS instruction to run at each location:

for /D %G in (*) do ( cd ./%G & svn relocate http://thing.domain.net http://192.168.0.1 & cd ..)

This didn't work as expected, only relocating the parent WC.

My solution was to edit the repositories themselves (I used Tortoise Repo Browser) to change the location of the externals. Following this change an update to the relocated parent was all that was required to bring everything into line.

It would probably be a good idea to get all the Tortoise users to clear their URL history so they don't inadvertently perform operations using the old URL (it still exists in the DNS lookup):

Settings->Saved Data->URL history->Clear

Crystie

Answer 4

I edited my dump file with vi but I had to use the "-b" switch to edit in binary mode such that any characters that could be interpreted to be line endings did not get converted.

e.g. vi -b filename.dump

Also, I found that, if your URL length changes, there are string lengths that also had to be modified. For example, consider an entry that looks like this:

Node-path: trunk/src/include

Node-kind: dir

Node-action: change

Prop-content-length: 192

Content-length: 192

K13

svn:externals

V 156

MGL_ABC svn://server_name/dir1/dir2

MGL_DEF svn://server_name/dir1/dir3

When you modify those URLs, if the length of the string changes, you need to also change the "192", "192" and "156" to match the new length. I found it difficult to compute the absolute length but easy to find the differential.
For example, let's say URL 1 becomes shorter by 3 characters and URL 2 becomes shorter by 4 characters. Then, you would have to subract '7' from each of those three string length numbers.

Eirena

Answer 5

You could:

a) check out the old revision, and change your hosts-file to point the old name to the new address, then svn update. In case the URL-path also changed... well then you might as well:

b) take the time to write a script that find the properties in the current (old revision-) working copy and changes the URLs there, without committing them. OR:

c) make a note of the revision(-s) where you checked in the new property values, check out the old version, and simply do a merge those revisions (-that only affect the properties) into your working copy.

d) or, possibly, use svndump to dump the repository data, string-replace the URL in the dump, then restore it.. I would not give you any guarantee that that even works ;-)

Amends

Answer 6

All my externals were in directories named flow. I fixed the URLs in my externals with this one-liner (bash shell):

for p in $(find -maxdepth 4 -name flow); do svn ps svn:externals "$(svn pg svn:externals $p/.. | perl -pe 's/^(\w+) svn\+ssh.*thing\.domain\.net(.*)/$2 $1/')" $p/..; done

Distinguished