Subversion update externals to a date

Question

I'm working on a large, established project under SVN control. Many parts of the code base are being checked out as externals, but are being actively worked on by other people.
I wanted to update my entire working copy, externals and all, so that it reflected the various repositories' HEADs at a specific point in time. My initial attempt was:

svn up -r'{20090324}'

This updates the current directory to the specified date, but updates all externals to the current date. Updating the externals one at a time works as expected.
I understand that due to the nature of externals, a single update couldn't work with a revision number, but why doesn't it work with a date?
What's the best way to achieve the point-in-time effect that I'm looking for, without having to maintain a script that hard-codes the various externals?

I'm running a Linux system.

Leporide

+1 excellent question. Personally I'm starting to think that externals should never have been allowed to track the HEAD of other repositories. Always specifying the exact revision of the external is much safer. – Psychophysics

Answer 1

This is inefficient, in that it calls svn update more often than (usually) required. Otherwise, it is short an sweet:

Unix:

find . -name .svn -execdir svn update -r {2010-08-30} \;

Windows:

forfiles /m .svn /s /c "cmd /c svn up -r {2010-08-30}"

Admonish

That works quite nicely. A bit slow on large directory trees but it gets the job done. And has the added benefit that it's small enough to remember. Thanks for registering to solve this! – Leporide

This won't touch externals to single files - they'll stay at their most recent revision – Aegaeon

I had no idea that forfiles launches cmd with the current directory set to the directory containing the file, as evident by executing forfiles /m .svn /s /c "cmd /c cd". I can't even find any documentation saying that it does that...but experimentally it does! – Sigfrid

Surely this will update each external twice (or more), once from the top directory, then when updating the external itself? Because the --ignore-externals flag Is missing, as fixed in maniek's solution below. – Preparatory

Using powershell, the command could be: ls -r -dir -hidden | where name -eq .svn | %{svn up --ignore-externals -r {2010-08-30} $_.parent.fullname} – Preparatory

Thanks a lot. One issue I found here: if the initial working copy has an svn:external to a folder that did not exist yet at the target date, the svn up for that folder will fail. The error message will be something like "svn: E160005: Target path 'xyz' does not exist". In that case, you mostly just need to delete that folder. – Bashaw

Answer 2

When using svn:externals, it is generally a bad idea to use an external without a revision number. It means that it becomes hard to correlate the version of the external with the version of the containing project; I know this the hard way, from trying to track down some history in a project that contained externals, and I would have to guess which revision corresponded to the revision in the containing project (sometimes it was earlier because someone had updated the external project and then updated the containing project, sometimes it was later because someone had edited files directly in the external checkout and then committed it).

Instead, as suggested by the tip box a couple paragraphs into the externals section in the subversion book, you should always commit externals with a revision number. That way, whenever you check out a particular revision of the containing project, the appropriate revision of the external will also be checked out. It does mean a little more work, as you have to update the revision number in the svn:externals property every time (we wrote a script to do it automatically), but in the long run it is a much better solution.

edit: Here is the skeleton of the script we used (a rake task) for conveniently updating the external and keeping everything in sync.

desc 'Update external for this project (rake update_external r=17789)'
task :update_external do |t|
  rev = ENV['r']
  rev =~ /^\d+$/ or raise "Invalid SVN revision number: r=<#{rev}>"

  # Update the project.
  sh "svn update"

  URL = 'svn+ssh://example.com/external/trunk'
  sh "svn propset svn:externals 'external -r#{rev} #{URL}' containing/directory"

  # Update again -- to put the externals back to the right revision.
  sh "svn update"
end

Subadar

Unfortunately, I don't have much chance of changing the current setup, though I very much like the idea of automatically updating the externals' revision numbers with a script. Nonetheless, I'm going to leave the question open for a while, because I'm sure there are others in the same boat as me. – Leporide

@Brian When in your process is your script executed? Is it run manually or automated? – Baseler

Answer 3

This is the best solution to the problem I found to date (it's a tricky problem - subversions devs should fix it in the core). This example deals with mplayer in particular but you should easily see the logic.

; fetch the rev I want without including the externals
svn checkout -r "$REV" --ignore-externals \
    svn://svn.mplayerhq.hu/mplayer/trunk

; grab the date of that rev from the svn info output
DATE=`svn info trunk|sed -n '/^Last Changed Date/s/.*: \(.*\) (.*/\1/p'`

; fetch the externals using that date
svn checkout -r "{$DATE}" \
        svn://svn.mplayerhq.hu/ffmpeg/trunk/libavutil \
        svn://svn.mplayerhq.hu/ffmpeg/trunk/libavformat \
        svn://svn.mplayerhq.hu/ffmpeg/trunk/libavcodec \
        svn://svn.mplayerhq.hu/ffmpeg/trunk/libpostproc

Ernestinaernestine

Answer 4

Based on Dave Cohen's answer, but quite a bit faster:

find . -name '.svn' -execdir svn up --ignore-externals -r '{2014-02-05}' \;

Moue

Answer 5

I still haven't got a perfect solution, but this one comes close:

svn propget svn:externals | sed -e 's/ .*$//g' | xargs svn up -r'{20090324}'

This works in my case because there's no recursive externals, and all externals are defined with no spaces in the directory or a revision number, so the regular expression can easily chop off the trailing repository path.

I'm sure there's better regex that will solve the problem generically, though.

Edit: Actually the more I think about this, the more problems I see. The biggest of which is that it's using the svn:externals from the current version, rather than the svn:externals of the version at the specified date. This is even more complex than I first thought.

Leporide

Answer 6

This is tricky, and I'm afraid I can't offer a good solution to your current situation - but Brian has given the answer on how to avoid it.

The avoidance comes down to a little bit of repository theory - basically it must not be possible to modify any source code for your project without a corresponding revision appearing in trunk.

By pointing all externals to tags or specific revisions, no changes from them can appear in the main project history without committing a change to the external reference. But if you point an external to a moving trunk, a change to the external will not show up in the main project's timeline at all - leaving you in the position you're in.

Personally, I've taken the view that externals should be treated and released as independent projects, hence all externals point to tags. During heavy parallel development, it's fine to 'switch' an external to trunk, or to have an unstable development branch temporarily pointing to an external trunk, but the mainline project trunk always points to a stable external, and it's a conscious decision to upgrade. This view may be overkill for your situation, but it's worth seeing other possibilities.

Kentiggerma

Answer 7

nice utility that will freeze externals given a path. we use this util to freeze externals after we create a tag from the trunk:

http://svnxf.codeplex.com/

Keck

Answer 8

KSvn might have this feature. http://subversion.tigris.org/links.html#desktop-integrations

Predisposition

Answer 9

The revision/date/etc you're updating the main WC to does not get passed through to the externals when they are being updated. In the absence of a specific revision specified in the externals definition, they will always track the head of whatever they point at. If you specify a revision there, then that's the only revision you'll ever get. I'm pretty sure what you're trying to do is impossible -- it's an approach I tried to use to solve a problem I was having, as I describe in this question. (I never did solve that problem, though I think the proxy idea that's mentioned there could do it. It probably won't help you though)

Plaintive

You should think yourself lucky that you had the sense to see the issues beforehand -- I hadn't even considered this problem until I suddenly had to roll everything back at once. – Leporide

Answer 10

As I have recently had a similar problem, I wrote a little script to checkout a repository at a specific revision while also checking out the externals at the date of that revision: https://gist.github.com/3413952

Useful if you need to find the source of a bug and want to have something similar to git's bisect feature.

English

Answer 11

The problem here is that you and your colleagues have not been using explicit revision numbers with your externals. You should begin at once!

Intuitively I might have thought that checking out something from a specific date would recursively "peg" the externals look-up to that date, even where the externals point to some HEAD revision. But, well, it doesn't.

Hottempered

Answer 12

Let svn do the recursion for you.

The tempfile and tee are only here so that you can see the full output:

SVN_UP_OUTPUT=$(mktemp SVN_UP_OUTPUT.XXXXX)
svn up -r$REVISION | tee $SVN_UP_OUTPUT
cat $SVN_UP_OUTPUT | egrep '^Fetching external' | egrep -o "'.*'" | sed -e "s/'//g" | while read DIR;do
    echo $$ svn up -r$REVISION "$DIR"
    svn up -r$REVISION "$DIR"
done
rm $SVN_UP_OUTPUT

If you don't care about the output, it can be shortened to this:

svn up -r$REVISION | egrep '^Fetching external' | egrep -o "'.*'" | sed -e "s/'//g" | while read DIR;do
    svn up -r$REVISION "$DIR"
done

And of course, in your case:

REVISION='{20090324}'

Aegaeon

Note: There's probably issues using this across added/removed externals. – Aegaeon

Answer 13

I like the windows solution:

forfiles /m .svn /s /c "cmd /c svn up -r {2010-08-30}"

Unfortunately, it does not work for files with extern definitons.

But the commands returns a message for each external file, so, it may be handled by hand:

Fetching external item into 'img\button_trigger.svg':
External at revision 8173.

Thank you all, Harald

Newborn