Is there any way to speed up git svn?

Question

I am trying to convert an SVN repository to Bit Bucket, with 18 years of history, over 6000 branches and over half a million commits.

Git svn went through the first 50k commits in about 6 hours and has spent 3 days converting the next 40k. It has crashed twice and stopped twice because of missing authors which somehow managed to evade svn log.

It has reached the part of the repo where branches started to be used more widely. It has now slowed to a crawl. The branches are causing it serious grief. It is down to maybe one commit for every 2-3 minutes. Each new branch seems to slow the whole operation down even more.

I'm not sure where it's spending time. Is it an i/o or cpu intensive operation? Atlassian recommends running the operation where SVN is; on the local disk - no network trip. I don't if that would help though as it's not spending time pulling down files from SVN. Is it running commits on the branch and that is not shown on the console?

I don't think it's feasible to pull this into Bit Bucket. I think it would be better to move all new development to Bit Bucket and continue running SVN. At the current rate of conversion it will take several years to complete.

Sophey

what is the OS you run the conversion from? Windows used to be orders of magnitude slower than linux/mac – Anglomania

Is this about a one-time migration and you only use Git afterwards, or do you want to commit back to SVN from the Git clone? – Smashed

It's a one time migration, the OS is Linux – Sophey

The problems I encountered above were due to issues with the svn repository: circular dependencies in the history, massive check-ins: 60GB file, 200k json files. With svn2git, you can use pstack to identify problems like this. The only solution is to repair the svn repo, not pretty but it has to be done if you want the migration to go smoothly. – Sophey

Answer 1

git-svn is not the right tool for one-time conversions of repositories or repository parts. It is a great tool if you want to use Git as frontend for an existing SVN server, but for one-time conversions you should not use git-svn, but svn2git which is much more suited for this use-case.

There are plenty tools called svn2git, the probably best one is the KDE one from https://github.com/svn-all-fast-export/svn2git. I strongly recommend using that svn2git tool. It is the best I know available out there and it is very flexible in what you can do with its rules files.

You will be easily able to configure svn2gits rule file to produce the result you want and it is a gazillion faster.

If you are not 100% about the history of your repository, svneverever from http://blog.hartwork.org/?p=763 is a great tool to investigate the history of an SVN repository when migrating it to Git.

---

Even though git-svn is easier to start with, here are some further reasons why using the KDE svn2git instead of git-svn is superior, besides its flexibility:

• the history is rebuilt much better and cleaner by svn2git (if the correct one is used), this is especially the case for more complex histories with branches and merges and so on
• the tags are real tags and not branches in Git
• with git-svn the tags contain an extra empty commit which also makes them not part of the branches, so a normal fetch will not get them until you give --tags to the command as by default only tags pointing to fetched branches are fetched also. With the proper svn2git tags are where they belong
• if you changed layout in SVN you can easily configure this with svn2git, with git-svn you will loose history eventually
• with svn2git you can also split one SVN repository into multiple Git repositories easily
• or combine multiple SVN repositories in the same SVN root into one Git repository easily
• the conversion is a gazillion times faster with the correct svn2git than with git-svn

You see, there are many reasons why git-svn is worse and the KDE svn2git is superior. :-)

Smashed

A fantastic tool, and a real credit to the KDE team that wrote it. – Sophey

How long did the conversion to Git take for your "over half a million commits" using svn2git? I'd like a rough estimate because I have a similar migration task now. – Frisco

The tool is available in Ubuntu Packages: sudo apt install svn-all-fast-export – Frisco

How long did the conversion to Git take. svn-all-fast-export processed our 300_000 commits in approx. 45 minutes (no branches, no tags). But only without the option --svn-ignore. This option seems to slow down the migration process dramatically because it reads and merges svn folder properties. – Frisco

Answer 2

You can just submit one branch from current repository contents without migrating whole SVN history.

Puett