How to migrate from RTC Jazz to Git?

Question

My team was forced to migrate from SVN to the RTC Jazz scm about a year ago. since we are very unhappy with RTC we have decided to abandon it and switch over to Git for version-control.

I haven't found a lot of info about migrating out of RTC in general, but I did find this Rational Adapter for Git (that I'm not sure can be used for this kind of migration).

What will be the best way to migrate our source code and change history from RTC jazz to Git?

Voice

Answer 1

Since we run into the same issue and we wanted to keep our history we started to create a python script, which automates the process to migrate RTC SCM Stream by Stream to GIT.

The script can be found here: https://github.com/rtcTo/rtc2git

Some details how the program works - Its based on the RTC CLI.

1. It inits an empty git repository clones it and in there it creates a workspace based on your oldest stream.
2. For a given stream, given by a config file, it reads the baselines of the component.
3. For each baseline of this component a compare command will be executed (to see all diffs, sadly comparing directly with latest stream throws an exception @RTC Version 5.0.1)
4. The output of the compare will be parsed to get to the necessary informations (author, comment, date etc..)
5. The change will be accepted in the workspace and shortly after the corresponding git command will be executed to do the same in git

For each stream a branch will be created

Basically that's it. If you want I can also provide some sample commands, which have helped me to get to the necessary information (information is quiet rare on specific stuff regarding RTC CLI).

EDIT: I consider the current version of the program as stable. So give it a try. :) I welcome any feedback.

Askew

Hi @user2970422 - thank you for contribution! Please add some more information about your solution, since the link may be once broken and your solution therefore wouldn't be understandable anymore. – Mccombs

@florianb This might be too long to inline here. – Voice

@Askew Looks very nice! I will try to review it soon – Voice

@Voice dont expect to much yet, since its still in creation (code is also not pretty yet). :D But to get an idea what its supposed to do and with some debugging you might can play around a bit. – Askew

@joker hello, which tool would you suggest python or java, I want to migrate huge changeset – Phenomenalism

@VedX python is easier to run and easier to adapt to changes, but has more difficulties with migrating huge repositories (due to ibm's cli api, merge conflicts happens more). the java version is harder to run (except you take docker image) and harder to adapt to changes, but has less/no problems with migrating huge repositories – Askew

Answer 2

I have tested the adapter, and I confirm it is not for migration. It is for linking a git commit with an RTC change set.

The migration itself is limited, and would involve only the source control part, not the work items or the builds.

I usually set a repo workspace to the last few baselines and import them into a git working tree. I don't import the author names though.

The import takes advantage of the --git-dir and --work-tree options of the git command.
(As illustrated in this answer or this one)

You can type from anywhere:

git --work-tree=/path/to/sandbox --git-dir=/path/to/repo/.git status|add|...

Try a status first, to see what you will import. You might add to the /path/to/repo/.git/exclude file some pattern to exclude from the import elements you don't want (like the .jazz5 folder!).
Of course, /path/to/sandbox can be any sub-folder within that sandbox: you don't have to import everything.

Repeat that process with the same sandbox, updated with different baselines/snapshots, from the oldest to the newest.

That give a crude history, which is enough to start.

Baresark

Thanks for you answer, I don't need the work items or the builds. Can you please be more specific on how to import it into a git working tree? Can I also keep the author names? – Voice

I will in an hour. I am outside, typing from my phone – Baresark

@Voice sorry for the belated edit: I was busy looking at landscape like i.sstatic.net/cZ1OV.png. – Baresark

@VonC, I wonder what forced you to check SO while being in such a wonderful place :-) – Fascination

Thanks, a very good reason indeed :) So the best revision resolution will be the historic baselines and snapshots? Is there a way keep all the revision history with authors, affected lines and all? – Voice

@Voice yes, baselines (or snapshots, for multiple components at a time) represent a coherent vision of the code base, as opposed to intermediate change sets. Plus, it isn't easy to get to an intermediate state with change sets alone (see https://mcmap.net/q/748979/-is-there-a-way-to-create-a-rtc-snapshot-or-baseline-based-on-a-past-date/6309) – Baresark

@Fascination I monitor certain tags with filter sets (stackexchange.com/filters), which are sending emails every 15 minutes, which triggers an alarm on my phone. – Baresark

@VonC, sorry for insisting, but isn't there a way to create a new Git repo with all our full revision history with all our commit comments? This is very valuable information. Thanks. – Voice

@Voice not natively through RTC itself (as a commercial tool, it has no incentive to facilitate such a complete export to a free tool). After that, you would need to use their API in order to iterate on all change sets and their associated metadata (as in https://mcmap.net/q/748980/-is-there-a-method-to-get-all-the-data-in-ibm-rational-team-concert), but the all process would be slow and cumbersome. I took only the last few baselines and their comment as a quick way to get started. – Baresark

Answer 3

I had the same job, namely to convert an entire stream with several components into separate git repositories. After giving the above mentioned python script a try, I found it too complicated (e.g. selecting all the URLs in eclipse for creating history-files, searching the baselines, etc) and not reliable at all. At least not for components with over 4000 change sets.

So I wrote another script in Perl, which works nearly out of the box, at least on Windows 7.0. The script includes a lot of workarounds for all those bugs in scm, so you don't have to bother with it.

Link to Bitbucket

It did it's job and maybe it can help you as well.

Ralf

Colatitude

Thanks for sharing. You know the reason for the complicated stuff was that I didn't found the appropriate lscm commands for this information, so Im really glad you shared this. Like that I can use this commands and improve my script as well. :) Can you explain me further what was unreliable? As far as I see your script does the same steps as mine in terms of accepting stuff. – Askew

First of all I used scm instead of lscm, because at least on my machines that was an enormous performance boost. Allthough it's still awefully slow (around 5 secs per scm call). The second thing is, that the python script indeed is reliable (sorry for that), but our sources aren't. Some of the change sets are not able to check out due to strange issues. So I had to skip them. And I made some hints in the source where to skip that "bad" change sets. That helped a lot. – Colatitude

But in reality they aren't, so I had to include a lot of other stuff. In the next days I'll make an update on GitHub. – Colatitude

Ok, thanks for clarifying. We indeed, suffer from the bad performance as well. Also a lot of stuff in the cli, doesnt seem to work properly... :( For example I wanted to use list changeset like you and if I specifiy for example 30000 as maximum (we have more changeset in one component than that) I get an servererror.... So I need to stick with the history files. However if you have some enhancements, which you think would be useful for my project as well, feel free to open a pull request :) – Askew

Had same some trouble extracting the changeset overview: Number too small and it omits changesets w/o any report. Number too big, any error you like: Server error, OOM excp., etc. Now I call fec.exe instead of scm.exe. Seems to behave a bit more relaxed. – Colatitude

Thanks for sharing this solution with us. It is much more applicable than the rtc2git solutions. But unfortunately it is just as slow as the original one. There were some national words in it what we had to fix in Migration.pl script. And there was the opportunity to implement a mapper function to keep the original authors. – Uninterested

@Uninterested Could you please elaborate on "It is much more applicable than the rtc2git solutions"? We are going to migrate soon and I'd like to know why you see a the Perl solution as the "more applicable" one. – Constantia