Why is git-cvsimport missing one major patchset?

Question

This is a followup to a previous question of mine.

I can look through the output of cvsps (saved in my ~/.cvsps directory) and see a patchset for a major, complicated commit. But this commit does not show up when I do cvsimport to update my existing git repository. I know that git-cvsimport will ignore patchsets that are newer than 10 minutes, in an effort to avoid importing a CVS commit that is incomplete. I think I've also heard that it will skip patchsets for which the associated branch cannot be identified. What other reasons might be causing git-cvsimport to be ignoring my commit? How can I debug this?

Here is a little bit of the cvsps output:

patchset: 50064
date: 1238445635
author: skiphoppy
tag: 
tag_flags: 0
branch: HEAD 
branch_add: 1
descr:
My log message

-=-END CVSPS DESCR-=-
members:
file:ordinaryChangedFile; pre_rev:1.44; post_rev:1.45; dead:0; branch_point:0
file:newFileRenamedAndModifiedFromOldFile; pre_rev:INITIAL; post_rev:1.1; dead:0; branch_point:0
file:deletedFileGoneBecauseRenamed; pre_rev:1.2; post_rev:1.3; dead:1; branch_point:0
...

Does that branch_add: 1 line have anything to do with it? There are only 9 such commits in the entire repository; the other 50090 commits say branch_add: 0. Other than that I don't know what the differences are from a cvsps or git-cvsimport point of view. The commit was complicated, involving renaming several files and making associated content modifications. Definitely difficult for CVS to handle. A snap for git, of course, but hard for git to handle when the real backend is CVS.

I've got to perform the same type of complicated commit three more times, and I can't afford to spend 4 1/2 hours each time to reimport a brand new git repository from CVS due to cvsimport not being able to see my commit.

More importantly, I'm frightened by the fact that my daily runs of git-cvsimport (which I intended to put into cron, or even write a daemon to run every five minutes or so) might miss commits in the future! I have found no way to git an existing repository to pick up these commits; I can only run git-cvsimport in a brand new repository and wait forever, losing the use of many of my other branches while I'm at it.

Confluent

Answer 1

Well, this may be the problem. My git-cvsimport lines look like this:

git cvsimport -p x ...

The -p x is is supposed to pass the -x option to cvsps to tell it to ignore the cached output that it left lying around from previous runs. I thought the main reason to do that was so that the last few patchsets, which might be incomplete, will be thrown out and completed the next time you run. Turns out there may be more issues that it fixes, and this may be one of them.

I learned to run git cvsimport this way from this blog entry, which is currently one of the highest hits on Google for "git cvs." It was only in the process above, where I was trying to run git-cvsimport through the Perl debugger on output from cvsps, that I had to check and see what arguments were really going to cvsps. I learned cvsps was being run like this:

cvsps --norc x --cvs-direct ...

Instead of:

cvsps --norc -x --cvs-direct ...

And I experimentally verified that I get different output from cvsps, with some patchsets missing (I have no idea what the pattern is), when x is passed instead of -x. Thanks to Murphy's law, cvsps doesn't seem to report that this was a problem, and git-cvsimport never sees it.

So anyway, git cvsimport needs to be run like this:

git cvsimport -p -x ...

My previous versions of this repository are completely hosed at this point, but I have been able to force the last of the problem commits into them (some of the earlier commits are missing, though). So I get to go through the four-hour import process one more time, and I'm hoping that will be it!

One last tip: git-cvsimport on Windows doesn't seem to work at all. I got less than 10% of the number of commits, although I did wind up with a tree that resembles the current state of our project. It just seems to be lacking almost all the history...

Confluent

Dude, this really helps. I used the same post and got burned. – Tearing

Glad to hear it helped. There are still bugs in cvsimport, but hopefully you will not run into them. :) Are you doing a one-time conversion, or are you going to be stuck interacting with the original CVS repository in the future? – Confluent

Answer 2

Thanks a lot! I had the same problem and could fix it with your help!

I even found a way to avoid re-running cvsimport from scratch. Simply setting the 'master' and the 'origin' branch to older git commits made cvsimport re-import the patches since that including the missing ones:

find a commit ID older than the missing commits:

git log

make 'origin' point to that commit

git branch -f origin

switch to origin (so that 'master' is not the current HEAD)

git checkout origin

now make 'master' point the older commit as well

git branch -f master origin

switch back to master

git checkout master

now, time is completely rolled back and you can do the cvsimport again, this time properly

git cvsimport ...

Guenzi

Resetting master isn't necessary if you don't mind rebasing or otherwise merging it later. Just rolling back origin solved this for me. – Lost

Answer 3

I encountered such thing recently - there was a commit in CVS that didn't show up in git after git cvsimport; later commits did (it bit me hard, because I produced nonworking patches afterwards.

However, the fix was simple and stupid - I just reset the cvs/master branch before the missing commit, rerun git cvsimport, and there it was.

Minnaminnaminnie

Cool. I am going to try that, although I didn't realize I could move cvs/master (it's a remote branch in my repo). I'm also worried because I can't see why that would work, if it failed the first time. But it's a pointer in the right direction, at least! – Confluent

It's a remote branch for me too. I vaguely recall doing something very nasty - either I used plumbing or (more probably) I just did "echo 123456789abcdef > .git/refs/remotes/cvs/master" (that 123... should be the sha1 of the commit you want to reset to) – Minnaminnaminnie

I, too, can't see why the commit was missed and got in the second time. You say you're a perl programmer, yo maybe you could debug this - even though it would be nasty since it only happen once in some time. – Minnaminnaminnie

... another possibility on "what I did with the remote branch" is that I changed it to a local branch with the same name. I'm really not sure now. – Minnaminnaminnie

Answer 4

Hmm, the good news is that git-cvsimport appears to be written in Perl, and I happen to be a Perl programmer, so maybe I can at least step through this with the Perl debugger to find out what happens during that commit.

For anyone who wants to try this, you'll have to locate the full path to git-cvsimport and feed it to perl (the Perl interpreter) with the -d switch:

perl -d /usr/local/libexec/git-core/git-cvsimport

I haven't gotten all the other options worked out to do this, yet ... I'll need to feed git-cvsimport the output of cvsps directly, I think...

Confluent

If you figure it out, I hope you'll submit a bug or patch or failing unit test to the git mailing list so that this issue can be solved for everyone that uses this (unfortunately) necessary tool. – Contraption

Got it!!! It finally happened again ... a patchset with a timestamp of exactly ten minutes ago will be skipped, subsequent patchsets will be imported, and then you're hosed. I'll be reporting it. :) – Confluent