"Call rejected by callee" errors happen when the instance that you are connected/connecting to is currently in an interactive mode: an open dialog for example. Or, in Excel, a cell being edited, or even being in a state where a cell being edited was interrupted by the user switching away from the application - when (s)he returns it may look that the edit was completed, but the interactive mode is not ended until a different cell is selected.
Because of this I don't understand why you were getting this error when connecting unless another instance was already open. If there is no instance open (and visible), Word cannot be in interactive mode and you shouldn't have been getting the error. Is it possible your remedy merely circumvented the real problem?
No matter what though, you are in a situation where you are trying to connect to an instance that is in interactive mode. Either beforehand, or caused by your code. As you switched from XP to Windows 7, UAC does come to mind as a possible culprit.
I'd do away with the work-around, and see where that takes you.
For Word automation I always make sure that:
- I connect to a dedicated instance by using a ConnectKind of ckNewInstance and
- make sure I do not make my dedicated instance visible or
- make sure I only make it visible after I am all done and can turn the instance over to the user.
If you have no option but to automate against a visible (and thus non-dedicated) Word instance, you will simply have to deal with the possibility of this error coming up. When it does, alert the user to what is happening and make sure you offer a retry.
Update
The thread on the Embarcadero forums mentioned in the comments by @Hendra includes a link to some very useful MSDN documentation: Fixing 'Application is Busy' and 'Call was Rejected By Callee' Errors
WordApp
? Is it a TWordApplication component, or a generic OleVariant using late binding? – ChelyuskinTWordApplication.ConnectKind
tockRunningOrNew
? This is supposed to connect to a running instance if there is one, and open a new one automatically if there isn't. – Chelyuskin