The documentation says it pretty clearly:
InvalidOperationException is used in cases when the failure to invoke
a method is caused by reasons other than invalid arguments.
...
If the method invocation failure is due to invalid arguments, then
ArgumentException or one of its derived classes, ArgumentNullException
or ArgumentOutOfRangeException, should be thrown instead.
Your method expects the arguments to be in a certain state, which could include anything including being in a "valid state" as defined by the argument type itself.
I think the main point of difference is the source of the problem:
Is it the argument or the object on which you call the method?