The mathematics behind Diffie-Hellman and RSA are different enough so that an RSA key can't work for DH.
Diffie-Hellman
DH is a key-exchange, it produces a shared key given two key-pairs. It doesn't encrypt data directly. (But it's easy to build encryption by combining DH with symmetric encryption).
Both sides agree on a group, in the simplest case defined by g
and p
where p
is a safe prime.
A private key is simply a number a
, the corresponding public key A
is g^a mod p
. A shared key an be computed since exponentiation is commutative, so
A^b = (g^a)^b = g^(ab) = (g^b)^a = B^a
For this to work both keypairs need to use the same group. The easiest way to achieve this is picking one particular group and hardcoding it into your protocol / application.
RSA
RSA encrypts a message m
with the public key, producing a ciphertext c
. So it only uses one key-pair, the key-pair of the recipient.
The private key is a fixed exponent e
(typically 65537) and a pair of primes p
and q
.
The corresponding public key is e
and the product of the primes n = p * q
.
Encryption happens by computing c = m^e mod n
which can be revered only if you know p
and q
but not if you only know n
.
Comparison
You can produce a shared key using RSA by generating a random key as sender and encrypting it using the recipient's public key. This works for encrypting a message to the recipient without interaction. But it does not authenticate the sender since they can control the shared key. So if you can't use RSA to achieve non interactive deniable authentication. But you can sign the message with RSA (but don't use the same keypair for signing and encryption), giving you non repudiable authentication.