Your system is massively insecure, but I'm not trying to dissuade you or anyone from playing around with stuff like this. You should continue to. But it is vital that you consider anything you create to be a "toy" system that should never be considered or advertised as "secure".
Let's break down the security question into two parts.
- How secure is the key exchange?
- How secure is the encryption you use once you've got a shared key?
Let me answer (2) first as that will be the simplest. It will be terribly insecure unless you are smarter than all of the people who've worked on and studied TLS over the years. TLS before version 1.2 (which few sites use) is vulnerable to Chosen Ciphertext Attacks (CCAs) in principle and to the BEAST attack in practice depending on cipher suit choice. And SSL 2.0 is more badly broken.
The point is that very very smart people, working on these protocols over years, got some things wrong. There is every reason to believe that you are I working on these sorts of things on our own will make huge mistakes. The basic encryption algorithms are fine. They aren't broken. But the protocols are.
So if you haven't studied and fully understood all of the details of SSL, why they are there and how they have gone wrong in some cases, then it is almost certain that any protocol you devise will be terrible.
Now to question (2). There are two issues with this. (a) Diffie-Hellman is not designed to provide the sorts of security you probably need; and (b) I don't think that you've implemented DH correctly.
2.a:
Diffie-Hellman Key exchange, when done right, is secure for key exchange, but it does nothing for authentication. This is why the question "is it secure" is often the wrong question. It is secure for some purposes, but massively insecure for others as it isn't designed for those other purposes.
As Josh3737 pointed out, there is no way for the client and the server to know that they are talking to the right party. If Sam is the server and Charlie is the Client, there is nothing that stops Mallory from setting up her own server that masquerades as Sam. So Cathy can go through the key exchange with Mallory, thinking that she is talking to Sam. Mallory can pretend to be Charlie when talking to Sam.
Once set up this way, Mallory can act as a Man In The Middle between Sam and Charlie. When Charlie sends data intended to Sam, Mallory will decrypt it using the shared key between C and M, read it (and possibly change it), and then re-encrypt it the the shared key between M and S and send that off to S.
To solve the authentication problem, you need some sort of Public Key Infrastructure (PKI) and these are really a pain. The system of Certificate Authorities and such that we have with SSL/TLS is fraught with problems, but it remains the best system out there.
2.b:
A 512 bit public modulus along with 512 bit private keys are not strong enough. DH keys need to be bigger. I wouldn't go with anything less than 2048 bits. You might get away with 1024 bits you aren't worried about someone being able to break today's secrets five years from now.
You didn't give enough information on how your primes were selected. Not every prime will work. You need to use a "safe prime" for your modulus, otherwise there are shortcuts available for an attacker to compute the discrete logarithm.