Ahh finally found my problem.
I'm using Visual Studio 2012, which doesn't always point out the "missing" file and makes it a little harder to fix.
MY SHORT SOLUTION
Relink the "linked" configuration file in the main project. It is a linked file because the original file remains in a referenced assembly, not in the current assembly. The file wasn't marked YELLOW so it wasn't obvious that it would be the problem, but when I double clicked on it, it failed to open, so I knew this was the problem file.
The silverlight application itself had a "linked" file called "ServiceReferences.ClientConfig" which contains information needed by a referenced project to make a web service call.
So the main project didn't have the web service reference itself, but it referenced a project which did, which is why it linked the "ServiceReferences.ClientConfig" file.
MY LONG SOLUTION
Keep rebuilding the main project file, and commenting out, and excluding project by project, file by file, until you CAN build the main project again. At that point it should be clear(er) which part of the solution is causing the problem.
copy local
to false you might be missing out on a referenced assembly. – Band