I found that it helped if you set your default browser to IE (I've only used IE8 with Silverlight). That way, when you press F5, Visual Studio does seem to be able to connect up the debugger to the Silverlight plugin. You should then be able to use the VS debugger.
I certainly had difficulty trying to debug Silverlight applications with Firefox. Firefox runs plugins (including the Silverlight plugin) in a separate process, and I guess VS isn't smart enough to realise that Silverlight isn't running in the process it started. You can manually attach the VS debugger to the plugin-container process, but (if my memory serves me correctly), I didn't find this to always be reliable. I certainly found debugging a lot more reliable after switching my machine's default browser to IE.
Also, in your comment on Detroitpro's answer, you wrote
It is telling the that no symbols have been loaded even before i connect with a browser, so i don't think that is is that[...]
If you put a breakpoint in before you start your Silverlight app in a browser, then you probably will get a 'no symbols have been loaded' message. When you start debugging, Visual Studio will start the ASP.NET dev server, to serve up an HTML page that contains your Silverlight app if nothing more. The 'no symbols' message means that VS couldn't find the class you put the breakpoint in in any of the processes its debugger is connected to. (At this point, the only process the VS debugger is attached to is the ASP.NET dev server.) Once you open a browser to view a page containing your Silverlight app, and connect the VS debugger to the process running the Silverlight plugin, it should get rid of the 'no symbols' message.
Finally, another way of telling whether the VS debugger has connected to a process running your code is to bring up the Modules window (Debug > Windows > Modules). If your app's DLL isn't in the modules list, you'll get the 'no symbols have been loaded' message if you attempt to put breakpoints in your app's code.