Unfortunately, there is no supported or easy way around having to install VS on the build agent machine in 2005 or 2008 (There will be a test agent installer in 2010). UPDATE: See this post from Steve Smith for more info which says pretty much the same thing
It's not just a matter of the assemblies being missing - if you want to run the tests, the runner is not just a separate little EXE and a DLL.
Yes, hard to believe! Needless to say, very few other test frameworks on the planet have this restriction, so unless you have a lot of tests, you could consider moving, for a variety of reasons which are covered in many places, example: The fundamental problems and impracticality of using MSTest...
EDIT: Prompted by Rihan's reply, I binged up the following Running mstest without Visual Studio. - It's not fully supported, but it 'works'...
EDIT 2: Running MSTest without Visual Studio - Gallio to the rescue looks a lot more promising in terms of being supported and non-hacky. NB see @Johannes Rudolph's comment on this post tho'
EDIT 3: Added info re 2010 status on this question
NOTE: I have a similar question for 2008 regarding what's required to support the /publish parameter of MSTest:- Running MSTEST.exe /publish on a TeamBuild server, what are the prerequisites?