vstset.console.exe assembly resolution fails

Question

I have a UnitTests.dll, to which Common.dll is referenced (both built with VS2015).

I have following directory structure:

C:\Test\
    - UnitTests.dll
    - UnitTests.runsettings    
C:\Bin\
    - Common.dll

UnitTests.runsettings content is as follows:

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
    <RunConfiguration>
        <TargetPlatform>x64</TargetPlatform>
    </RunConfiguration>
    <MSTest>
        <MapInconclusiveToFailed>True</MapInconclusiveToFailed>
        <CaptureTraceOutput>False</CaptureTraceOutput>
        <DeleteDeploymentDirectoryAfterTestRunIsComplete>False</DeleteDeploymentDirectoryAfterTestRunIsComplete>
        <DeploymentEnabled>False</DeploymentEnabled>
        <AssemblyResolution>
            <Directory Path="C:\Bin\" includeSubDirectories="true" />
        </AssemblyResolution>
    </MSTest>    
</RunSettings>

---

I invoke the tests:

C:\Test> vstest.console.exe UnitTests.dll /settings:UnitTests.runsettings /inIsolation

vstest.console.exe refers to C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe.

---

I receive following error:

Starting test execution, please wait... 
Failed   TestMethod1 
Error Message:
    Test method UnitTests.UnitTest1.TestMethod1 threw exception: System.IO.FileNotFoundException: Could not load file or assembly 'Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies. The system cannot find the file specified.

Stack Trace:
    at UnitTests.UnitTest1.TestMethod1()

More, with Fusion Log enabled:

Starting test execution, please wait... 
Failed   TestMethod1 
Error Message:
    Test method UnitTests.UnitTest1.TestMethod1 threw exception: System.IO.FileNotFoundException: Could not load file or assembly 'Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies. The system cannot find the file specified.
=== Pre-bind state information === 
LOG: DisplayName = Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
  (Fully-specified)
LOG: Appbase = file:///C:/Test 
LOG: Initial PrivatePath = NULL 
Calling assembly : UnitTests, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null.
=== LOG: This bind starts in default load context. 
LOG: Using application configuration file: 
C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.executionengine.exe.Config 
LOG: Using host configuration file: 
LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework64\v4.0.30319\config\machine.config. 
LOG: Policy not being applied to reference at this time (private, custom, partial, or location-based assembly bind). 
LOG: Attempting download of new URL file:///C:/Test/Common.DLL. 
LOG: Attempting download of new URL file:///C:/Test/Common/Common.DLL. 
LOG: Attempting download of new URL file:///C:/Test/Common.EXE. 
LOG: Attempting download of new URL file:///C:/Test/Common/Common.EXE.

Stack Trace:
    at UnitTests.UnitTest1.TestMethod1()

Do I face some kind of a caching issue? How to persuade vstest.console.exe to look for the dependencies inside C:\Bin\ (as theoretically pointed by AssemblyResolution element)?

---

UPDATE:

Submitted as a bug to MSFT on connect (with repro steps - under the DETAILS tab at the bottom).

Existing limitations of assembly resolution, and forcing the usage of DeploymentItem attribute, doesn't scale at all. Redundant maintenance cost (where developers are forced to manually keep track of which dependencies are required for the unit tests to run) introduces friction to the tests. Any issues, which appear as a result of that friction, are very hard to troubleshoot.

Biochemistry

Answer 1

It seems that the Runsettings MSDN documentation has a typo in the Directory element, the attribute path must be lowercase to work. Your runsettings file should work if you specify

    <AssemblyResolution>
        <Directory path="C:\Bin\" includeSubDirectories="true" />
    </AssemblyResolution>

Danseur

Answer 2

Set the DeploymentItemAttribute for all required files:

[DeploymentItem("Common.dll")]
[DeploymentItem("Common2.dll")]
[TestMethod]
public void TestFoo()
{
    Bar();
}

Norge

DeploymentItem attribute works when you provide full path, and set DeploymentEnabled element to True in the .runsettings file. DeploymentItem attribute does not recognize environment variables, despite it is shown in the examples on MSDN. Disappointing is also the fact that the AssemblyResolution element does not work anymore (like previously, in .testsettings file), yet it still exists. – Biochemistry

report it as bug to MS on connect: connect.microsoft.com/VisualStudio/Feedback/ – Norge

I was actually about to do it. – Biochemistry

Submitted: connect.microsoft.com/VisualStudio/Feedback/Details/3105814 (with repro steps - under the DETAILS tab at the bottom). – Biochemistry