How to create a test run and result using the Team Foundation Server API?

Question

I have found several samples about retrieving test results using the TFS API, but no documentation on creating results programmatically. My aim is to create a lightweight alternative to using Microsoft Test Manager for running manual tests. Does anyone have experience with this? Are there any examples out there that I'm missing?

Here's what I have so far:

ITestCaseResult CreateNewTestCaseResult(ITestSuiteEntry testCaseEntry)
{
    var run = testCaseEntry.TestSuite.Plan.CreateTestRun(false /* not automated */);
    run.AddTest(testCaseEntry.TestCase.Id, suiteEntry.TestSuite.DefaultConfigurations[0].Id, suiteEntry.TestSuite.Plan.Owner);
    run.Save(); // so that results object is created
    return run.QueryResults()[0];
}

I'm not sure if this is the correct way to initate a new run, and I'm not sure how to record results for each action of the test.

Phip

Answer 1

Update 15 Aug 2012:

The sample below has now been integrated into my open source TFS Test Steps Editor tool. In the latest version it gained the ability to publish test results to TFS. See the source on GitHub.

---

I now have working code for publishing test results. Note, the following code accepts ITestPoint (this represents a test case within a particular suite) and has some of my internal classes (not included) that just provide outcome and attachment paths for each step.

var tfsRun = _testPoint.Plan.CreateTestRun(false);

tfsRun.DateStarted = DateTime.Now;
tfsRun.AddTestPoint(_testPoint, _currentIdentity);
tfsRun.DateCompleted = DateTime.Now;
tfsRun.Save(); // so results object is created

var result = tfsRun.QueryResults()[0];
result.Owner = _currentIdentity;
result.RunBy = _currentIdentity;
result.State = TestResultState.Completed;
result.DateStarted = DateTime.Now;
result.Duration = new TimeSpan(0L);
result.DateCompleted = DateTime.Now.AddMinutes(0.0);

var iteration = result.CreateIteration(1);
iteration.DateStarted = DateTime.Now;
iteration.DateCompleted = DateTime.Now;
iteration.Duration = new TimeSpan(0L);
iteration.Comment = "Run from TFS Test Steps Editor by " + _currentIdentity.DisplayName;

for (int actionIndex = 0; actionIndex < _testEditInfo.TestCase.Actions.Count; actionIndex++)
{
    var testAction = _testEditInfo.TestCase.Actions[actionIndex];
    if (testAction is ISharedStepReference)
        continue;

    var userStep = _testEditInfo.SimpleSteps[actionIndex];

    var stepResult = iteration.CreateStepResult(testAction.Id);
    stepResult.ErrorMessage = String.Empty;
    stepResult.Outcome = userStep.Outcome;

    foreach (var attachmentPath in userStep.AttachmentPaths)
    {
        var attachment = stepResult.CreateAttachment(attachmentPath);
        stepResult.Attachments.Add(attachment);
    }

    iteration.Actions.Add(stepResult);
}

var overallOutcome = _testEditInfo.SimpleSteps.Any(s => s.Outcome != TestOutcome.Passed)
    ? TestOutcome.Failed
    : TestOutcome.Passed;

iteration.Outcome = overallOutcome;

result.Iterations.Add(iteration);

result.Outcome = overallOutcome;
result.Save(false);

Phip

Answer 2

Test Action does not appear to have properties for setting pass/fail or to add attachments.

public interface ITestAction : INotifyPropertyChanged {
    int Id { get; }
    ITestBase Owner { get; }
    ITestActionGroup Parent { get; }

    ITestAction CopyToNewOwner(ITestBase newOwner);
    void MoveToNewOwner(ITestBase newOwner); }

That is done at the parent level (TestCase).

ITestCaseResult result = run.QueryResults()[0];
IAttachmentCollection collection = result.Attachments;
string x = result.Comment;

Here is how you correctly initiate a new run:

namespace SampleRunCreation
{
    class Program
    {
        static void Main(string[] args)
        {
            TfsTeamProjectCollection tfs = TfsTeamProjectCollectionFactory.GetTeamProjectCollection(new Uri("http://pradeepn-tcm:8080/tfs/DefaultCollection"));
            ITestManagementTeamProject project = tfs.GetService<ITestManagementService>().GetTeamProject("Pradeep");

            // Create a test case.
            ITestCase testCase = CreateTestCase(project, "My test case");

            // Create test plan.
            ITestPlan plan = CreateTestPlan(project, "My test plan");

            // Create test configuration. You can reuse this instead of creating a new config everytime.
            ITestConfiguration config = CreateTestConfiguration(project, string.Format("My test config {0}", DateTime.Now));

            // Create test points. 
            IList<ITestPoint> testPoints = CreateTestPoints(project,
                                                            plan,
                                                            new List<ITestCase>(){testCase}, 
                                                            new IdAndName[] { new IdAndName(config.Id, config.Name) });

            // Create test run using test points.
            ITestRun run = CreateTestRun(project, plan, testPoints);

            // Query results from the run.
            ITestCaseResult result = run.QueryResults()[0];

            // Fail the result.
            result.Outcome = TestOutcome.Failed;
            result.State = TestResultState.Completed;
            result.Save();

            Console.WriteLine("Run {0} completed", run.Id);
        }

        private static ITestCase CreateTestCase(ITestManagementTeamProject project,
                                                string title)
        {
            // Create a test case.
            ITestCase testCase = project.TestCases.Create();
            testCase.Owner = null;
            testCase.Title = title;
            testCase.Save();
            return testCase;
        }

        private static ITestPlan CreateTestPlan(ITestManagementTeamProject project, string title)
        {
            // Create a test plan.
            ITestPlan testPlan = project.TestPlans.Create();
            testPlan.Name = title;
            testPlan.Save();
            return testPlan;
        }

        private static ITestConfiguration CreateTestConfiguration(ITestManagementTeamProject project, string title)
        {
            ITestConfiguration configuration = project.TestConfigurations.Create();
            configuration.Name = title;
            configuration.Description = "DefaultConfig";
            configuration.Values.Add(new KeyValuePair<string, string>("Browser", "IE"));
            configuration.Save();
            return configuration;
        }

        public static IList<ITestPoint> CreateTestPoints(ITestManagementTeamProject project,
                                                         ITestPlan testPlan, 
                                                         IList<ITestCase> testCases, 
                                                         IList<IdAndName> testConfigs)
        {
            // Create a static suite within the plan and add all the test cases.
            IStaticTestSuite testSuite = CreateTestSuite(project);
            testPlan.RootSuite.Entries.Add(testSuite);
            testPlan.Save();

            testSuite.Entries.AddCases(testCases);
            testPlan.Save();

            testSuite.SetEntryConfigurations(testSuite.Entries, testConfigs);
            testPlan.Save();

            ITestPointCollection tpc = testPlan.QueryTestPoints("SELECT * FROM TestPoint WHERE SuiteId = " + testSuite.Id);
            return new List<ITestPoint>(tpc);
        }

        private static IStaticTestSuite CreateTestSuite(ITestManagementTeamProject project)
        {
            // Create a static test suite.
            IStaticTestSuite testSuite = project.TestSuites.CreateStatic();
            testSuite.Title = "Static Suite";
            return testSuite;
        }

        private static ITestRun CreateTestRun(ITestManagementTeamProject project,
                                             ITestPlan plan,
                                             IList<ITestPoint> points)
        {
            ITestRun run = plan.CreateTestRun(false);
            foreach (ITestPoint tp in points)
            {
                run.AddTestPoint(tp, null);
            }

            run.Save();
            return run;
        }
    }
}

Reference

Strawn

That's a start, basically the same point I've gotten to in my example. What I'm looking for is recording pass/fail for each action within the test case, setting comments, and attaching files to each action. – Phip

I don't think from reading through the API that you can do exactly what you are wanting to do. There are no associations between comments/attachments and actions inside the API. – Strawn

I did some more reading, I think we're nearly there. ITestCaseResult has Iterations (collection of ITestIterationResult), each of which has Actions (collection of ITestActionResult), each of which has Outcome and Comment. I will stub something out and report back. – Phip

But how would you add an attachment per action? That's cool yeah let me know always good to learn something new. – Strawn

Oh my bad, ITestActionResult has Attachments collection too! Will post code soon. Thanks for pointing me in the right direction. – Phip

Ah cool. Yeah I would be interested in seeing it all when it is done. – Strawn

0down vote @NgM how do i use your code above to add a test case under a particular TestPlan and then under TestSuite ? So finally i want my test case to appear in MTM at Project -> Test Plan -> TestSuite . – Coneflower