Can TestNG see my Spock (JUnit) test results?

Question

Need to get TestNG to run my Spock Tests as TestNG is used throughout the rest of the system.
Since TestNG supports running JUnit tests, I tried this:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >

<suite name="suiteTestName" verbose="1" junit="true">
    <test name="myTestName">
        <classes>
            <class name="mypackage.artifact.SomeArtifactSpecification" />
        </classes>
    </test>
</suite>

So this actually ran the Groovy 'SomeArtifactSpecification' Spock Unit Test perfectly. I could see some I/O from the tests, but since I ran from within Eclipse I used the debugger to verify that all the tests were executing properly. My classpath has TestNG-6.8.7 and JUnit-4.10 on it. The problem is that TestNG doesn't seem to know that it did! Here is the end output:

===============================================
myTestName
Total tests run: 0, Failures: 0, Skips: 0
===============================================

Does anyone know if it is possible to get TestNG to recognize these tests?

Additional Details - added for the Bounty

In one of the comments below, Peter states that it depends upon what TestNG is doing. I investigated that as much as I could. It seems like both Junit and TestNG invoke the 'run' method of Spock's Sputnik class line 63. However, when invoked directly from JUnit, Sputnik is called from JUnit4TestClassReference. When called from TestNG, Sputnik is invoked from JUnitCore.

So the question remains: How to get TestNG to recognize the test outputs from Spock? If I am to live long and prosper, I really need to know!!! It seems like the answer may be to get TestNG to run with JUnit4TestClassReference, or to use some type of clever wrapper to make TestNG do that...

For the gory details, I posted two stack traces below. The first is from the TestNG run, which does not work, the second is from running Spock directly without TestNG.

First Stack Trace from TestNG:

org.testng.remote.RemoteTestNG at localhost:54288   
Thread [main] (Suspended (breakpoint at line 65 in ArtifactManagerSpecification))   
    ArtifactManagerSpecification.$spock_feature_0_0() line: 65  
    NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]  
    NativeMethodAccessorImpl.invoke(Object, Object[]) line: 57  
    DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43  
    Method.invoke(Object, Object...) line: 601  
    ReflectionUtil.invokeMethod(Object, Method, Object...) line: 138    
    ParameterizedSpecRunner(BaseSpecRunner).invokeRaw(Object, MethodInfo, Object...) line: 330  
    ParameterizedSpecRunner(BaseSpecRunner).invoke(Object, MethodInfo, Object...) line: 311 
    ParameterizedSpecRunner(BaseSpecRunner).invokeFeatureMethod() line: 285 
    ParameterizedSpecRunner(BaseSpecRunner).doRunIteration() line: 256  
    NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]  
    NativeMethodAccessorImpl.invoke(Object, Object[]) line: 57  
    DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43  
    Method.invoke(Object, Object...) line: 601  
    ReflectionUtil.invokeMethod(Object, Method, Object...) line: 138    
    ParameterizedSpecRunner(BaseSpecRunner).invokeRaw(Object, MethodInfo, Object...) line: 330  
    ParameterizedSpecRunner(BaseSpecRunner).invoke(Object, MethodInfo, Object...) line: 311 
    ParameterizedSpecRunner(BaseSpecRunner).runIteration(Object[], int) line: 223   
    ParameterizedSpecRunner(BaseSpecRunner).initializeAndRunIteration(Object[], int) line: 214  
    ParameterizedSpecRunner(BaseSpecRunner).runSimpleFeature() line: 205    
    ParameterizedSpecRunner(BaseSpecRunner).doRunFeature() line: 199    
    NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]  
    NativeMethodAccessorImpl.invoke(Object, Object[]) line: 57  
    DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43  
    Method.invoke(Object, Object...) line: 601  
    ReflectionUtil.invokeMethod(Object, Method, Object...) line: 138    
    ParameterizedSpecRunner(BaseSpecRunner).invokeRaw(Object, MethodInfo, Object...) line: 330  
    ParameterizedSpecRunner(BaseSpecRunner).invoke(Object, MethodInfo, Object...) line: 311 
    ParameterizedSpecRunner(BaseSpecRunner).runFeature() line: 175  
    ParameterizedSpecRunner(BaseSpecRunner).runFeatures() line: 152 
    ParameterizedSpecRunner(BaseSpecRunner).doRunSpec() line: 112   
    NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]  
    NativeMethodAccessorImpl.invoke(Object, Object[]) line: 57  
    DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43  
    Method.invoke(Object, Object...) line: 601  
    ReflectionUtil.invokeMethod(Object, Method, Object...) line: 138    
    ParameterizedSpecRunner(BaseSpecRunner).invokeRaw(Object, MethodInfo, Object...) line: 330  
    ParameterizedSpecRunner(BaseSpecRunner).invoke(Object, MethodInfo, Object...) line: 311 
    ParameterizedSpecRunner(BaseSpecRunner).runSpec() line: 91  
    ParameterizedSpecRunner(BaseSpecRunner).run() line: 82  
    Sputnik.run(RunNotifier) line: 63   
    JUnitCore.run(Runner) line: 157 
    JUnitCore.run(Request) line: 136    
    JUnit4TestRunner.start(Class, String...) line: 81   
    JUnit4TestRunner.run(Class, String...) line: 69 
    TestRunner$1.run() line: 682    
    TestRunner.runWorkers(List<IWorker<ITestNGMethod>>, String, ListMultiMap<Integer,TestMethodWorker>) line: 1005  
    TestRunner.privateRunJUnit(XmlTest) line: 713   
    TestRunner.run() line: 614  
    SuiteRunner.runTest(TestRunner) line: 334   
    SuiteRunner.runSequentially() line: 329 
    SuiteRunner.privateRun() line: 291  
    SuiteRunner.run() line: 240 
    SuiteRunnerWorker.runSuite(SuiteRunnerMap, XmlSuite) line: 52   
    SuiteRunnerWorker.run() line: 86    
    RemoteTestNG(TestNG).runSuitesSequentially(XmlSuite, SuiteRunnerMap, int, String) line: 1224    
    RemoteTestNG(TestNG).runSuitesLocally() line: 1149  
    RemoteTestNG(TestNG).run() line: 1057   
    RemoteTestNG.run() line: 111    
    RemoteTestNG.initAndRun(String[], CommandLineArgs, RemoteArgs) line: 204    
    RemoteTestNG.main(String[]) line: 175   

Second Stack Trace (that works) running Spock directly

Thread [main] (Suspended (breakpoint at line 65 in ArtifactManagerSpecification))   
ArtifactManagerSpecification.$spock_feature_0_0() line: 65  
NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]  
NativeMethodAccessorImpl.invoke(Object, Object[]) line: 57  
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43  
Method.invoke(Object, Object...) line: 601  
ReflectionUtil.invokeMethod(Object, Method, Object...) line: 138    
ParameterizedSpecRunner(BaseSpecRunner).invokeRaw(Object, MethodInfo, Object...) line: 330  
ParameterizedSpecRunner(BaseSpecRunner).invoke(Object, MethodInfo, Object...) line: 311 
ParameterizedSpecRunner(BaseSpecRunner).invokeFeatureMethod() line: 285 
ParameterizedSpecRunner(BaseSpecRunner).doRunIteration() line: 256  
NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]  
NativeMethodAccessorImpl.invoke(Object, Object[]) line: 57  
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43  
Method.invoke(Object, Object...) line: 601  
ReflectionUtil.invokeMethod(Object, Method, Object...) line: 138    
ParameterizedSpecRunner(BaseSpecRunner).invokeRaw(Object, MethodInfo, Object...) line: 330  
ParameterizedSpecRunner(BaseSpecRunner).invoke(Object, MethodInfo, Object...) line: 311 
ParameterizedSpecRunner(BaseSpecRunner).runIteration(Object[], int) line: 223   
ParameterizedSpecRunner(BaseSpecRunner).initializeAndRunIteration(Object[], int) line: 214  
ParameterizedSpecRunner(BaseSpecRunner).runSimpleFeature() line: 205    
ParameterizedSpecRunner(BaseSpecRunner).doRunFeature() line: 199    
NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]  
NativeMethodAccessorImpl.invoke(Object, Object[]) line: 57  
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43  
Method.invoke(Object, Object...) line: 601  
ReflectionUtil.invokeMethod(Object, Method, Object...) line: 138    
ParameterizedSpecRunner(BaseSpecRunner).invokeRaw(Object, MethodInfo, Object...) line: 330  
ParameterizedSpecRunner(BaseSpecRunner).invoke(Object, MethodInfo, Object...) line: 311 
ParameterizedSpecRunner(BaseSpecRunner).runFeature() line: 175  
ParameterizedSpecRunner(BaseSpecRunner).runFeatures() line: 152 
ParameterizedSpecRunner(BaseSpecRunner).doRunSpec() line: 112   
NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]  
NativeMethodAccessorImpl.invoke(Object, Object[]) line: 57  
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43  
Method.invoke(Object, Object...) line: 601  
ReflectionUtil.invokeMethod(Object, Method, Object...) line: 138    
ParameterizedSpecRunner(BaseSpecRunner).invokeRaw(Object, MethodInfo, Object...) line: 330  
ParameterizedSpecRunner(BaseSpecRunner).invoke(Object, MethodInfo, Object...) line: 311 
ParameterizedSpecRunner(BaseSpecRunner).runSpec() line: 91  
ParameterizedSpecRunner(BaseSpecRunner).run() line: 82  
Sputnik.run(RunNotifier) line: 63   
JUnit4TestClassReference(JUnit4TestReference).run(TestExecution) line: 50   
TestExecution.run(ITestReference[]) line: 38    
RemoteTestRunner.runTests(String[], String, TestExecution) line: 467    
RemoteTestRunner.runTests(TestExecution) line: 683  
RemoteTestRunner.run() line: 390    
RemoteTestRunner.main(String[]) line: 197   

Caudal

From what I know, TestNG can only run JUnit 3 tests, but not JUnit 4 tests. In that case, it won't be able to run Spock specs. – Catgut

While I am far (ok, very far!) from the expert, they apparently have added support for JUnit 4 a couple of months back. That support IS in the version I am using. – Caudal

They'd also have to add support for custom JUnit runners (@RunWith). I didn't see anything about this (or JUnit 4 support in general) in the docs, but then I only had a quick look. – Catgut

I don't really understand. Wouldn't that just be JUnit honoring the @RunWith in the spock.lang.Specification? In any event, on the above, if I put a breakpoint in the constructor of Sputnik (great name BTW!), it does go there right away. It goes to the "run" method of Sputnik as well.... – Caudal

It all depends on how TestNG happens to execute JUnit tests. – Catgut

@Mattew Farwell - just FYI: I choose not to have the stack traces as you did them because to me they don't compare against one another nearly as easily. The scrolling messes that up to me. – Caudal

Perhaps the spec executes correctly but something goes wrong when TestNG interprets the results (i.e. org.junit.runner.notification.RunListener callbacks). – Catgut

Answer 1

I investigated this further and at least I found the cause.

In org.testng.junit.JUnit4TestMethod below, Spock test names are unable to be properly resolved. I am guessing, but I think Spock uses a type of name mangling to support the nice text method name it allows. Thus, even though the TestNG code below gets a method name like "This is my test", it can not resolve that via the means shown. I believe the real method name is something like: $spock_feature_0_0

private static Method getMethod(Description desc) {
    Class<?> c = desc.getTestClass();
    String method = desc.getMethodName();
    try {
        return c.getMethod(method);
    } catch (Throwable t) {
        Utils.log("JUnit4TestMethod", 2,
                "Method '" + method + "' not found in class '" + c.getName() + "': " + t.getMessage());
        return null;
    }
}

As a result, this throws an exception on the call c.getMethod(method) - it can't resolve the Spock method name to an actual method on the class.

What seems to really 'muck up the situation' is that the code above is called by Junit (org.junit.runner.notification.RunNotifier) in this code:

private abstract class SafeNotifier {
    void run() {
        synchronized (fListeners) {
            for (Iterator<RunListener> all= fListeners.iterator(); all.hasNext();)
                try {
                    notifyListener(all.next());
                } catch (Exception e) {
                    all.remove(); // Remove the offending listener first to avoid an infinite loop
                    fireTestFailure(new Failure(Description.TEST_MECHANISM, e));
                }
        }
    }

it would be really nice for this code to not quietly remove the listener, but rather log it...

Caudal

Yes, this is very likely the source of the problem. – Catgut

To prove your theory, wouldn't renaming all your Spock test names to conventional Java method names be enough? – Lighting