EMMA with Junit, should I be testing with the instrumented classes or java classes?

Question

I've been going through this tutorial for my ANT Script for emma/junit, it states

In the <classpath> element of <junit>, change the locations of your real classes, so that you are pointing at the ${instr.dir} attribute instead of your normal target/classes folder.

Here is my emma section from of my build.xml for Apache Ant (pastebin link to full build.xml)

<!-- =================================================================== -->
<!-- Run the tests with EMMA
* depends from  compile                                                                                                                        -->
<!-- =================================================================== -->
<target name="emmatest" depends="compile" description="Tests the project with Emma Query" >
    <!-- ======================================================================= -->
    <!-- EMMA INSTRUMENTATION -->
    <!-- ======================================================================= -->
    <emma enabled="${emma.enabled}" >
        <instr instrpathref="emma.coverage.classes"
             destdir="${instr.dir}"
             metadatafile="${coverage.dir}/metadata.emma"
             merge="true"
             >
            <filter value="${emma.filter}" />
        </instr>
    </emma> 
    <junit fork="true" forkmode="once" >
         <test name="test.nz.ac.massey.cs.sdc.log4jassignment.s06005586.AppenderLayoutTest" todir="${coverage.dir}">
          <formatter type="xml"/>
          </test>
         <classpath>
                <path refid="emma.lib" />
                <pathelement location="${instr.dir}"/>
                <fileset dir="${lib.dir}">
                   <include name="**/*.jar"/>
                </fileset>
        </classpath>
         <jvmarg value="-Demma.coverage.out.file=${coverage.dir}/coverage.emma" />
         <jvmarg value="-Demma.coverage.out.merge=false" />
    </junit>
</target>

The Build Section Results for emma: JUNIT Fail Result, pointing to ${instr.dir}

 emmatest:
 [instr] processing instrumentation path ...
 [instr] instrumentation path processed in 109 ms
 [instr] [6 class(es) instrumented, 0 resource(s) copied]
 [instr] metadata merged into [C:\Users\Michael\workspace\log4jassignment.s06005586\build\emma-reports\metadata.emma] {in 42 ms}
 [junit] Test test.nz.ac.massey.cs.sdc.log4jassignment.s06005586.AppenderLayoutTest FAILED

The Build Section results for junitreport: JUNIT Pass Result, pointing to real classes ${build.classes}

test:
[junit] Running test.nz.ac.massey.cs.sdc.log4jassignment.s06005586.AppenderLayoutTest
[junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.504 sec
[junit] Running test.nz.ac.massey.cs.sdc.log4jassignment.s06005586.StressTest
[junit] Tests run: 6, Failures: 0, Errors: 0, Time elapsed: 0.526 sec

I was wondering why is my JUNIT Test AppenderLayoutTest failing with emma but passing with junit. I'm also pretty confident my coverage file isn't being created because the JUNIT for AppenderLayoutTest is failing.

Am I testing with the wrong set of classes? or what I missed?

Update: 8/28/2012 6:21 PM

I then changed the

<test 
name="test.nz.ac.massey.cs.sdc.log4jassignment.s06005586.AppenderLayoutTest"
todir="${coverage.dir}">

with

<batchtest>
    <fileset dir="${instr.dir}" includes="**/*Test.class" />
</batchtest>

And the updated Build Section Results for emma: JUNIT Fail Result, pointing to ${instr.dir} where..

emmatest:
[instr] processing instrumentation path ...
[instr] instrumentation path processed in 105 ms
[instr] [6 class(es) instrumented, 0 resource(s) copied]
[instr] metadata merged into [C:\Users\Michael\workspace\log4jassignment.s06005586\build\emma-reports\metadata.emma] {in 89 ms}
[junit] TEST test.nz.ac.massey.cs.sdc.log4jassignment.s06005586.AppenderLayoutTest FAILED
[junit] TEST test.nz.ac.massey.cs.sdc.log4jassignment.s06005586.StressTest FAILED
[junit] Tests FAILED

Update: 7:02 PM

I added <formatter type="plain" usefile="false" />, And this is the output

emmatest:
[instr] [EMMA v2.1, build 5320 (stable) (2005/06/20 22:08:27)]
[instr] instrumentation path:
[instr] {
[instr]   C:\Users\Michael\workspace\log4jassignment.s06005586\build\classes
[instr] }
[instr] instrumentation output mode: copy
[instr] metadata output file: C:\Users\Michael\workspace\log4jassignment.s06005586\build\emma-reports\metadata.emma
[instr] metadata output merge mode: true
[instr] processing dir path entry [C:\Users\Michael\workspace\log4jassignment.s06005586\build\classes] ...
[instr] instrumentation path processed in 147 ms
[instr] [6 class(es) instrumented, 0 resource(s) copied]
[instr] metadata contains 6 entries
[instr] metadata merged into [C:\Users\Michael\workspace\log4jassignment.s06005586\build\emma-reports\metadata.emma] {in 49 ms}
[junit] Running test.nz.ac.massey.cs.sdc.log4jassignment.s06005586.AppenderLayoutTest
[junit] Testsuite: test.nz.ac.massey.cs.sdc.log4jassignment.s06005586.AppenderLayoutTest
[junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
[junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
[junit]     Caused an ERROR
[junit] Illegal local variable table length 17 in method test.nz.ac.massey.cs.sdc.log4jassignment.s06005586.AppenderLayoutTest.<init>()V
[junit] java.lang.ClassFormatError: Illegal local variable table length 17 in method test.nz.ac.massey.cs.sdc.log4jassignment.s06005586.AppenderLayoutTest.<init>()V
[junit]     at java.lang.Class.forName0(Native Method)
[junit]     at java.lang.Class.forName(Unknown Source)
[junit] TEST test.nz.ac.massey.cs.sdc.log4jassignment.s06005586.AppenderLayoutTest FAILED
[junit] Running test.nz.ac.massey.cs.sdc.log4jassignment.s06005586.StressTest
[junit] Testsuite: test.nz.ac.massey.cs.sdc.log4jassignment.s06005586.StressTest
[junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
[junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
[junit]     Caused an ERROR
[junit] Expecting a stackmap frame at branch target 11 in method test.nz.ac.massey.cs.sdc.log4jassignment.s06005586.StressTest.<init>()V at offset 4
[junit] java.lang.VerifyError: Expecting a stackmap frame at branch target 11 in method test.nz.ac.massey.cs.sdc.log4jassignment.s06005586.StressTest.<init>()V at offset 4
[junit]     at java.lang.Class.forName0(Native Method)
[junit]     at java.lang.Class.forName(Unknown Source)
[junit] TEST test.nz.ac.massey.cs.sdc.log4jassignment.s06005586.StressTest FAILED
[junit] Tests FAILED

what is [junit] java.lang.ClassFormatError: Illegal local variable table length 17 in method test.nz.ac.massey.cs.sdc.log4jassignment.s06005586.AppenderLayoutTest.<init>()V

Endocarp

Answer 1

Can you try replacing

<test 
name="test.nz.ac.massey.cs.sdc.log4jassignment.s06005586.AppenderLayoutTest"
todir="${coverage.dir}">

with

<batchtest>
    <fileset dir="${instr.dir}" includes="**/*Test.class" />
</batchtest>

Lastly I would ask you to put some sysout in your test and see what is going wrong in the test. This might something to do with configuration I feel.

Extravasate

Thanks I just tried, I updated my question. But the tests are still failing, I feel I'm a little closer to the solution now though. Any other ideas would be greatly appreciated ! Regards – Endocarp

Lastly I would ask you to put some sysout in your test and see what is going wrong in the test. This might something to do with configuration I feel. – Extravasate

I haven't used ant much, and I read from this website to add the attribute showoutput="true" to my <junit> task but no sysout is being printed. I'm using the default configuration from emma.jar and emma_ant.jar. – Endocarp

This means your test Class is not being executed... Check the path ${instr.dir} are the instrumented classes present? – Extravasate

you have two folders inside emma-instr can you try deleting everything and test again. – Extravasate

I did and the same thing happens it creates the package path to the classes in the folder emma-instr (I copy/pasted the code from Add EMMA instrumentation to your test target on UsingEMMAWithANTForJUnitTestCoverageReporting) – Endocarp

let us continue this discussion in chat – Extravasate

Answer 2

Looking at the latest trace you shared it seems can use

-XX:-UseSplitVerifier

as VM argument to get rid of the error related to JDK7.

or Check this out

Extravasate