clover: how does it work?

Question

I am evaluating clover currently and wonder how to use it best. First I'd like to understand how it works conceptually.

1) What does instrumentation mean? Are the test-calls attached to implementation's statements?

2) How is this done? Are the tests actually executed with some fancy execution context (similar to JRebel e.g.) for this? Or is it more like static analysis ?

3) After a "clover-run", some DB is saved to disk, and based on this, reports are generated right? Is the DB-Format accessible? I mean Can I launch my own analysis on it, e.g. using my own reporting tools ? What information does the DB contain exactly? Can I see the mapping between test and implementation there ?

4) Are there other tools that find the mapping between test and implementation? Not just the numbers, but which test, actually covers a line of code ...

Thanks, Bastl.

Nidia

Can you break this down into a number of questions? You may find answers to a number of these on SO already. – Crusted

Answer 1

How is this done? Are the tests actually executed with some fancy execution context (similar to JRebel e.g.) for this? Or is it more like static analysis?

During code instrumentation by Clover it detects which methods are test methods (by default it recognizes JUnit3/4 and TestNG). Such methods gets additional instrumentation instructions. Shortly speaking, entering a test method will usually instantiate a dedicated coverage recorder which measures coverage exclusively for this test. More information about per test recording strategies available in Clover:

• https://confluence.atlassian.com/display/CLOVER/Clover+Performance+Results
• https://confluence.atlassian.com/display/CLOVER/About+Distributed+Per-Test+Coverage

After a "clover-run", some DB is saved to disk, and based on this, reports are generated right?

A Clover database (clover.db) contains information about code structure (packages, files, classes, methods, statements, branches), it has also information about test methods. There are also separate coverage recording files (produced at runtime) containing information about number of "hits" of given code element. Clover supports both global coverage (i.e. for the whole run) as well as per-test coverage (i.e. coverage from a single test).

More information is here:

• https://confluence.atlassian.com/display/CLOVER/Managing+the+Coverage+Database

Is the DB-Format accessible?

The API is still in development (https://jira.atlassian.com/browse/CLOV-1375), but there is a possibility to get basic information. See:

• https://confluence.atlassian.com/display/CLOVER/Database+Structure

for more details about DB model and code samples.

But the question is: do you really need to manually read this DB? You wrote that:

Can I see the mapping between test and implementation there ?

Such mapping is already provided by Clover - in the HTML report for example if you click on a source line it will pop up a list of test methods hitting this line.

PS: I'm a Clover developer at Atlassian, feel free to contact me if you have any questions.

Bathymetry

Thanks Marek, I accepted your answer, but sadly the topic is outdated for me. – Nidia

Answer 2

What does instrumentation mean?

Additional code is woven in with your code.

Are the test-calls attached to implementation's statements?

I am not sure what you mean but it could be instructions or call to methods. Trivial methods will be inlined by the JIT at runtime.

How is this done?

There are many ways to do it, but often the Instrumentation class is to used to capture when a class is being loaded and a library like Objectweb's ASM is used to manipulate the code.

Are the tests actually executed with some fancy execution context

The context counts which lines have been executed.

Or is it more like static analysis ?

No, it is based on what is called.

After a "clover-run", some DB is saved to disk, and based on this, reports are generated right? Is the DB-Format accessible?

You had best ask the producers of clover as to the content of their files.

Are there other tools that find the mapping between test and implementation? Not just the numbers, but which test, actually covers a line of code ...

There are many code coverage tools available including EMMA, JaCoCo, Cobertura, IDEA has one builtin.

Selfreliance

about the last: I specifically ask about the mapping between test-implementation and actual code. Does any of the tools provide that? E.g. I can see that one line (of 1,5 mio LOCs) is covered, but by which test !? – Nidia

If you run a given test it will show you want it covers. If you use a debugger you can see what tests run a particular line. Its not clear to me why you want to know collectively which tests run which lines as this doesn't appear to be focused on a specific problem. – Selfreliance

The Clover HTML reports will tell you each of the tests that hit a line of code. See screen shot #3 - atlassian.com/software/clover/overview/screenshot-tour – Giuditta

@PeterLawrey: I'll give you a hypothetical problem where you need to know which LOC were hit by a given test: if you want to know in a programmatic way which tests have considerable overlap. I'm working on a tool to detect unit tests with considerable overlap for the purposes of automatically finding tests which test the same thing, and part of this requires knowing the specific LOC that a test hits. – Shellback

@AdamParkin: That sounds interesting! Can you already share something? – Nidia

I can share the paper that inspired me: ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6063019 and hindawi.com/journals/ase/2010/932686 – Shellback