Spring Batch - Counting Processed Rows

Question

So I am creating a Spring Batch job for reading a CSV file and for certain rows which contain incomplete data; it checks, outputs to the log that the row is incomplete, and skips. It works great except at the end of the job I want it to log how many rows it found that were incomplete. Just something simple like "X incomplete rows were found".

I've Googled and searched around for a solution but not found anything really.

Any help is appreciated and any more info needed just ask.

Shoplifter

Not much we can tell you about how to change your script if we can't see it. – Costly

What do you want to see? I did say if any more info needed just ask. – Shoplifter

It's the usual Spring Batch stuff... an ItemProcessor is where it checks for incomplete data. It is Spring Batch, not a batch script. I have a feeling you may have misunderstood the question? docs.spring.io/spring-batch – Shoplifter

At the end of the step you can use the StepExecution to retrieve the different skip counts (read, write, processing). So you basically could write a StepExecutionListener which records this. – Lockridge

Cheers I'll do some reading on StepExecutionListener :) Do you have any advice on how/where to store the count from the ItemProcessor so that I can access it from the StepExecutionListener? – Shoplifter

Spring Batch already does that for you... it keeps track of reads, writes, processing... So if you do it correctly, Spring Batch will provide you with those numbers... – Lockridge

Answer 1

Manage to solve this, here's how I did it:

In the ItemProcessor I added an attribute and a method for getting access to the ExecutionContext from within the process method,

private ExecutionContext executionContext;

@BeforeStep
public void beforeStep(StepExecution stepExecution)
{
    this.executionContext = stepExecution.getExecutionContext();
}

...and then in the process() method when I find one of the rows I want to log, I can do this,

this.executionContext.putInt( "i_ThoseRows", this.executionContext.getInt( "i_ThoseRows", 0 ) + 1 );

Finally I add another method to the ItemProcessor to print the result at the end of the step,

@AfterStep
public void afterStep(StepExecution stepExecution)
{
    System.out.println( "Number of 'Those rows': " + this.executionContext.getInt( "i_ThoseRows", 0 ) );
}

Hope it helps someone

Shoplifter

Answer 2

Spring Batch itself keeps track of how many records it reads, writes, processes and how many it skips (for each of those numbers). That information is stored in the StepExecution. The StepExecution can be accessed from a StepExecutionListener. In this case an implementation of the afterStep method will suffice.

public class SkippedItemStepExecutionListener extends StepExecutionListenerSupport {

    @Override
    public ExitStatus afterStep(StepExecution stepExecution) {
        int skipped = stepExecution.getSkipCount(); // Total for read+write+process
        // Log it to somewhere.        
        return null;
    }
}

How to add it to your job/step is explained in the reference guide

Links

1. StepExecution javadoc
2. StepExecutionListener javadoc
3. Listener Configuration Reference

Lockridge

Thanks for the reply. I have set up a StepExecutionListener and it can log the number of skipped rows, etc. The thing is that it is just for certain conditions I want it to add to the count, not any skipped row. Just when a certain condition that I check for in the process() method where if it is 'one of those rows' it increments a count and then write it out at the end using the StepExecutionListener. – Shoplifter

If that is the only thing you do in the process method you can use the getProcessSkipCount(), whereas the getSkipCount() method returns thet total for all skipped records. If you really need something more fine grained you probably are going to need a SkipListener instead of add a property yourself to the ExecutionContext to keep the state. – Lockridge

Managed to get this solved, check my answer. Thanks for the help :) – Shoplifter

Answer 3

To complement @dogfight answer:

from spring batch docs:

The annotations are analysed by the XML parser for the elements, so all you need to do is use the XML namespace to register the listeners with a step

So to call the listener callback annotated functions beforeStep() and afterStep() you need to register you ItemProcessor as listener in the step:

<listeners>
    <listener ref="MyItemProcessor">
</listeners>

Otherwise you will have a NullPointerException when use the executionContext.

Bhatt