Spring batch item writer rest API

Question

Is it possible to read data from DB, process it and in ItemWriter send to another system using RestAPI (REST TEMPLATE) in Spring batch project? All I can see is fetch data and write it in a csv file.

Ames

Web service response on http call only, in your http call you can call other api or other http web url. But if you are talking about reading from database and returning to web ui or then the process is simple, let's know more clear question – Joseph

@Joseph , So basically i am working on a project where I have to fetch data from DB and send it to another downstream system, not a web UI. System A triggers a batch process to fetch from DB and send it to System B, where it will process data. – Ames

You can create a custom writer that sends the REST request, as shown in the answer by "Asif A Fasih" (except doing a POST or PUT operation instead of GET as this is what typically an item writer does). – Carmoncarmona

Thanks @MahmoudBenHassine, I asked a follow up question with Asif and let me as this to you too. " I have to send payload in loop, waiting for response for 1st payload will increase time and block us from sending 2nd payload till 1st payload's response is not received, Any way I can send List of payloads atomically and concurrently?? " – Ames

Answer 1

It is possible to create your own custom ItemWriter.

In your case, please add the spring-boot-starter-web dependency to either your pom.xml or build.gradle

Example:

package com.example.batch;

import lombok.extern.log4j.Log4j2;
import org.springframework.batch.item.ItemWriter;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.ResponseEntity;
import org.springframework.web.client.RestTemplate;

import java.util.List;

@Log4j2
public class RestItemWriter implements ItemWriter<String> {
    @Autowired
    RestTemplate restTemplate;

    public RestItemWriter(RestTemplate restTemplate) {
        this.restTemplate = restTemplate;
    }

    @Override
    public void write(List<? extends String> items) throws Exception {
        ResponseEntity<Users> users = restTemplate.getForEntity("https://jsonplaceholder.typicode.com/users/1", Users.class);

        log.info("Status code is: " + users.getStatusCode());
    }
}


package com.example.batch;

import com.fasterxml.jackson.annotation.JsonInclude;
import lombok.Getter;
import lombok.Setter;

@Getter
@Setter
@JsonInclude(JsonInclude.Include.NON_NULL)
public class Users {

    public String id;
    public String name;
    public String username;
    public String email;
    public String phone;
}

More information about custom item writers here

Langelo

Thanks @Asif, I have to send payload in loop, waiting for response for 1st payload will increase time and block us from sending 2nd payload till 1st payload's response is not received, Any way I can send List of payloads atomically and concurrently?? – Ames

have you looked into using WebFlux? docs.spring.io/spring-framework/docs/current/reference/html/… – Langelo

yes I looked into it, but I want to know if this could be done using CompleteableFuture? I have seen lot of examples of CF being used for DB call, but not for rest api call. Is it possible? – Ames

yes, it is possible. I found this blog which has some good examples - twilio.com/blog/… – Langelo

Answer 2

Of course! you can send the processed records to another system using REST call in ItemWriter.

Use the below code in your RestItemWriter class.

private RestTemplate restTemplate;

@Override
public void write(@NonNull Chunk<? extends Payload> chunk) throws Exception {

    for((Payload payload : chunk){

        HttpHeaders headers = new HttpHeaders();
        headers.setContentType(MediaType.APPLICATION_JSON);
        headers.set("Authorization", "xx-tokenString-xx");
                    
        HttpEntity<Payload> requestEntity = new HttpEntity<>(payload, headers);

        try{
            ResponseEntity<Object> response = restTemplate.exchange(url, HttpMethod.PATCH, requestEntity, Object.class);
            LOGGER.info("Request hits the server {}", response.getBody());
        } catch(HttpClientErrorException e){
            LOGGER.error("HttpClientErrorException occured during connection {}", e.getMessage());
        } catch (Exception e) {
            LOGGER.error("Exception occured during connection {}", e.getMessage());
        }

    }
}

To make HTTP Patch request using RestTemplate, below configurations are mandatory, for other HTTP calls you may ignore it.

 @Bean
    public RestTemplate restTemplate() {
        LOGGER.info("restTemplate Bean has bean created");
        CloseableHttpClient httpClient = HttpClients.createDefault();
        HttpComponentsClientHttpRequestFactory factory = new HttpComponentsClientHttpRequestFactory(httpClient);
        RestTemplate restTemplate = new RestTemplate(factory);
        return restTemplate;
    }

You have to add the below dependency in pom.xml to use CloseableHttpClient, and HttpClients.

<dependency>
    <groupId>org.apache.httpcomponents.client5</groupId>
    <artifactId>httpclient5</artifactId>
</dependency>

Quimper