9.Spring Batch의 Chunk와 ItemWriter
Spring Batch

9.Spring Batch의 Chunk와 ItemWriter

반응형

Flat Files - FlatFileItemWriter 개념 및 API 소개

기본개념

  • 2차원 데이터(표)로 표현된 유형의 파일을 처리하는 ItemWriter
  • 고정 위치로 정의된 데이터 필드나 특수 문자에 의해 구별된 데이터의 행을 기록한다
  • Resource 와 LineAggregator 두 가지가 요소가 필요하다

구조

ㅋ

LineAggregator

  • Item 을 받아서 String 으로 변환하여 리턴한다
    • item은 객체를 의미한다.
  • FieldExtractor를 사용해서 처리할 수 있다
  • 구현체
    • 총 3개의 PassThroughLineAggregator, DelimitedLineAggregator, FormatterLineAggregator 구현체가 있다.

FieldExtractor

  • 전달 받은 Item 객체의 필드를 배열로 만들고 배열을 합쳐서 문자열을 만들도록 구현하도록 제공하는 인터페이스
  • 구현체
    • 총 2개의 BeanWrapperFieldExtractor, PassThroughFieldExtractor 구현체가 있다.

LineAggregator의 구조

  • 먼저 LineAggreagator에서 객체를 받는다.
  • 다음으로 PassThrougLineAggreator에서 전달된 아이템을 문자열로 반환한다.
  • 이후 DelimitedLineAggreagator에서 전달된 배열을 구분자로 구분하여 문자열로 합치거나 FormatterLineAggreagator에서 전달된 배열을 고정길이로 구분하여 문자열로 합친다.
    • 이는 ItemReader와 완벽히 반대되는 성질임을 알 수 있다.
  • 마지막으로 FiledExtrator에서 객체를 인자로 받고 배열로 반환하는 작업을 걸친다.
  • 아래의 그림은 위의 LineAggregator의 전체적인 흐름을 보여주는 예제이다.

  • FlatFileItemWriter에서 객체를 LineAggregator에 전달하고 DelimitedLineAggregator 또는 FormatterLineAggregator에 전달하여 문자열을 합칠 방식을 결정한다.
  • 이후 FieldExtractor를 통해 객체를 반환한다. 반환할 타입을 결정하는것은 BeanWrapperFieldExtractor(전달된 객체의 필드 → 배열로 반환) 또는 PassThroughFieldExtractor(전달된 Collection → 배열로 반환)로 결정한다.

API 소개

FlatFileItemWriter Delimeted(구분자 방식) 예제

기본개념

  • 객체의 필드 사이에 구분자를 삽입해서 한 문자열로 변환한다
    • 예를 들어 [1,2,3,4,5] 라는 배열이 있다
    • 위 배열을 “%”라는 구분자를 사용해서 [1%2%3%4%5]로 표현한다는 의미이다.

구조

DelimitedLineAggregator 핵심 메소드 분석

  • 좌측 상단 첫 번째줄 코드를 보면 ExtractorLIneAggregator가 LineAggergator를 구현하는것을 볼수 있다.
  • 좌측 하단 첫 번째줄 코드를 보면 DelimitedLineAggrator가 ExtractorLineAggreator를 상속받는 것을 볼 수 있다.
  • 우측 상단은 배치 코드가 아니라 배치의 로직을 확인하기 위해 정의한 예제이다. 우측 상단의 resource밑에 코드를 보면 delimited()를 명시함으로써 구분자 방식을 선택한것을 알 수 있고 delimiter(”|”)를 통해 배열 사이에 “|”를 추가하려는 것을 알 수 있다.
  • 우측 상단에 정의된 names에 매개변수로 정의한 배열을 BeanWrapperFileExtrator의 names로 전달이 된다(우측 하단 사진 확인). 이후 extract를 통해서 배열로 반환되는 로직을 확인할 수 있다.
  • 우측 하단에서 values.toArray()를 통해 반환된 배열은 좌측 하단의 doAggregate로 전달이 되어 하나의 문자열로 완성된다.

예제코드

package com.example.springbatch_9_1_flatfileitemwriter_delimetedandformat;

import lombok.RequiredArgsConstructor;
import org.springframework.batch.core.Job;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.launch.support.RunIdIncrementer;
import org.springframework.batch.item.ItemReader;
import org.springframework.batch.item.ItemWriter;
import org.springframework.batch.item.file.builder.FlatFileItemWriterBuilder;
import org.springframework.batch.item.support.ListItemReader;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.io.FileSystemResource;

import java.util.Arrays;
import java.util.List;

/**
 * packageName    : com.example.springbatch_9_1_flatfileitemwriter_delimetedandformat
 * fileName       : FlatFilesDelimitedConfiguration
 * author         : namhyeop
 * date           : 2022/08/14
 * description    :
 * FlatFilesDelimited 예제
 * ===========================================================
 * DATE              AUTHOR             NOTE
 * -----------------------------------------------------------
 * 2022/08/14        namhyeop       최초 생성
 */
@RequiredArgsConstructor
@Configuration
public class FlatFilesDelimitedConfiguration {

    private final JobBuilderFactory jobBuilderFactory;
    private final StepBuilderFactory stepBuilderFactory;

    @Bean
    public Job job() {
        return jobBuilderFactory.get("batchJob")
                .incrementer(new RunIdIncrementer())
                .start(step1())
                .build();
    }

    @Bean
    public Step step1() {
        return stepBuilderFactory.get("step1")
                .<Customer, Customer>chunk(10)
                .reader(customItemReader())
                .writer(customItemWriter())
                .build();
    }

    @Bean
    public ItemWriter<? super Customer> customItemWriter() {
        return new FlatFileItemWriterBuilder<>()
                .name("flatFileWriter")
                //합친 문자열을 저장할 위치
                .resource(new FileSystemResource("/Users/namhyeop/Desktop/git자료/Spring_Boot_Study/8.Spring_Batch/SpringBatch_9_1_FlatFileItemWriter_DelimetedAndFormat/src/main/resources/customer.txt"))
                //(1).파일에 지속적으로 내용을 덮어 쓰는 옵션
                .append(true)
                //(2).Reader에서 읽은 내용이 아무것도 없을 경우 삭제하는 옵션
//                .shouldDeleteIfEmpty(true)
                .delimited()
                .delimiter("|")
                //객체의 필드 자료 입력
                .names(new String[]{"id", "name", "age"})
                .build();
    }

    @Bean
    public ItemReader<? extends Customer> customItemReader() {
        List<Customer> customers = Arrays.asList(
                new Customer(1, "Kin Nam Hyeop", 27),
                new Customer(2, "Kim ji Hwan", 27),
                new Customer(3, "Hwan ji kim", 28));

        ListItemReader<Customer> reader = new ListItemReader<>(customers);
        //reader가 아무것도 없는 경우 파일을 삭제하는 예제, CustomeItemWriter에서 (2)번 주석을 해제하자
//        ListItemReader<Customer> reader = new ListItemReader<>(Collections.emptyList());
        return reader;
    }
}
package com.example.springbatch_9_1_flatfileitemwriter_delimetedandformat;

import lombok.AllArgsConstructor;
import lombok.Data;

/**
 * packageName    : com.example.springbatch_9_1_flatfileitemwriter_delimetedandformat
 * fileName       : Customer
 * author         : namhyeop
 * date           : 2022/08/14
 * description    :
 * Customer 객체
 * ===========================================================
 * DATE              AUTHOR             NOTE
 * -----------------------------------------------------------
 * 2022/08/14        namhyeop       최초 생성
 */
@Data
@AllArgsConstructor
public class Customer {
    private long id;
    private String name;
    private int age;
}
  • 실행 후 resources에 파일이 정상적으로 생성되었는지 확인하자

FlatFileItemWriter - format(고정길이 방식) 예제

기본개념

  • 객체의 필드를 사용자가 설정한 Formatter 구문을 통해 문자열로 변환한다

구조

  • DelimitedLineAggregator와 방식은 동일하나 차이점이 있다면 구분자로 나누는게 아니라 포맷 형식으로 나눈다는 점에 차이가 있다.
  • 우측 상단을 보면 format을 통해 나누는것을 확인할 수 있다.
  • 이후 로직은 DelimitedLineAggregator와 방식이 동일하다

예제코드

package com.example.springbatch_9_2_flatfileitemwriter_format;

import lombok.RequiredArgsConstructor;
import org.springframework.batch.core.Job;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.launch.support.RunIdIncrementer;
import org.springframework.batch.item.ItemWriter;
import org.springframework.batch.item.file.builder.FlatFileItemWriterBuilder;
import org.springframework.batch.item.support.ListItemReader;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.io.FileSystemResource;

import java.util.Arrays;
import java.util.List;

/**
 * packageName    : com.example.springbatch_9_2_flatfileitemwriter_format
 * fileName       : FlatFilesFormattedConfiguration
 * author         : namhyeop
 * date           : 2022/08/14
 * description    :
 * format 방식 예제
 * ===========================================================
 * DATE              AUTHOR             NOTE
 * -----------------------------------------------------------
 * 2022/08/14        namhyeop       최초 생성
 */
@RequiredArgsConstructor
@Configuration
public class FlatFilesFormattedConfiguration {

    private final JobBuilderFactory jobBuilderFactory;
    private final StepBuilderFactory stepBuilderFactory;

    @Bean
    public Job job(){
        return jobBuilderFactory.get("batchJob")
                .incrementer(new RunIdIncrementer())
                .start(step1())
                .build();
    }

    @Bean
    public Step step1() {
        return stepBuilderFactory.get("step1")
                .<Customer, Customer>chunk(10)
                .reader(customItemReader())
                .writer(customItemWriter())
                .build();
    }

    @Bean
    public ItemWriter customItemWriter() {
        return new FlatFileItemWriterBuilder<Customer>()
                .name("flatFileWriter")
                .resource(new FileSystemResource("/Users/namhyeop/Desktop/git자료/Spring_Boot_Study/8.Spring_Batch/springbatch_9_2_flatfileitemwriter_format/src/main/resources/customer.txt"))
                //덮어쓰기 옵션
                //.append(true)
                .formatted()
                //format 지정시 맨앞에 '%'를 붙쳐주는게 문법이다. 맨 뒤에는 %를 안붙친다
                .format("%-2d%-15s%-2d")
                .names(new String[]{"id","name","age"})
                .build();
    }

    @Bean
    public ListItemReader customItemReader(){
        List<Customer> customers = Arrays.asList(new Customer(1, "Kim Nam Hyeop", 27),
                new Customer(2, "Kim ji Hwan", 27),
                new Customer(3, "Ji KIm Hwan", 28));

        ListItemReader<Customer> reader = new ListItemReader<>(customers);
        return reader;
    }
}
package com.example.springbatch_9_2_flatfileitemwriter_format;

import lombok.AllArgsConstructor;
import lombok.Data;

/**
 * packageName    : com.example.springbatch_9_2_flatfileitemwriter_format
 * fileName       : Customer
 * author         : namhyeop
 * date           : 2022/08/14
 * description    :
 * Customer 객체
 * ===========================================================
 * DATE              AUTHOR             NOTE
 * -----------------------------------------------------------
 * 2022/08/14        namhyeop       최초 생성
 */
@Data
@AllArgsConstructor
public class Customer {
    private long id;
    private String name;
    private int age;
}
  • 결과 값이 "%-2d%-15s%-2d" 형식으로 나오는지 확인하자

XML StaxEventItemWriter

기본개념

  • XML 쓰는 과정은 읽기 과정에 대칭적이다.
  • StaxEventItemWriter는 Resource, marshaller, rootTagName가 필요하다.

💡 marshaller는 객체 → XML, unmarshaller는 XML → 객체

API

StaxEventItemWriter의 흐름

  • XStreamMarshaller에는 marshall 기능과 unMarshall 기능 둘 다 가지고 있다.

예제코드

dependency 추가 필수

<dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-oxm</artifactId>
        </dependency>
        <dependency>
            <groupId>com.thoughtworks.xstream</groupId>
            <artifactId>xstream</artifactId>
            <version>1.4.16</version>
        </dependency>
package com.example.springbatch_9_3_xmlstaxeventitemwriter;

import lombok.RequiredArgsConstructor;
import org.springframework.batch.core.Job;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.launch.support.RunIdIncrementer;
import org.springframework.batch.item.ItemWriter;
import org.springframework.batch.item.database.JdbcPagingItemReader;
import org.springframework.batch.item.database.Order;
import org.springframework.batch.item.database.support.MySqlPagingQueryProvider;
import org.springframework.batch.item.xml.StaxEventItemWriter;
import org.springframework.batch.item.xml.builder.StaxEventItemWriterBuilder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.io.FileSystemResource;
import org.springframework.oxm.Marshaller;
import org.springframework.oxm.xstream.XStreamMarshaller;

import javax.sql.DataSource;
import java.util.Date;
import java.util.HashMap;
import java.util.Map;

/**
 * packageName    : PACKAGE_NAME
 * fileName       : com.example.springbatch_9_3_xmlstaxeventitemwriter.XMLConfiguration
 * author         : namhyeop
 * date           : 2022/08/14
 * description    :
 * DB에서 조회한 객체를 XML 파일로 반환하는 예제
 * ===========================================================
 * DATE              AUTHOR             NOTE
 * -----------------------------------------------------------
 * 2022/08/14        namhyeop       최초 생성
 */

@RequiredArgsConstructor
@Configuration
public class XMLConfiguration {

    private final JobBuilderFactory jobBuilderFactory;
    private final StepBuilderFactory stepBuilderFactory;
    private final DataSource dataSource;

    @Bean
    public Job job() {
        return jobBuilderFactory.get("batchJob")
                .incrementer(new RunIdIncrementer())
                .start(step1())
                .build();
    }

    @Bean
    public Step step1() {
        return stepBuilderFactory.get("step1")
                .<Customer, Customer>chunk(10)
                .reader(customItemReader())
                .writer(customItemWriter())
                .build();
    }

    @Bean
    public JdbcPagingItemReader<Customer> customItemReader() {
        JdbcPagingItemReader<Customer> reader = new JdbcPagingItemReader<>();

        reader.setDataSource(this.dataSource);
        reader.setFetchSize(10);
        reader.setRowMapper(new CustomerRowMapper());

        MySqlPagingQueryProvider queryProvider = new MySqlPagingQueryProvider();
        queryProvider.setSelectClause("id, firstName, lastName, birthdate");
        queryProvider.setFromClause("from customer");
        queryProvider.setWhereClause("where firstname like :firstname");

        Map<String, Order> sortKeys = new HashMap<>(1);
        sortKeys.put("id", Order.ASCENDING);
        queryProvider.setSortKeys(sortKeys);
        reader.setQueryProvider(queryProvider);

        HashMap<String, Object> parameters = new HashMap<>();
        parameters.put("firstname", "A%");

        reader.setParameterValues(parameters);
        return reader;
    }

    @Bean
    public StaxEventItemWriter customItemWriter() {
        return new StaxEventItemWriterBuilder<Customer>()
                .name("customersWriter")
                .marshaller(itemMarshaller())
                .resource(new FileSystemResource("/Users/namhyeop/Desktop/git자료/Spring_Boot_Study/8.Spring_Batch/springbatch_9_3_XMLStaxEventItemWriter/src/main/resources/customer.xml"))
                .rootTagName("customer")
                .overwriteOutput(true)
                .build();
    }

    @Bean
    public XStreamMarshaller itemMarshaller(){
        HashMap<String, Class<?>> aliases = new HashMap<>();
        aliases.put("customer", Customer.class);
        aliases.put("id", Long.class);
        aliases.put("firstName", String.class);
        aliases.put("lastName", String.class);
        aliases.put("birthdate", Date.class);
        XStreamMarshaller xStreamMarshaller = new XStreamMarshaller();
        xStreamMarshaller.setAliases(aliases);
        return xStreamMarshaller;
    }
}

 

package com.example.springbatch_9_3_xmlstaxeventitemwriter;

import org.springframework.jdbc.core.RowMapper;

import java.sql.ResultSet;
import java.sql.SQLException;

/**
 * packageName    : com.example.springbatch_9_3_xmlstaxeventitemwriter
 * fileName       : CustomerRowMapper
 * author         : namhyeop
 * date           : 2022/08/14
 * description
 * DB의 정보를 객체로 매핑 시켜주는 RowMapper
 * ===========================================================
 * DATE              AUTHOR             NOTE
 * -----------------------------------------------------------
 * 2022/08/14        namhyeop       최초 생성
 */
public class CustomerRowMapper implements RowMapper<Customer> {

    @Override
    public Customer mapRow(ResultSet rs, int rowNum) throws SQLException {
        return new Customer(rs.getLong("id"),
                rs.getString("firstName"),
                rs.getString("lastName"),
                rs.getDate("birthdate"));
    }
}
package com.example.springbatch_9_3_xmlstaxeventitemwriter;

import lombok.AllArgsConstructor;
import lombok.Data;

import java.util.Date;

/**
 * packageName    : com.example.springbatch_9_3_xmlstaxeventitemwriter
 * fileName       : Customer
 * author         : namhyeop
 * date           : 2022/08/14
 * description    :
 * Customer 객체
 * ===========================================================
 * DATE              AUTHOR             NOTE
 * -----------------------------------------------------------
 * 2022/08/14        namhyeop       최초 생성
 */
@Data
@AllArgsConstructor
public class Customer {
    private final Long id;
    private final String firstName;
    private final String lastName;
    private final Date birthdate;
}
  • JDBCPaging을 사용해 DB에서 객체를 먼저 조회한다
  • 이후 조회한 객체를 Marshller를 통해 XML로 만든다.

DB 설정

spring:
  profiles:
    active: mysql
##  batch:
##    job:
##      enabled: false
##      names: ${job.name:NONE}
---
spring:
  config:
    activate:
      on-profile: local
  datasource:
    hikari:
      jdbc-url: jdbc:h2:mem:testdb;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE
      username: sa
      password:
      driver-class-name: org.h2.Driver
  batch:
    jdbc:
      initialize-schema: embedded
---
spring:
  config:
    activate:
      on-profile: mysql
  datasource:
    hikari:
      jdbc-url: jdbc:mysql://localhost:3307/springbatch?useUnicode=true&characterEncoding=utf8
      username: root
      password: 1234
      driver-class-name: com.mysql.jdbc.Driver
  batch:
    jdbc:
      initialize-schema: always

DB Schema

drop table customer;
CREATE TABLE customer(
                         id mediumint(8) unsigned NOT NULL auto_increment,
                         firstName varchar(255) default null,
                         lastName varchar(255) default null,
                         birthdate varchar(255),
                         PRIMARY KEY (id)
) AUTO_INCREMENT=1;

select id, firstName, lastName, birthdate from customer where firstName like ? order by lastName, firstName
select * from customer where firstName like ? order by lastName, firstName

data 파일

INSERT INTO `customer` (`id`, `firstName`, `lastName`, `birthdate`)
VALUES (1, "Reed", "Edwards", "1952-08-16 12:34:53"),
       (2, "Hoyt", "Park", "1981-02-18 08:07:58"),
       (3, "Leila", "Petty", "1972-06-11 08:43:55"),
       (4, "Denton", "Strong", "1989-03-11 18:38:31"),
       (5, "Zoe", "Romero", "1990-10-02 13:06:31"),
       (6, "Rana", "Compton", "1957-06-09 12:51:11"),
       (7, "Velma", "King", "1988-02-02 05:52:25"),
       (8, "Uriah", "Carter", "1972-08-31 07:32:05"),
       (9, "Michael", "Graves", "1958-04-13 18:47:44"),
       (10, "Leigh", "Stone", "1967-06-23 23:41:43");
INSERT INTO `customer` (`id`, `firstName`, `lastName`, `birthdate`)
VALUES (11, "Iliana", "Glenn", "1965-02-27 14:33:56"),
       (12, "Harrison", "Haley", "1956-06-28 03:15:41"),
       (13, "Leonard", "Zamora", "1956-03-28 15:03:09"),
       (14, "Hiroko", "Wyatt", "1960-08-22 23:53:50"),
       (15, "Cameron", "Carlson", "1969-05-12 11:10:09"),
       (16, "Hunter", "Avery", "1953-11-19 12:52:42"),
       (17, "Aimee", "Cox", "1976-10-15 12:56:50"),
       (18, "Yen", "Delgado", "1990-02-06 10:25:36"),
       (19, "Gemma", "Peterson", "1989-04-02 23:42:09"),
       (20, "Lani", "Faulkner", "1970-09-18 17:22:14");
INSERT INTO `customer` (`id`, `firstName`, `lastName`, `birthdate`)
VALUES (21, "Iola", "Cannon", "1954-01-12 16:56:45"),
       (22, "Whitney", "Shaffer", "1951-03-19 01:27:18"),
       (23, "Jerome", "Moran", "1968-03-16 05:26:22"),
       (24, "Quinn", "Wheeler", "1979-06-19 16:24:22"),
       (25, "Mira", "Wilder", "1961-12-27 12:11:07"),
       (26, "Tobias", "Holloway", "1968-08-13 20:36:19"),
       (27, "Shaine", "Schneider", "1958-03-08 09:47:10"),
       (28, "Harding", "Gonzales", "1952-04-11 02:06:25"),
       (29, "Calista", "Nieves", "1970-02-17 13:29:59"),
       (30, "Duncan", "Norman", "1987-09-13 00:54:49");
INSERT INTO `customer` (`id`, `firstName`, `lastName`, `birthdate`)
VALUES (31, "Fatima", "Hamilton", "1961-06-16 14:29:11"),
       (32, "Ali", "Browning", "1979-03-27 17:09:37"),
       (33, "Erin", "Sosa", "1990-08-23 10:43:58"),
       (34, "Carol", "Harmon", "1972-01-14 07:19:39"),
       (35, "Illiana", "Fitzgerald", "1970-08-19 02:33:46"),
       (36, "Stephen", "Riley", "1954-06-05 08:34:03"),
       (37, "Hermione", "Waller", "1969-09-08 01:19:07"),
       (38, "Desiree", "Flowers", "1952-06-25 13:34:45"),
       (39, "Karyn", "Blackburn", "1977-03-30 13:08:02"),
       (40, "Briar", "Carroll", "1985-03-26 01:03:34");
INSERT INTO `customer` (`id`, `firstName`, `lastName`, `birthdate`)
VALUES (41, "Chaney", "Green", "1987-04-20 18:56:53"),
       (42, "Robert", "Higgins", "1985-09-26 11:25:10"),
       (43, "Lillith", "House", "1982-12-06 02:24:23"),
       (44, "Astra", "Winters", "1952-03-13 01:13:07"),
       (45, "Cherokee", "Stephenson", "1955-10-23 16:57:33"),
       (46, "Yuri", "Shaw", "1958-07-14 15:10:07"),
       (47, "Boris", "Sparks", "1982-01-01 10:56:34"),
       (48, "Wilma", "Blake", "1963-06-07 16:32:33"),
       (49, "Brynne", "Morse", "1964-09-21 01:05:25"),
       (50, "Ila", "Conley", "1953-11-02 05:12:57");
INSERT INTO `customer` (`id`, `firstName`, `lastName`, `birthdate`)
VALUES (51, "Sharon", "Watts", "1964-01-09 16:32:37"),
       (52, "Kareem", "Vaughan", "1952-04-18 15:37:10"),
       (53, "Eden", "Barnes", "1954-07-04 01:26:44"),
       (54, "Kenyon", "Fulton", "1975-08-23 22:17:52"),
       (55, "Mona", "Ball", "1972-02-11 04:15:45"),
       (56, "Moses", "Cortez", "1979-04-24 15:26:46"),
       (57, "Macy", "Banks", "1956-12-31 00:41:15"),
       (58, "Brenna", "Mendez", "1972-10-02 07:58:27"),
       (59, "Emerald", "Ewing", "1985-11-28 21:15:20"),
       (60, "Lev", "Mcfarland", "1951-05-20 14:30:07");
INSERT INTO `customer` (`id`, `firstName`, `lastName`, `birthdate`)
VALUES (61, "Norman", "Tanner", "1959-07-29 15:41:45"),
       (62, "Alexa", "Walters", "1977-12-06 16:41:17"),
       (63, "Dara", "Hyde", "1989-08-04 14:06:43"),
       (64, "Hu", "Sampson", "1978-11-01 17:10:23"),
       (65, "Jasmine", "Cardenas", "1969-02-15 20:08:06"),
       (66, "Julian", "Bentley", "1954-07-11 03:27:51"),
       (67, "Samson", "Brown", "1967-10-15 07:03:59"),
       (68, "Gisela", "Hogan", "1985-01-19 03:16:20"),
       (69, "Jeanette", "Cummings", "1986-09-07 18:25:52"),
       (70, "Galena", "Perkins", "1984-01-13 02:15:31");
INSERT INTO `customer` (`id`, `firstName`, `lastName`, `birthdate`)
VALUES (71, "Olga", "Mays", "1981-11-20 22:39:27"),
       (72, "Ferdinand", "Austin", "1956-08-08 09:08:02"),
       (73, "Zenia", "Anthony", "1964-08-21 05:45:16"),
       (74, "Hop", "Hampton", "1982-07-22 14:11:00"),
       (75, "Shaine", "Vang", "1970-08-13 15:58:28"),
       (76, "Ariana", "Cochran", "1959-12-04 01:18:36"),
       (77, "India", "Paul", "1963-10-10 05:24:03"),
       (78, "Karina", "Doyle", "1979-12-01 00:05:21"),
       (79, "Delilah", "Johnston", "1989-03-04 23:50:01"),
       (80, "Hilel", "Hood", "1959-08-22 06:40:48");
INSERT INTO `customer` (`id`, `firstName`, `lastName`, `birthdate`)
VALUES (81, "Kennedy", "Hoffman", "1963-10-14 20:18:35"),
       (82, "Kameko", "Bell", "1976-06-08 15:35:54"),
       (83, "Lunea", "Gutierrez", "1964-06-07 16:21:24"),
       (84, "William", "Burris", "1980-05-01 17:58:23"),
       (85, "Kiara", "Walls", "1955-12-27 18:57:15"),
       (86, "Latifah", "Alexander", "1980-06-19 10:39:50"),
       (87, "Keaton", "Ward", "1964-10-12 16:03:18"),
       (88, "Jasper", "Clements", "1970-03-05 00:29:49"),
       (89, "Claire", "Brown", "1972-02-11 00:43:58"),
       (90, "Noble", "Morgan", "1955-09-05 05:35:01");
INSERT INTO `customer` (`id`, `firstName`, `lastName`, `birthdate`)
VALUES (91, "Evangeline", "Horn", "1952-12-28 14:06:27"),
       (92, "Jonah", "Harrell", "1951-06-25 17:37:35"),
       (93, "Mira", "Espinoza", "1982-03-26 06:01:16"),
       (94, "Brennan", "Oneill", "1979-04-23 08:49:02"),
       (95, "Dacey", "Howe", "1983-02-06 19:11:00"),
       (96, "Yoko", "Pittman", "1982-09-12 02:18:52"),
       (97, "Cody", "Conway", "1971-05-26 07:09:58"),
       (98, "Jordan", "Knowles", "1981-12-30 02:20:01"),
       (99, "Pearl", "Boyer", "1957-10-19 14:26:49"),
       (100, "Keely", "Montoya", "1985-03-24 01:18:09");

JsonFileItemWriter

기본개념

  • 객체를 받아 JSON String 으로 변환하는 역할을 한다

API

예제코드

package com.example.springbatch_9_4_jsonfileitemwriter;

import lombok.RequiredArgsConstructor;
import org.springframework.batch.core.Job;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.launch.support.RunIdIncrementer;
import org.springframework.batch.item.ItemWriter;
import org.springframework.batch.item.database.JdbcPagingItemReader;
import org.springframework.batch.item.database.Order;
import org.springframework.batch.item.database.support.MySqlPagingQueryProvider;
import org.springframework.batch.item.json.JacksonJsonObjectMarshaller;
import org.springframework.batch.item.json.builder.JsonFileItemWriterBuilder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.io.FileSystemResource;

import javax.sql.DataSource;
import java.util.HashMap;
import java.util.Map;

/**
 * packageName    : PACKAGE_NAME
 * fileName       : com.example.springbatch_9_3_xmlstaxeventitemwriter.XMLConfiguration
 * author         : namhyeop
 * date           : 2022/08/14
 * description    :
 * DB에서 조회한 객체를 Json 파일로 반환하는 예제
 * ===========================================================
 * DATE              AUTHOR             NOTE
 * -----------------------------------------------------------
 * 2022/08/14        namhyeop       최초 생성
 */

@RequiredArgsConstructor
@Configuration
public class JsonConfiguration {

    private final JobBuilderFactory jobBuilderFactory;
    private final StepBuilderFactory stepBuilderFactory;
    private final DataSource dataSource;

    @Bean
    public Job job() throws Exception {
        return jobBuilderFactory.get("batchJob")
                .incrementer(new RunIdIncrementer())
                .start(step1())
                .build();
    }

    @Bean
    public Step step1() throws Exception {
        return stepBuilderFactory.get("step1")
                .<Customer, Customer>chunk(10)
                .reader(customItemReader())
                .writer(customItemWriter())
                .build();
    }

    @Bean
    public JdbcPagingItemReader<Customer> customItemReader() {

        JdbcPagingItemReader<Customer> reader = new JdbcPagingItemReader<>();

        reader.setDataSource(this.dataSource);
        reader.setFetchSize(10);
        reader.setRowMapper(new CustomerRowMapper());

        MySqlPagingQueryProvider queryProvider = new MySqlPagingQueryProvider();
        queryProvider.setSelectClause("id, firstName, lastName, birthdate");
        queryProvider.setFromClause("from customer");
        queryProvider.setWhereClause("where firstname like :firstname");

        Map<String, Order> sortKeys = new HashMap<>(1);

        sortKeys.put("id", Order.ASCENDING);
        queryProvider.setSortKeys(sortKeys);
        reader.setQueryProvider(queryProvider);

        HashMap<String, Object> parameters = new HashMap<>();
        parameters.put("firstname", "A%");

        reader.setParameterValues(parameters);

        return reader;
    }

    @Bean
    public ItemWriter<? super Customer> customItemWriter() {
        return new JsonFileItemWriterBuilder<Customer>()
                .name("jsonFileWriter")
                .jsonObjectMarshaller(new JacksonJsonObjectMarshaller<>())
                .resource(new FileSystemResource("/Users/namhyeop/Desktop/git자료/Spring_Boot_Study/8.Spring_Batch/springbatch_9_4_JsonFileItemWriter/src/main/resources/customer.json"))
                .build();
    }
}
package com.example.springbatch_9_4_jsonfileitemwriter;

import org.springframework.jdbc.core.RowMapper;

import java.sql.ResultSet;
import java.sql.SQLException;

/**
 * packageName    : com.example.springbatch_9_3_xmlstaxeventitemwriter
 * fileName       : CustomerRowMapper
 * author         : namhyeop
 * date           : 2022/08/14
 * description
 * DB의 정보를 객체로 매핑 시켜주는 RowMapper
 * ===========================================================
 * DATE              AUTHOR             NOTE
 * -----------------------------------------------------------
 * 2022/08/14        namhyeop       최초 생성
 */
public class CustomerRowMapper implements RowMapper<Customer> {

    @Override
    public Customer mapRow(ResultSet rs, int rowNum) throws SQLException {
        return new Customer(rs.getLong("id"),
                rs.getString("firstName"),
                rs.getString("lastName"),
                rs.getDate("birthdate"));
    }
}
package com.example.springbatch_9_4_jsonfileitemwriter;

import lombok.AllArgsConstructor;
import lombok.Data;

import java.util.Date;

/**
 * packageName    : com.example.springbatch_9_4_jsonfileitemwriter
 * fileName       : Cu
 * author         : namhyeop
 * date           : 2022/08/14
 * description    :
 * ===========================================================
 * DATE              AUTHOR             NOTE
 * -----------------------------------------------------------
 * 2022/08/14        namhyeop       최초 생성
 */
@Data
@AllArgsConstructor
public class Customer {
    private final Long id;
    private final String firstName;
    private final String lastName;
    private final Date birthdate;
}

DB-JdbcBatchItemWriter

기본개념

  • JdbcCursorItemReader 설정과 마찬가지로 datasource 를 지정하고, sql 속성에 실행할 쿼리를 설정
  • JDBC의 Batch 기능을 사용하여 bulk insert/update/delete 방식으로 처리
  • 확인
    • 예를 들어 1000개의 데이터를 insert한다고 할 때 1000건을 한 개씩 다로 insert 하는것은 트랜잭션에 무리를 준다.
    • 그렇기에 1000개의 데이터를 Buffer에 담아두고 최종적으로 commit할 때 데이터를 insert 하는것을 bulk한다고
  • 단건 처리가 아닌 일괄처리이기 때문에 성능에 이점을 가진다

API

JdbcBatchItemWriter가 실행되는 과정

  • Step에서 JdbcBatchItemWriter가 실행되고 ColumnMapped() 방식인지 아니면 일반 클래스인 BeanMapped() 방식인지 선택되고 이후 JdbcTemplate을 통해 Bulk Insert 연산이 실행된다.
  • ColumnMappe 방시과 BeanMapped 방식의 의미는 아래의 그림과 같다.

예제 코드

create table customer
(
    id        mediumint unsigned auto_increment
        primary key,
    firstName varchar(255) null,
    lastName  varchar(255) null,
    birthdate varchar(255) null
);
create table customer2
(
    id        mediumint unsigned auto_increment
        primary key,
    firstName varchar(255) null,
    lastName  varchar(255) null,
    birthdate varchar(255) null
);
//customer에 데이터 입력, customer2는 customer에서 읽은 데이터를 customer2에 넣는 용도
INSERT INTO `customer` (`id`, `firstName`, `lastName`, `birthdate`)
VALUES (1, "Reed", "Edwards", "1952-08-16 12:34:53"),
       (2, "Hoyt", "Park", "1981-02-18 08:07:58"),
       (3, "Leila", "Petty", "1972-06-11 08:43:55"),
       (4, "Denton", "Strong", "1989-03-11 18:38:31"),
       (5, "Zoe", "Romero", "1990-10-02 13:06:31"),
       (6, "Rana", "Compton", "1957-06-09 12:51:11"),
       (7, "Velma", "King", "1988-02-02 05:52:25"),
       (8, "Uriah", "Carter", "1972-08-31 07:32:05"),
       (9, "Michael", "Graves", "1958-04-13 18:47:44"),
       (10, "Leigh", "Stone", "1967-06-23 23:41:43");
INSERT INTO `customer` (`id`, `firstName`, `lastName`, `birthdate`)
VALUES (11, "Iliana", "Glenn", "1965-02-27 14:33:56"),
       (12, "Harrison", "Haley", "1956-06-28 03:15:41"),
       (13, "Leonard", "Zamora", "1956-03-28 15:03:09"),
       (14, "Hiroko", "Wyatt", "1960-08-22 23:53:50"),
       (15, "Cameron", "Carlson", "1969-05-12 11:10:09"),
       (16, "Hunter", "Avery", "1953-11-19 12:52:42"),
       (17, "Aimee", "Cox", "1976-10-15 12:56:50"),
       (18, "Yen", "Delgado", "1990-02-06 10:25:36"),
       (19, "Gemma", "Peterson", "1989-04-02 23:42:09"),
       (20, "Lani", "Faulkner", "1970-09-18 17:22:14");
INSERT INTO `customer` (`id`, `firstName`, `lastName`, `birthdate`)
VALUES (21, "Iola", "Cannon", "1954-01-12 16:56:45"),
       (22, "Whitney", "Shaffer", "1951-03-19 01:27:18"),
       (23, "Jerome", "Moran", "1968-03-16 05:26:22"),
       (24, "Quinn", "Wheeler", "1979-06-19 16:24:22"),
       (25, "Mira", "Wilder", "1961-12-27 12:11:07"),
       (26, "Tobias", "Holloway", "1968-08-13 20:36:19"),
       (27, "Shaine", "Schneider", "1958-03-08 09:47:10"),
       (28, "Harding", "Gonzales", "1952-04-11 02:06:25"),
       (29, "Calista", "Nieves", "1970-02-17 13:29:59"),
       (30, "Duncan", "Norman", "1987-09-13 00:54:49");
INSERT INTO `customer` (`id`, `firstName`, `lastName`, `birthdate`)
VALUES (31, "Fatima", "Hamilton", "1961-06-16 14:29:11"),
       (32, "Ali", "Browning", "1979-03-27 17:09:37"),
       (33, "Erin", "Sosa", "1990-08-23 10:43:58"),
       (34, "Carol", "Harmon", "1972-01-14 07:19:39"),
       (35, "Illiana", "Fitzgerald", "1970-08-19 02:33:46"),
       (36, "Stephen", "Riley", "1954-06-05 08:34:03"),
       (37, "Hermione", "Waller", "1969-09-08 01:19:07"),
       (38, "Desiree", "Flowers", "1952-06-25 13:34:45"),
       (39, "Karyn", "Blackburn", "1977-03-30 13:08:02"),
       (40, "Briar", "Carroll", "1985-03-26 01:03:34");
INSERT INTO `customer` (`id`, `firstName`, `lastName`, `birthdate`)
VALUES (41, "Chaney", "Green", "1987-04-20 18:56:53"),
       (42, "Robert", "Higgins", "1985-09-26 11:25:10"),
       (43, "Lillith", "House", "1982-12-06 02:24:23"),
       (44, "Astra", "Winters", "1952-03-13 01:13:07"),
       (45, "Cherokee", "Stephenson", "1955-10-23 16:57:33"),
       (46, "Yuri", "Shaw", "1958-07-14 15:10:07"),
       (47, "Boris", "Sparks", "1982-01-01 10:56:34"),
       (48, "Wilma", "Blake", "1963-06-07 16:32:33"),
       (49, "Brynne", "Morse", "1964-09-21 01:05:25"),
       (50, "Ila", "Conley", "1953-11-02 05:12:57");
INSERT INTO `customer` (`id`, `firstName`, `lastName`, `birthdate`)
VALUES (51, "Sharon", "Watts", "1964-01-09 16:32:37"),
       (52, "Kareem", "Vaughan", "1952-04-18 15:37:10"),
       (53, "Eden", "Barnes", "1954-07-04 01:26:44"),
       (54, "Kenyon", "Fulton", "1975-08-23 22:17:52"),
       (55, "Mona", "Ball", "1972-02-11 04:15:45"),
       (56, "Moses", "Cortez", "1979-04-24 15:26:46"),
       (57, "Macy", "Banks", "1956-12-31 00:41:15"),
       (58, "Brenna", "Mendez", "1972-10-02 07:58:27"),
       (59, "Emerald", "Ewing", "1985-11-28 21:15:20"),
       (60, "Lev", "Mcfarland", "1951-05-20 14:30:07");
INSERT INTO `customer` (`id`, `firstName`, `lastName`, `birthdate`)
VALUES (61, "Norman", "Tanner", "1959-07-29 15:41:45"),
       (62, "Alexa", "Walters", "1977-12-06 16:41:17"),
       (63, "Dara", "Hyde", "1989-08-04 14:06:43"),
       (64, "Hu", "Sampson", "1978-11-01 17:10:23"),
       (65, "Jasmine", "Cardenas", "1969-02-15 20:08:06"),
       (66, "Julian", "Bentley", "1954-07-11 03:27:51"),
       (67, "Samson", "Brown", "1967-10-15 07:03:59"),
       (68, "Gisela", "Hogan", "1985-01-19 03:16:20"),
       (69, "Jeanette", "Cummings", "1986-09-07 18:25:52"),
       (70, "Galena", "Perkins", "1984-01-13 02:15:31");
INSERT INTO `customer` (`id`, `firstName`, `lastName`, `birthdate`)
VALUES (71, "Olga", "Mays", "1981-11-20 22:39:27"),
       (72, "Ferdinand", "Austin", "1956-08-08 09:08:02"),
       (73, "Zenia", "Anthony", "1964-08-21 05:45:16"),
       (74, "Hop", "Hampton", "1982-07-22 14:11:00"),
       (75, "Shaine", "Vang", "1970-08-13 15:58:28"),
       (76, "Ariana", "Cochran", "1959-12-04 01:18:36"),
       (77, "India", "Paul", "1963-10-10 05:24:03"),
       (78, "Karina", "Doyle", "1979-12-01 00:05:21"),
       (79, "Delilah", "Johnston", "1989-03-04 23:50:01"),
       (80, "Hilel", "Hood", "1959-08-22 06:40:48");
INSERT INTO `customer` (`id`, `firstName`, `lastName`, `birthdate`)
VALUES (81, "Kennedy", "Hoffman", "1963-10-14 20:18:35"),
       (82, "Kameko", "Bell", "1976-06-08 15:35:54"),
       (83, "Lunea", "Gutierrez", "1964-06-07 16:21:24"),
       (84, "William", "Burris", "1980-05-01 17:58:23"),
       (85, "Kiara", "Walls", "1955-12-27 18:57:15"),
       (86, "Latifah", "Alexander", "1980-06-19 10:39:50"),
       (87, "Keaton", "Ward", "1964-10-12 16:03:18"),
       (88, "Jasper", "Clements", "1970-03-05 00:29:49"),
       (89, "Claire", "Brown", "1972-02-11 00:43:58"),
       (90, "Noble", "Morgan", "1955-09-05 05:35:01");
INSERT INTO `customer` (`id`, `firstName`, `lastName`, `birthdate`)
VALUES (91, "Evangeline", "Horn", "1952-12-28 14:06:27"),
       (92, "Jonah", "Harrell", "1951-06-25 17:37:35"),
       (93, "Mira", "Espinoza", "1982-03-26 06:01:16"),
       (94, "Brennan", "Oneill", "1979-04-23 08:49:02"),
       (95, "Dacey", "Howe", "1983-02-06 19:11:00"),
       (96, "Yoko", "Pittman", "1982-09-12 02:18:52"),
       (97, "Cody", "Conway", "1971-05-26 07:09:58"),
       (98, "Jordan", "Knowles", "1981-12-30 02:20:01"),
       (99, "Pearl", "Boyer", "1957-10-19 14:26:49"),
       (100, "Keely", "Montoya", "1985-03-24 01:18:09");
package com.example.springbatch_9_5_jobbatchitemwriter;

import lombok.RequiredArgsConstructor;
import org.springframework.batch.core.Job;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.launch.support.RunIdIncrementer;
import org.springframework.batch.item.ItemWriter;
import org.springframework.batch.item.database.JdbcPagingItemReader;
import org.springframework.batch.item.database.Order;
import org.springframework.batch.item.database.builder.JdbcBatchItemWriterBuilder;
import org.springframework.batch.item.database.support.MySqlPagingQueryProvider;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import javax.sql.DataSource;
import java.util.HashMap;

/**
 * packageName    : com.example.springbatch_9_5_jobbatchitemwriter
 * fileName       : JdbcBatchConfiguration
 * author         : namhyeop
 * date           : 2022/08/15
 * description    :
 * DB에서 읽은 테이블 정보를 다른 테이블에 JdbcBatchItemWriter를 사용해서 입력하는 예체
 * ===========================================================
 * DATE              AUTHOR             NOTE
 * -----------------------------------------------------------
 * 2022/08/15        namhyeop       최초 생성
 */
@RequiredArgsConstructor
@Configuration
public class JdbcBatchConfiguration {

    private final JobBuilderFactory jobBuilderFactory;
    private final StepBuilderFactory stepBuilderFactory;
    private final DataSource dataSource;

    @Bean
    public Job job() {
        return jobBuilderFactory.get("batchJob")
                .incrementer(new RunIdIncrementer())
                .start(step1())
                .build();
    }

    @Bean
    public Step step1() {
        return stepBuilderFactory.get("step1")
                .<Customer, Customer>chunk(10)
                .reader(customItemReader())
                .writer(customItemWriter())
                .build();
    }

    @Bean
    public ItemWriter<? super Customer> customItemWriter() {
        return new JdbcBatchItemWriterBuilder<Customer>()
                .dataSource(dataSource)
                .sql("insert into customer2 values (:id, :firstName, :lastName, :birthdate)")
                .beanMapped()
                .build();
    }

    @Bean
    public JdbcPagingItemReader<Customer> customItemReader() {
        JdbcPagingItemReader<Customer> reader = new JdbcPagingItemReader<>();

        reader.setDataSource(dataSource);
        reader.setFetchSize(10);
        reader.setRowMapper(new CustomRowMapper());

        MySqlPagingQueryProvider queryProvider = new MySqlPagingQueryProvider();
        queryProvider.setSelectClause("id, firstName, lastName, birthdate");
        queryProvider.setFromClause("from customer");
        queryProvider.setWhereClause("where firstname like :firstname");

        HashMap<String, Order> sortKeys = new HashMap<>(1);
        sortKeys.put("id", Order.ASCENDING);
        queryProvider.setSortKeys(sortKeys);
        reader.setQueryProvider(queryProvider);

        HashMap<String, Object> parameters = new HashMap<>();
        parameters.put("firstname", "A%");

        reader.setParameterValues(parameters);
        return reader;
    }
}
package com.example.springbatch_9_5_jobbatchitemwriter;

import org.springframework.jdbc.core.RowMapper;

import java.sql.ResultSet;
import java.sql.SQLException;

/**
 * packageName    : com.example.springbatch_9_3_xmlstaxeventitemwriter
 * fileName       : CustomerRowMapper
 * author         : namhyeop
 * date           : 2022/08/14
 * description
 * DB의 정보를 객체로 매핑 시켜주는 RowMapper
 * ===========================================================
 * DATE              AUTHOR             NOTE
 * -----------------------------------------------------------
 * 2022/08/14        namhyeop       최초 생성
 */
public class CustomRowMapper implements RowMapper<Customer> {

    @Override
    public Customer mapRow(ResultSet rs, int rowNum) throws SQLException {
        return new Customer(rs.getLong("id"),
                rs.getString("firstName"),
                rs.getString("lastName"),
                rs.getDate("birthdate"));
    }
}
package com.example.springbatch_9_5_jobbatchitemwriter;

import lombok.AllArgsConstructor;
import lombok.Data;

import java.util.Date;

/**
 * packageName    : com.example.springbatch_9_5_jobbatchitemwriter
 * fileName       : Customer
 * author         : namhyeop
 * date           : 2022/08/15
 * description    :
 * Customer 객체
 * ===========================================================
 * DATE              AUTHOR             NOTE
 * -----------------------------------------------------------
 * 2022/08/15        namhyeop       최초 생성
 */
@Data
@AllArgsConstructor
public class Customer {

    private final Long id;
    private final String firstName;
    private final String lastName;
    private final Date birthdate;
}
  • 실행 후 Customer에서 조회한 정보가 Customer2에 insert되었는지 확인하자

DB-JpaItemWriter

기본개념

  • JPA Entity 기반으로 데이터를 처리하며 EntityManagerFactory 를 주입받아 사용한다
  • Entity를 하나씩 chunk 크기 만큼 insert 혹은 merge 한 다음 flush 한다
  • ItemReader 나 ItemProcessor 로 부터 아이템을 전발 받을 때는 Entity 클래스 타입으로 받아야 한다

API

JpaItemWriter가 실행되는 과정

예제코드

                //modelmapper 디펜던스 추가, process에서 전달받은 input객체를 output객체로 전환하기 위해서 ModelMapper를 사용함
                <dependency>
            <groupId>org.modelmapper</groupId>
            <artifactId>modelmapper</artifactId>
            <version>2.3.0</version>
        </dependency>
package com.example.springboot_9_6_jpaitemwriter;

import lombok.AllArgsConstructor;
import lombok.Data;

import java.util.Date;

/**
 * packageName    : com.example.springbatch_9_5_jobbatchitemwriter
 * fileName       : Customer
 * author         : namhyeop
 * date           : 2022/08/15
 * description    :
 * Customer 객체
 * ===========================================================
 * DATE              AUTHOR             NOTE
 * -----------------------------------------------------------
 * 2022/08/15        namhyeop       최초 생성
 */
@Data
@AllArgsConstructor
public class Customer {

    private final Long id;
    private final String firstName;
    private final String lastName;
    private final Date birthdate;
}
package com.example.springboot_9_6_jpaitemwriter;

import lombok.AllArgsConstructor;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;

import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.Id;
import java.util.Date;

/**
 * packageName    : com.example.springbatch_9_5_jobbatchitemwriter
 * fileName       : Customer
 * author         : namhyeop
 * date           : 2022/08/15
 * description    :
 * Customer Entity, Jpa를 사용하기에 Getter,Setter로만 이루어진 pojo객체 필요
 * ===========================================================
 * DATE              AUTHOR             NOTE
 * -----------------------------------------------------------
 * 2022/08/15        namhyeop       최초 생성
 */
@Getter
@Setter
@NoArgsConstructor
@Entity
public class Customer2 {
    @Id
    private Long id;
    private String firstName;
    private String lastName;
    private Date birthdate;
}
package com.example.springboot_9_6_jpaitemwriter;

import org.modelmapper.ModelMapper;
import org.springframework.batch.item.ItemProcessor;

/**
 * packageName    : com.example.springboot_9_6_jpaitemwriter
 * fileName       : CustomItemProcessor
 * author         : namhyeop
 * date           : 2022/08/15
 * description    :
 * Customer객체를 Customer2로 변환하는 processor
 * ===========================================================
 * DATE              AUTHOR             NOTE
 * -----------------------------------------------------------
 * 2022/08/15        namhyeop       최초 생성
 */
public class CustomItemProcessor implements ItemProcessor<Customer,Customer2> {

    private ModelMapper modelMapper = new ModelMapper();

    @Override
    public Customer2 process(Customer item) throws Exception {
        //ModelMapper는 item객체가 들어올 경우 Custsomer2로 전환해준다.
        Customer2 customer2 = modelMapper.map(item, Customer2.class);
        //전환한 customer2를 반환
        return customer2;
    }
}
package com.example.springboot_9_6_jpaitemwriter;

import org.springframework.jdbc.core.RowMapper;

import java.sql.ResultSet;
import java.sql.SQLException;

/**
 * packageName    : com.example.springbatch_9_3_xmlstaxeventitemwriter
 * fileName       : CustomerRowMapper
 * author         : namhyeop
 * date           : 2022/08/14
 * description
 * DB의 정보를 객체로 매핑 시켜주는 RowMapper
 * ===========================================================
 * DATE              AUTHOR             NOTE
 * -----------------------------------------------------------
 * 2022/08/14        namhyeop       최초 생성
 */
public class CustomRowMapper implements RowMapper<Customer> {

    @Override
    public Customer mapRow(ResultSet rs, int rowNum) throws SQLException {
        return new Customer(rs.getLong("id"),
                rs.getString("firstName"),
                rs.getString("lastName"),
                rs.getDate("birthdate"));
    }
}
package com.example.springboot_9_6_jpaitemwriter;

import lombok.RequiredArgsConstructor;
import org.springframework.batch.core.Job;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.launch.support.RunIdIncrementer;
import org.springframework.batch.item.ItemProcessor;
import org.springframework.batch.item.ItemWriter;
import org.springframework.batch.item.database.JdbcPagingItemReader;
import org.springframework.batch.item.database.Order;
import org.springframework.batch.item.database.builder.JpaItemWriterBuilder;
import org.springframework.batch.item.database.support.MySqlPagingQueryProvider;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import javax.persistence.EntityManagerFactory;
import javax.sql.DataSource;
import java.util.HashMap;

/**
 * packageName    : com.example.springbatch_9_5_jobbatchitemwriter
 * fileName       : JdbcBatchConfiguration
 * author         : namhyeop
 * date           : 2022/08/15
 * description    :
 * DB에서 읽은 테이블 정보를 다른 테이블에 JdbcBatchItemWriter를 사용해서 입력하는 예체
 * ===========================================================
 * DATE              AUTHOR             NOTE
 * -----------------------------------------------------------
 * 2022/08/15        namhyeop       최초 생성
 */
@RequiredArgsConstructor
@Configuration
public class JdbcBatchConfiguration {

    private final JobBuilderFactory jobBuilderFactory;
    private final StepBuilderFactory stepBuilderFactory;
    private final DataSource dataSource;
    private final EntityManagerFactory entityManagerFactory;

    @Bean
    public Job job() {
        return jobBuilderFactory.get("batchJob")
                .incrementer(new RunIdIncrementer())
                .start(step1())
                .build();
    }

    @Bean
    public Step step1() {
        return stepBuilderFactory.get("step1")
                .<Customer, Customer2>chunk(10)
                .reader(customItemReader())
                .processor(customItemProcessor())
                .writer(customItemWriter())
                .build();
    }

//    @Bean
//    public JpaItemWriter<Customer2> customItemWriter() {
//        return new JpaItemWriterBuilder<Customer2>()
//                .entityManagerFactory(entityManagerFactory)
//                .usePersist(true)
//                .build();
//    }

    @Bean
    public ItemWriter<? super Customer2> customItemWriter() {
        return new JpaItemWriterBuilder<Customer2>()
                .usePersist(true)
                .entityManagerFactory(entityManagerFactory)
                .build();
    }

//    @Bean
//    public ItemProcessor<Customer,Customer2> customItemProcessor() {
//        return new CustomItemProcessor();
//    }
    @Bean
    public ItemProcessor<? super Customer, ? extends Customer2> customItemProcessor() {
        return new CustomItemProcessor();
    }

    @Bean
    public JdbcPagingItemReader<Customer> customItemReader() {
        JdbcPagingItemReader<Customer> reader = new JdbcPagingItemReader<>();

        reader.setDataSource(dataSource);
        reader.setFetchSize(10);
        reader.setRowMapper(new CustomRowMapper());

        MySqlPagingQueryProvider queryProvider = new MySqlPagingQueryProvider();
        queryProvider.setSelectClause("id, firstName, lastName, birthdate");
        queryProvider.setFromClause("from customer");
        queryProvider.setWhereClause("where firstname like :firstname");

        HashMap<String, Order> sortKeys = new HashMap<>(1);
        sortKeys.put("id", Order.ASCENDING);
        queryProvider.setSortKeys(sortKeys);
        reader.setQueryProvider(queryProvider);

        HashMap<String, Object> parameters = new HashMap<>();
        parameters.put("firstname", "A%");

        reader.setParameterValues(parameters);
        return reader;
    }
}
  • customer 데이터가 cutomer2로 잘 전달되었는지 확인하자

ItemWriterAdapter

기본개념

  • 배치 Job 안에서 이미 있는 DAO 나 다른 서비스를 ItemWriter 안에서 사용하고자 할 때 위임 역할을 한다
  • 전체적인 흐름은 좌측 그림처럼 customItemWriter에서 호출할 service와 실행할 메소드를 설정한다.
  • 그러면 Spring Batch의 ItemWriterAdpater(우측 상단)으로 이동하여 invokeDelegateMethodWithArgument에 객체를 전달하게 된다.
  • 이후 invokeDelegateMethodWithArgument에서 실핼항 서비스와 실행할 메소드 정보를 CustomService를 통해 실행한다.

예제코드

package com.example.springboot_9_7_itemwriteadapter;

import lombok.RequiredArgsConstructor;
import org.springframework.batch.core.Job;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.launch.support.RunIdIncrementer;
import org.springframework.batch.item.*;
import org.springframework.batch.item.adapter.ItemWriterAdapter;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

/**
 * packageName    : com.example.springboot_9_7_itemwriteadapter
 * fileName       : ItemWriteAdapterConfiguration
 * author         : namhyeop
 * date           : 2022/08/15
 * description    :
 * Write 실행 도중 별도의 Service를 사용하는 예제
 * ===========================================================
 * DATE              AUTHOR             NOTE
 * -----------------------------------------------------------
 * 2022/08/15        namhyeop       최초 생성
 */
@RequiredArgsConstructor
@Configuration
public class ItemWriteAdapterConfiguration {

    private final JobBuilderFactory jobBuilderFactory;
    private final StepBuilderFactory stepBuilderFactory;

    @Bean
    public Job job(){
        return jobBuilderFactory.get("batchJob")
                .incrementer(new RunIdIncrementer())
                .start(step1())
                .build();
    }

    @Bean
    public Step step1(){
        return stepBuilderFactory.get("step1")
                .<String,String>chunk(10)
                .reader(new ItemReader<String>() {
                    int i = 0;
                    @Override
                    public String read() throws Exception, UnexpectedInputException, ParseException, NonTransientResourceException {
                        i++;
                        return i > 10 ? null : "item" + i;
                    }
                })
                .writer(customItemWriter())
                .build();
    }

    @Bean
    public ItemWriter<? super String> customItemWriter() {
        ItemWriterAdapter<String> writer = new ItemWriterAdapter<>();
        writer.setTargetObject(customService());
        writer.setTargetMethod("customWrite");

        return writer;
    }

    @Bean
    public CustomService customService() {
        return new CustomService();
    }
}
package com.example.springboot_9_7_itemwriteadapter;

import org.springframework.stereotype.Service;

/**
 * packageName    : com.example.springboot_9_7_itemwriteadapter
 * fileName       : CustomService
 * author         : namhyeop
 * date           : 2022/08/15
 * description    :
 * write 실행 도중 실행될 서비스
 * ===========================================================
 * DATE              AUTHOR             NOTE
 * -----------------------------------------------------------
 * 2022/08/15        namhyeop       최초 생성
 */

public class CustomService<T> {

    public void customWrite(T item){
        System.out.println("item = " + item);
    }
}

DB-JpaItemWriter

기본개념

  • JPA Entity 기반으로 데이터를 처리하며 EntityManagerFactory 를 주입받아 사용한다
  • Entity를 하나씩 chunk 크기 만큼 insert 혹은 merge 한 다음 flush 한다
  • ItemReader 나 ItemProcessor 로 부터 아이템을 전발 받을 때는 Entity 클래스 타입으로 받아야 한다

API

JpaItemWriter가 실행되는 과정

예제코드

				//modelmapper 디펜던스 추가, process에서 전달받은 input객체를 output객체로 전환하기 위해서 ModelMapper를 사용함
				<dependency>
            <groupId>org.modelmapper</groupId>
            <artifactId>modelmapper</artifactId>
            <version>2.3.0</version>
        </dependency>

package com.example.springboot_9_6_jpaitemwriter;

import lombok.AllArgsConstructor;
import lombok.Data;

import java.util.Date;

/**
 * packageName    : com.example.springbatch_9_5_jobbatchitemwriter
 * fileName       : Customer
 * author         : namhyeop
 * date           : 2022/08/15
 * description    :
 * Customer 객체
 * ===========================================================
 * DATE              AUTHOR             NOTE
 * -----------------------------------------------------------
 * 2022/08/15        namhyeop       최초 생성
 */
@Data
@AllArgsConstructor
public class Customer {

    private final Long id;
    private final String firstName;
    private final String lastName;
    private final Date birthdate;
}

package com.example.springboot_9_6_jpaitemwriter;

import lombok.AllArgsConstructor;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;

import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.Id;
import java.util.Date;

/**
 * packageName    : com.example.springbatch_9_5_jobbatchitemwriter
 * fileName       : Customer
 * author         : namhyeop
 * date           : 2022/08/15
 * description    :
 * Customer Entity, Jpa를 사용하기에 Getter,Setter로만 이루어진 pojo객체 필요
 * ===========================================================
 * DATE              AUTHOR             NOTE
 * -----------------------------------------------------------
 * 2022/08/15        namhyeop       최초 생성
 */
@Getter
@Setter
@NoArgsConstructor
@Entity
public class Customer2 {
    @Id
    private Long id;
    private String firstName;
    private String lastName;
    private Date birthdate;
}

package com.example.springboot_9_6_jpaitemwriter;

import org.modelmapper.ModelMapper;
import org.springframework.batch.item.ItemProcessor;

/**
 * packageName    : com.example.springboot_9_6_jpaitemwriter
 * fileName       : CustomItemProcessor
 * author         : namhyeop
 * date           : 2022/08/15
 * description    :
 * Customer객체를 Customer2로 변환하는 processor
 * ===========================================================
 * DATE              AUTHOR             NOTE
 * -----------------------------------------------------------
 * 2022/08/15        namhyeop       최초 생성
 */
public class CustomItemProcessor implements ItemProcessor<Customer,Customer2> {

    private ModelMapper modelMapper = new ModelMapper();

    @Override
    public Customer2 process(Customer item) throws Exception {
        //ModelMapper는 item객체가 들어올 경우 Custsomer2로 전환해준다.
        Customer2 customer2 = modelMapper.map(item, Customer2.class);
        //전환한 customer2를 반환
        return customer2;
    }
}

package com.example.springboot_9_6_jpaitemwriter;

import org.springframework.jdbc.core.RowMapper;

import java.sql.ResultSet;
import java.sql.SQLException;

/**
 * packageName    : com.example.springbatch_9_3_xmlstaxeventitemwriter
 * fileName       : CustomerRowMapper
 * author         : namhyeop
 * date           : 2022/08/14
 * description
 * DB의 정보를 객체로 매핑 시켜주는 RowMapper
 * ===========================================================
 * DATE              AUTHOR             NOTE
 * -----------------------------------------------------------
 * 2022/08/14        namhyeop       최초 생성
 */
public class CustomRowMapper implements RowMapper<Customer> {

    @Override
    public Customer mapRow(ResultSet rs, int rowNum) throws SQLException {
        return new Customer(rs.getLong("id"),
                rs.getString("firstName"),
                rs.getString("lastName"),
                rs.getDate("birthdate"));
    }
}

package com.example.springboot_9_6_jpaitemwriter;

import lombok.RequiredArgsConstructor;
import org.springframework.batch.core.Job;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.launch.support.RunIdIncrementer;
import org.springframework.batch.item.ItemProcessor;
import org.springframework.batch.item.ItemWriter;
import org.springframework.batch.item.database.JdbcPagingItemReader;
import org.springframework.batch.item.database.Order;
import org.springframework.batch.item.database.builder.JpaItemWriterBuilder;
import org.springframework.batch.item.database.support.MySqlPagingQueryProvider;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import javax.persistence.EntityManagerFactory;
import javax.sql.DataSource;
import java.util.HashMap;

/**
 * packageName    : com.example.springbatch_9_5_jobbatchitemwriter
 * fileName       : JdbcBatchConfiguration
 * author         : namhyeop
 * date           : 2022/08/15
 * description    :
 * DB에서 읽은 테이블 정보를 다른 테이블에 JdbcBatchItemWriter를 사용해서 입력하는 예체
 * ===========================================================
 * DATE              AUTHOR             NOTE
 * -----------------------------------------------------------
 * 2022/08/15        namhyeop       최초 생성
 */
@RequiredArgsConstructor
@Configuration
public class JdbcBatchConfiguration {

    private final JobBuilderFactory jobBuilderFactory;
    private final StepBuilderFactory stepBuilderFactory;
    private final DataSource dataSource;
    private final EntityManagerFactory entityManagerFactory;

    @Bean
    public Job job() {
        return jobBuilderFactory.get("batchJob")
                .incrementer(new RunIdIncrementer())
                .start(step1())
                .build();
    }

    @Bean
    public Step step1() {
        return stepBuilderFactory.get("step1")
                .<Customer, Customer2>chunk(10)
                .reader(customItemReader())
                .processor(customItemProcessor())
                .writer(customItemWriter())
                .build();
    }

//    @Bean
//    public JpaItemWriter<Customer2> customItemWriter() {
//        return new JpaItemWriterBuilder<Customer2>()
//                .entityManagerFactory(entityManagerFactory)
//                .usePersist(true)
//                .build();
//    }

    @Bean
    public ItemWriter<? super Customer2> customItemWriter() {
        return new JpaItemWriterBuilder<Customer2>()
                .usePersist(true)
                .entityManagerFactory(entityManagerFactory)
                .build();
    }

//    @Bean
//    public ItemProcessor<Customer,Customer2> customItemProcessor() {
//        return new CustomItemProcessor();
//    }
    @Bean
    public ItemProcessor<? super Customer, ? extends Customer2> customItemProcessor() {
        return new CustomItemProcessor();
    }

    @Bean
    public JdbcPagingItemReader<Customer> customItemReader() {
        JdbcPagingItemReader<Customer> reader = new JdbcPagingItemReader<>();

        reader.setDataSource(dataSource);
        reader.setFetchSize(10);
        reader.setRowMapper(new CustomRowMapper());

        MySqlPagingQueryProvider queryProvider = new MySqlPagingQueryProvider();
        queryProvider.setSelectClause("id, firstName, lastName, birthdate");
        queryProvider.setFromClause("from customer");
        queryProvider.setWhereClause("where firstname like :firstname");

        HashMap<String, Order> sortKeys = new HashMap<>(1);
        sortKeys.put("id", Order.ASCENDING);
        queryProvider.setSortKeys(sortKeys);
        reader.setQueryProvider(queryProvider);

        HashMap<String, Object> parameters = new HashMap<>();
        parameters.put("firstname", "A%");

        reader.setParameterValues(parameters);
        return reader;
    }
}
  • customer 데이터가 cutomer2로 잘 전달되었는지 확인하자

ItemWriterAdapter

기본개념

  • 배치 Job 안에서 이미 있는 DAO 나 다른 서비스를 ItemWriter 안에서 사용하고자 할 때 위임 역할을 한다
  • 전체적인 흐름은 좌측 그림처럼 customItemWriter에서 호출할 service와 실행할 메소드를 설정한다.
  • 그러면 Spring Batch의 ItemWriterAdpater(우측 상단)으로 이동하여 invokeDelegateMethodWithArgument에 객체를 전달하게 된다.
  • 이후 invokeDelegateMethodWithArgument에서 실핼항 서비스와 실행할 메소드 정보를 CustomService를 통해 실행한다.

예제코드

package com.example.springboot_9_7_itemwriteadapter;

import lombok.RequiredArgsConstructor;
import org.springframework.batch.core.Job;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.launch.support.RunIdIncrementer;
import org.springframework.batch.item.*;
import org.springframework.batch.item.adapter.ItemWriterAdapter;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

/**
 * packageName    : com.example.springboot_9_7_itemwriteadapter
 * fileName       : ItemWriteAdapterConfiguration
 * author         : namhyeop
 * date           : 2022/08/15
 * description    :
 * Write 실행 도중 별도의 Service를 사용하는 예제
 * ===========================================================
 * DATE              AUTHOR             NOTE
 * -----------------------------------------------------------
 * 2022/08/15        namhyeop       최초 생성
 */
@RequiredArgsConstructor
@Configuration
public class ItemWriteAdapterConfiguration {

    private final JobBuilderFactory jobBuilderFactory;
    private final StepBuilderFactory stepBuilderFactory;

    @Bean
    public Job job(){
        return jobBuilderFactory.get("batchJob")
                .incrementer(new RunIdIncrementer())
                .start(step1())
                .build();
    }

    @Bean
    public Step step1(){
        return stepBuilderFactory.get("step1")
                .<String,String>chunk(10)
                .reader(new ItemReader<String>() {
                    int i = 0;
                    @Override
                    public String read() throws Exception, UnexpectedInputException, ParseException, NonTransientResourceException {
                        i++;
                        return i > 10 ? null : "item" + i;
                    }
                })
                .writer(customItemWriter())
                .build();
    }

    @Bean
    public ItemWriter<? super String> customItemWriter() {
        ItemWriterAdapter<String> writer = new ItemWriterAdapter<>();
        writer.setTargetObject(customService());
        writer.setTargetMethod("customWrite");

        return writer;
    }

    @Bean
    public CustomService customService() {
        return new CustomService();
    }
}

package com.example.springboot_9_7_itemwriteadapter;

import org.springframework.stereotype.Service;

/**
 * packageName    : com.example.springboot_9_7_itemwriteadapter
 * fileName       : CustomService
 * author         : namhyeop
 * date           : 2022/08/15
 * description    :
 * write 실행 도중 실행될 서비스
 * ===========================================================
 * DATE              AUTHOR             NOTE
 * -----------------------------------------------------------
 * 2022/08/15        namhyeop       최초 생성
 */

public class CustomService<T> {

    public void customWrite(T item){
        System.out.println("item = " + item);
    }
}

 

반응형

'Spring Batch' 카테고리의 다른 글

10.Spring Batch의 Chunk와 ItemProcessor  (0) 2024.11.08
8.Spring Batch의 Chunk와 ItemReader  (0) 2024.11.08
7.Spring Batch의 Chunk와 동작원리 살펴보기  (0) 2024.11.08
6.Spring Batch의 Flow  (1) 2024.11.07
5.Spring Batch의 Step  (0) 2024.11.07

    티스토리툴바