How to use Java 8's Stream to implement list deduplication?

1. The distinct() method of Stream

distinct() is a method provided by Stream in Java 8. It returns a stream composed of different elements in the stream. distinct() uses hashCode() and eqauls() methods to get distinct elements.

Therefore, the class that needs to be deduplicated must implement the hashCode() and equals() methods. In other words, we can achieve certain special needs by overriding the customized hashCode() and equals() methods.

distinct() method is declared as follows:

Stream<T> distinct();

1.1 For deduplication of String list

Because the String class has overridden the equals() and hashCode() methods, so You can successfully remove duplicates.

@Test
public void listDistinctByStreamDistinct() {
  // 1. 对于 String 列表去重
  List<String> stringList = new ArrayList<String>() {{
    add("A");
    add("A");
    add("B");
    add("B");
    add("C");
  }};
  out.print("去重前：");
  for (String s : stringList) {
    out.print(s);
  }
  out.println();
  stringList = stringList.stream().distinct().collect(Collectors.toList());
  out.print("去重后：");
  for (String s : stringList) {
    out.print(s);
  }
  out.println();

}

The results are as follows:

Before deduplication: AABBC
After deduplication: ABC

1.2 Deduplication of entity class lists

Note: In the code, we use the @Data annotation of the Lombok plug-in, which can automatically override the equals() and hashCode() methods.

/**
* 定义一个实体类
*/  
@Data
public class Student {
  private String stuNo;
  private String name;
}

@Test
public void listDistinctByStreamDistinct() throws JsonProcessingException {
    ObjectMapper objectMapper = new ObjectMapper();
    // 1. 对于 Student 列表去重
    List<Student> studentList = getStudentList();
    out.print("去重前：");
    out.println(objectMapper.writeValueAsString(studentList));
    studentList = studentList.stream().distinct().collect(Collectors.toList());
    out.print("去重后：");
    out.println(objectMapper.writeValueAsString(studentList));
  }

The results are as follows:

Before deduplication: [{"stuNo":"001","name":"Tom"},{"stuNo":"002" ,"name":"Mike"},{"stuNo":"001","name":"Tom"}]
After deduplication: [{"stuNo":"001","name":" Tom"},{"stuNo":"002","name":"Mike"}]

2. Remove duplicates based on a certain attribute of the Object in List