How to merge two streams in Java without duplicates according to some property?

Question

Suppose we have two streams as follows:

IntStream stream1 = Arrays.stream(new int[] {13, 1, 3, 5, 7, 9});
IntStream stream2 = Arrays.stream(new int[] {1, 2, 6, 14, 8, 10, 12});
stream1.merge(stream2); // some method which is used to merge two streams.

Is there any convenient way to merge these two streams without duplicates to [13, 1, 2, 3, 5, 6, 7, 8, 9, 10, 12, 14] by using the Java 8 stream API (the order doesn't matter). Or can we only handle one stream at the same time?

Furthermore, if the two streams are object streams, how to keep only distinct objects without overriding the equals() and hashCode() methods? For example:

public class Student {

    private String no;

    private String name;
}

Student s1 = new Student("1", "May");
Student s2 = new Student("2", "Bob");
Student s3 = new Student("1", "Marry");

Stream<Student> stream1 = Stream.of(s1, s2);
Stream<Student> stream2 = Stream.of(s2, s3);
stream1.merge(stream2);  // should return Student{no='1', name='May'} Student{no='2', name='Bob'}

As long as their no is the same we consider them to be the same student. (so May and Marry are the same person because their numbers are both "1").

I've found the distinct() method, but this method is based on Object#equals(). If we are not allowed to overwrite the equals() method, how can we merge stream1 and stream2 to one stream which has no duplicate items?

Selfish

Instead of Arrays.stream(new int[] {13, 1, 3, 5, 7, 9}); you can write directly IntStream.of(13, 1, etc). – Lehr

As mentioned in https://mcmap.net/q/546724/-how-to-merge-two-streams-in-java-without-duplicates-according-to-some-property, Stream.concat(stream1, stream2) will concatenate the streams. Excluding duplicates according to a property has its own full answer here: https://mcmap.net/q/63615/-java-8-distinct-by-property/1108305. – Tenantry

Answer 1

@Jigar Joshi has answered the first part of your question which is "how to merge two IntStream's into one".

Your other question of "how to merge two Stream<T> without overwriting the equals() and hashCode() method?" can be done using the toMap collector, i.e. assuming you don't want the result as a Stream<T>. Example:

Stream.concat(stream1, stream2)
      .collect(Collectors.toMap(Student::getNo, 
               Function.identity(), 
               (l, r) -> l, 
               LinkedHashMap::new)
      ).values();

if you want the result as a Stream<T> then one could do:

 Stream.concat(stream1, stream2)
       .collect(Collectors.collectingAndThen(
               Collectors.toMap(Student::getNo,
                    Function.identity(),
                    (l, r) -> l,
                    LinkedHashMap::new), 
                    f -> f.values().stream()));

This is possibly not as efficient as it can be but it's another way to return a Stream<T> where the T items are all distinct but without using overriding equals and hashcode as you've mentioned.

Fabria

Answer 2

You can use concat()

IntStream.concat(stream1, stream2)

Pinniped

Thank you, it worked. But it is a pity that contact() method does not remove the duplicate items for us, so we have to use one more distinct() method. – Selfish

@weaver why would you expect concat to remove duplicate items? – Fabria

In order to remove duplicates you can call .distinct() after concatenation. – Tutorial

@weaver That's the point of streams... chain operations on the stream. – Moores

There's a full answer on removing duplicates from a stream, which could be applied to a concatenated object stream: https://mcmap.net/q/63615/-java-8-distinct-by-property/1108305 – Tenantry

Answer 3

for first question you can use "flatMap"

    IntStream stream1 = Arrays.stream(new int[] {13, 1, 3, 5, 7, 9});
    IntStream stream2 = Arrays.stream(new int[] {1, 2, 6, 14, 8, 10, 12});

    List<Integer> result = Stream.of(stream1, stream2).flatMap(IntStream::boxed)
            .collect(Collectors.toList());
    //result={13, 1, 3, 5, 7, 9, 1, 2, 6, 14, 8, 10, 12}

EDIT

thanks to @Vinicius for advising, we can use

Stream<Integer> result = Stream.of(stream1, stream2).flatMap(IntStream::boxed).distinct();

here we will get a stream of all elements distinct based on the equals method.

Tired

Just put a distinct() after the flatMap and that is the better answer – Quirinal

A variant for the stream of non-primitives: Stream.of(stream1, stream2, ... streamN).flatMap(Function.identity()) – Fundy

Answer 4

Here's a solution using the StreamEx library:

Stream<Student> merged =
        StreamEx.of(stream1).append(stream2).distinct(Student::getNo)

This uses the StreamEx append method to concatenate the two streams. It then uses the StreamEx variation of the distinct method which uses a function to determine whether two elements are to be considered equal:

public S distinct​(Function<? super T,​?> keyExtractor)

Returns a stream consisting of the distinct elements of this stream (according to object equality of the results of applying the given function).

Tenantry