Test Dataflow with DirectRunner and got lots of verifyUnmodifiedThrowingCheckedExceptions

Question

I was testing my Dataflow pipeline using DirectRunner from my Mac and got lots of "WARNING" message like this, may I know how to get rid of them because it is too much that I can not even see my debug message.

Thanks

Apr 05, 2018 2:14:48 PM org.apache.beam.sdk.util.MutationDetectors$CodedValueMutationDetector verifyUnmodifiedThrowingCheckedExceptions
WARNING: Coder of type class org.apache.beam.sdk.coders.SerializableCoder has a #structuralValue method which does not return true when the encoding of the elements is equal. 
Element com.apigee.analytics.platform.core.service.schema.EventRow@4a590d0b

Jair

Answer 1

It may help to ensure that all serialized values have proper equals() implementations since SerializableCoder expects them:

The structural value of the object is the object itself. The SerializableCoder should be only used for objects with a proper Object#equals implementation.

You can implement your own Coder for your POJOs. SerializableCoder does not guarantee a deterministic encoding according to docs:

SerializableCoder does not guarantee a deterministic encoding, as Java serialization may produce different binary encodings for two equivalent objects.

This article explains custom coders in details.

Kinin

Do you have any examples or resources on how to implement a Coder? I'm struggling with this same exception and I can't seem to make heads or tails just from the javadoc for the Coder – Tergiversate

@Tergiversate I just added good article about custom coder to original answer. – Kinin

I'm having these as well. Not sure what the resolution is - should I implement a custom coder? – Corny

Answer 2

Just add https://projectlombok.org/features/EqualsAndHashCode

@EqualsAndHashCode 
public class YourRecord implements Serializable {

Ruthful

Answer 3

I had this same problem. I was using SerializableCoder for a class implementing Serializable, and my tests were failing because the PAssert() containsInAnyOrder() method was not using MyClass.equals() to evaluate object equality. The signature of my equals() method was:

public boolean equals(MyClass other) {...}

All I had to do to fix it was to define equals in terms of Object:

public boolean equals(Object other) {...}

This made the warnings go away, and made the tests pass.

Sirotek