Class not found Kryo exception in hive 0.13 - Hadoop
Asked Answered
P

0

7

I have a GenericUDF (see code below) that was running fine on Hadoop-1 and Hive-0.12. But when testing the same GenericUDF using Hive-0.13 + Hadoop-2, I am getting the below error.

Vertex failed, vertexName=Map 12, vertexId=vertex_1409698731658_42202_1_00, diagnostics=[Vertex Input: ccv initializer failed., org.apache.hive.com.esotericsoftware.kry o.KryoException: Unable to find class: com.xxx.xxx.Id1

Here is the code for my UDF.

package com.xxx.xxx;
import org.apache.hadoop.hive.*;

public class Id1 extends GenericUDF {
    private MapredContext context;
    private long sequenceNum = 0;
    private static final int padLength = 10;
    StringBuilder sb = null;

    public ObjectInspector initialize(ObjectInspector[] arguments)
            throws UDFArgumentException {
        sequenceNum = 0;
        sb = new StringBuilder();
        return PrimitiveObjectInspectorFactory.javaStringObjectInspector;
    }

    public Object evaluate(DeferredObject[] arguments) throws HiveException {
        int sbLength = sb.toString().length();
        if (sbLength > 0)
            sb.replace(0, sbLength, "");
        String taskId = null;
        if (context.getJobConf() != null)
            taskId = context.getJobConf().get("mapred.taskid");
        sequenceNum++;
        if (taskId != null) {
            sb.append(taskId.replace("attempt_", ""));
        }
        int i = 0;
        String seqStr = String.valueOf(sequenceNum);
        sb.append(seqStr);

        return sb.toString();
    }

    public String getDisplayString(String[] children) {
        return "id1()";
    }

    @Override
    public void configure(MapredContext context) {
        this.context = context;
    }
}

I am certain this has something to do with Hive-0.13, but not able to see any post related to this error.

Pensioner answered 11/10, 2014 at 0:56 Comment(3)
How did u create Function..?Heathen
did you recompile the UDF against the 2.0 libs?Leap
Are you able to use other UDFs? If it is only this UDF that causes trouble, can you narrow the issue down to a specific line? (for instance by removing lines untill the problem disappears).Jonijonie

© 2022 - 2024 — McMap. All rights reserved.