Jenkins slave jobs failing on "Unexpected termination of channel"

S

10

52

I am currently seeing a set of errors across my builds.

Is this expected behaviour if you loose Jenkins (say to a box crash, or a kill -9)?

Or is there something worse going on (like a bad network connection)?

The stack and error is:

hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:158)
        at $Proxy175.join(Unknown Source)   at
hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:861)     at
hudson.Launcher$ProcStarter.join(Launcher.java:345)     at
hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:82)
        at
hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
        at
hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:703)
        at hudson.model.Build$RunnerImpl.build(Build.java:178)  at
hudson.model.Build$RunnerImpl.doRun(Build.java:139)     at
hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:473)
        at hudson.model.Run.run(Run.java:1410)  at
hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)     at
hudson.model.ResourceController.execute(ResourceController.java:88)
        at hudson.model.Executor.run(Executor.java:238) Caused by:
hudson.remoting.RequestAbortedException: java.io.IOException:
Unexpected termination of the channel   at
hudson.remoting.Request.abort(Request.java:273)     at
hudson.remoting.Channel.terminate(Channel.java:732)     at
hudson.remoting.Channel$ReaderThread.run(Channel.java:1157) Caused by:
java.io.IOException: Unexpected termination of the channel  at
hudson.remoting.Channel$ReaderThread.run(Channel.java:1133) Caused by:
java.io.EOFException    at
java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2554)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
        at hudson.remoting.Channel$ReaderThread.run(Channel.java:1127)

Sap answered 3/4, 2012 at 17:25 Comment(2)

What version of Jenkins are you using? – Barbados 4/4, 2012 at 14:11

Just putting it out here, these kinds of issues are due to network. Since my jenkins server and slave are in the same network, I edited my /etc/hosts file and added DNS entry for jenkins server with private IP. That solved the issue for me. Also, I tried to equate the java servers on both my master and slave. restarted both master and slave. – Stricture 10/6, 2022 at 13:29

P

49

You'll see that error if the Jenkins master loses connectivity with the slave. It could be due to any of the three issues you listed above:

Manually killing the slave process
The slave server becoming unavailable
A network problem between the master and the slave

Panada answered 3/4, 2012 at 17:41 Comment(2)

If it's simply slow response from the slave, try turning off the response time check as a temporary fix.Go to Manage Jenkins > Manage Nodes > Configure, and uncheck the "Response time" box. That will get you through until you can give it more resources if it's just a slow slave. – Fateful 19/6, 2013 at 19:37

@gareth_bowles, How can we say it as a network problem between the master and the slave – Ferretti 30/8, 2017 at 12:8

I

9

I resolved it by a simple restart of the master Jenkins server.

Indore answered 30/4, 2015 at 14:32 Comment(2)

Don't know why someone downvoted you. This was that solution for me. groups.google.com/forum/#!topic/jenkinsci-users/ZsNnfs5bEAs – Limestone 4/4, 2016 at 4:13

This also worked for me, jenkins 2.289.3 with nodes running java 8 – Eliason 28/9, 2021 at 7:45

S

5

Since 1.520, Jenkins requires Java 6 or later, one of the possibility of this error is that you dont have java 6 or later on slave/remote. Resolution is to update your remote machine with java 6 or later or set correct environment variable which SSH uses while it logs in into slave machine.

Selfopinionated answered 13/8, 2013 at 5:42 Comment(2)

Dear Halim, you just made my day. – Copacetic 15/5, 2014 at 9:4

Thanks. I was going to leave a note about this but you had the answer. I switched from JDK 6 to 7 which solved the issue. The troubleshooting steps were to look at the Jenkins slave log (on the jenkins master server) which indicated class not found errors which led me down the path of trying a different jdk. – Brothers 14/2, 2017 at 15:51

S

4

Another thing where you can see this error (which actually leads to this) is lack of memory/cpu cores for build. I spent some time testing it, and resolved this issue on our site by increasing java heap size (and as well adding more memory to machines), increasing number of cores (so one machine have at least 4G of RAM with two cores available with one build slot), and also do some tuning (if you are on linux) number of open file descriptors and processes which can be run concurrently.

Spurge answered 26/4, 2012 at 10:16 Comment(0)

P

3

For me the node configuration had a wrong "Remote FS Root". So check if all the config parameters on the master for this slave node are correct. Jenkins doesn't complain a thing but terminates the connection.

I struggled a lot before determining it was the node config. At least the error message on the master should be more explicit or informative.

Paisley answered 13/6, 2013 at 13:48 Comment(1)

is there a way to determine if the Remote FS Root is correct? – Inspiration 29/5, 2018 at 15:9

P

3

The version of java has to be 1.6 (latest) or newer for Jenkins 1.531 that I am using.

For one of my slave nodes I found that I had to set the Java path in the 'Launch method' Advanced button 'JavaPath' for the node before this would work correctly. The issue was that the node kept using an older version of Java which I was not able to upgrade.

Phenylketonuria answered 29/11, 2013 at 10:26 Comment(0)

M

2

IF you are using aws ELB for jenkins master and jenkins slave communication ELB Idle time is causing this issue. Please go and update your idle time using http://docs.aws.amazon.com/elasticloadbalancing/latest/classic/config-idle-timeout.html#config-idle-timeout-console

Set it to more than 60 seconds

Margetts answered 30/11, 2017 at 13:11 Comment(0)

G

1

I know this question is kind of old but I came across this problem the other day.

For me the problem was occurring during the install of the JDK. The install failed because the security settings of Windows were set so the notification box that says: "You need to give permission for this operation" was popping up, causing the install to fail.

I changed the notification settings to "Never notify" and it solved this issue.

Gheber answered 7/8, 2013 at 21:2 Comment(0)

C

1

I had similar issues with Jenkins Slaves terminating when I upgraded Jenkins to version 2.9.2 from 2.1.5.

All issues related to upgrade like unexpected termination of Slave fixed by installing Java 8 version on Centos.

yum install java-1.8.0-openjdk-devel

Cineraria answered 28/11, 2017 at 23:56 Comment(0)

H

-1

On CentOS Linux 7, I found that the Jenkins slave doesn't work with OpenJDK. I tried Java 7 and 8. It requires Oracle Java.

So, if you're on Linux, install Oracle Java, and set the path to point to it. You can do this in a number of ways, e.g.:

Set $JAVA_HOME for the user you're launching the slave as, and add it to $PATH.
Use update-alternatives to set the default java.
Set JavaPath, under the node's Advanced settings in Jenkins.

Hawkie answered 17/11, 2017 at 18:23 Comment(0)

Recommended topics

Hot tags