Kafka Connect source connector's tasks going up and down (RUNNING or missing) frequently

Question

Occasionally with Kafka Connect, I see my JdbcSourceConnector's task go up and down--that is, the REST interface sometimes reports one task that is RUNNING and sometimes reports no tasks (the connector remains RUNNING this whole time). During these times, the task seems to be working when its running. Then, if I delete and re-create the connector, the problem seems to go away. I suspect something is wrong--that tasks shouldn't churn like this, right? But the INFO/WARN logs on the server don't seem to give me many clues, although there are lots of INFO lines to sort through.

• Is it normal for JdbcSourceConnector tasks to oscillate between nonexisting and RUNNING?
• Assuming not, what should I look for in the log to help figure it out? (I see lots of INFO lines)
• Any idea what could be causing this?

I have monitoring on my REST connectors' statuses, and this one gives me the following (where the value is the number of RUNNING statuses; 2 is Connector RUNNING and task RUNNING, but 1 is Connector RUNNING without a task). Today at 9:01 AM I deleted and created the connector, thus "solving" the problem. Any thoughts?

I have Kafka Connect version "5.5.0-ccs" for use with Confluent platform 5.4, running on Openshift with 2 pods. I have 6 separate connectors each with max 1 task, and I typically see 3 connectors with their tasks on one pod and 3 on the other. For the example above, this was the only 1 of the 6 tasks that showed this behavior, but I have seen where 2 or 3 of them are doing it.

Stinson

Can you update your question to include version number and details of topology (how many workers, max tasks, how many connectors, etc) – Gravel

There you go. Let me know if you need more info--not sure about the concept of "workers"--isn't it always one worker per connector--thus I have 6? Or is it one worker per pod (server), thus I have 2? – Stinson

Worker = JVM process, which runs one or more tasks. Connectors are logical config that can run across one or more tasks, which Kafka Connect will distribute across workers based on load and availability. – Gravel

is this behaviour new with 5.5? – Gravel

It is not new. (Note I am running Kafka Connect 5.5 against Confluent Platform 5.4) – Stinson

(Although I only have had monitoring (to get the graph) on 5.5, I noticed similar behavior when on 5.4 just in manually GETting the status from the API ad hoc.) – Stinson