WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect

Question

I have two nodes that each one has docker with Mesos,marathon,and zookeeper have been installed on it. This is my docker compose file on master node:

version: '3.7'
 services:
  zookeeper:
   image: ubuntu_mesos_home_marzieh
   command: /home/zookeeper-3.4.8/bin/zkServer.sh restart
  environment:
    ZOOKEEPER_SERVER_ID: 1
    ZOOKEEPER_CLIENT_PORT: 2190
    ZOOKEEPER_TICK_TIME: 2000
    ZOOKEEPER_INIT_LIMIT: 10
    ZOOKEEPER_SYNC_LIMIT: 5
    ZOOKEEPER_SERVERS: 150.20.11.133:2888:3888;150.20.11.136:2888:3888
  network_mode: host
  expose:
    - 2190
    - 2888
    - 3888
  ports:
    - 2190:2190
    - 2888:2888
    - 3888:3888

 master-mesos:
  image: ubuntu_mesos_home_marzieh
  command: bash -c "sleep 30; /home/mesos-1.7.0/build/bin/mesos- 
    master.sh --ip=150.20.11.133 --work_dir=/var/run/mesos 
    --hostname=150.20.11.133" 
  privileged: true
  restart: always
  network_mode: host
  depends_on:
    - zookeeper
  environment:
    - LIBPROCESS_IP=150.20.11.133
    - MESOS_HOSTNAME="150.20.11.133"
    - MESOS_QUORUM=1
    - MESOS_LOG_DIR=/var/log/mesos
    - MESOS_WORK_DIR=/var/run/mesos
  expose:
    - 5050
    - 4040
    - 7077
    - 8080
  ports:
    - 5050:5050
    - 4040:4040
    - 7077:7077
    - 8080:8080

 marathon:
  image: ubuntu_mesos_home_marzieh
  command: bash -c "sleep 60; /usr/share/marathon/bin/marathon"
  privileged: true
  network_mode: host
  depends_on:
    - zookeeper
    - master-mesos
  environment:
    - HOSTNAME="150.20.11.133"
    - MARATHON_ZK=zk://150.20.11.133:2190/marathon
    - MARATHON_MASTER=zk://150.20.11.133:2190/mesos
    - MESOS_NATIVE_JAVA_LIBRARY=/usr/local/lib/libmesos.so
    - MARATHON_HTTP_PORT=8080
  expose:
    - 8080
    - 2190
  ports:
    - 8080:8080
    - 2190:2190

This is docker compose for slave node:

 version: '3.7'
  services:
   zookeeper:
     image: ubuntu_mesos_home_marzieh
     command: /home/zookeeper-3.4.8/bin/zkServer.sh start
     environment:
       ZOOKEEPER_SERVER_ID: 2
       ZOOKEEPER_CLIENT_PORT: 2190
       ZOOKEEPER_TICK_TIME: 2000
       ZOOKEEPER_INIT_LIMIT: 10
       ZOOKEEPER_SYNC_LIMIT: 5
       ZOOKEEPER_SERVERS: 
       150.20.11.133:2888:3888;150.20.11.136:2888:3888
    network_mode: host
    expose:
     - 2190
     - 2888
     - 3888
    ports:
     - 2190:2190
     - 2888:2888
     - 3888:3888

  slave-mesos:
     image: ubuntu_mesos_home_marzieh
     command: bash -c "sleep 40; /home/mesos-1.7.0/build/bin/mesos- 
     slave.sh --master=150.20.11.133:5050 --work_dir=/var/run/mesos
     --systemd_enable_support=false"
     privileged: true
     restart: always
     privileged: true
     network_mode: host
     depends_on:
      - zookeeper
     environment:
       - MESOS_RESOURCES=ports(*):[11000-11999]
       - LIBPROCESS_IP=150.20.11.136
       - MESOS_HOSTNAME="150.20.11.136"
       - MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins 
       - MESOS_LOG_DIR=/var/log/mesos
       - MESOS_WORK_DIR=/var/run/mesos
       - MESOS_LOGGING_LEVEL=INFO
     expose:
       - 5051
     ports:
       - 5051:5051

   marathon:
    image: ubuntu_mesos_home_marzieh
    command: bash -c "sleep 60; /usr/share/marathon/bin/marathon"
    privileged: true
    network_mode: host
    depends_on:
      - zookeeper
      - slave-mesos
    environment:
      - MARATHON_ZK=zk://150.20.11.133:2190/marathon
      - MARATHON_MASTER=zk://150.20.11.133:2190/mesos
      - MESOS_NATIVE_JAVA_LIBRARY=/usr/local/lib/libmesos.so
      - MARATHON_HTTP_PORT=8080
    expose:
      - 8080
      - 2190
    ports:
      - 8080:8080
      - 2190:2190

I configured Zookeeper on "zoo.cfg" and made "myid" file for every node. Also I configured Mesos and Marathon. But when I ran docker compose on master node (150.20.11.133), I got this error:

marathon_1 | java.net.ConnectException: Connection refused marathon_1 | at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

marathon_1 | at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) marathon_1 | at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)

marathon_1 | at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144) marathon_1 | [2019-02-16 13:38:14,087] INFO Opening socket connection to server 172.28.10.133/172.28.10.133:2190. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn:main-SendThread(172.28.10.133:2190))

marathon_1 | [2019-02-16 13:38:14,087] WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn:main-SendThread(172.28.10.133:2190))

marathon_1 | java.net.ConnectException: Connection refused marathon_1 | at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

marathon_1 | at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)

marathon_1 | at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)

marathon_1 | at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144) marathon_1 | [2019-02-16 13:38:14,173] ERROR Failed to connect to ZK. Marathon will exit now. (mesosphere.marathon.core.storage.store.impl.zk.RichCuratorFramework:main)

I don't know what happen. I tried to open every port with ufw or I disabled firewall, but error did not go away. Would you please tell me what is wrong with docker compose file?

Any help would be appreciated.

Thanks.

Xanthous

Answer 1

I just restarted "zookeeper" and everything went fine.

./ZOOKEEPER_HOME/bin/zkServer.sh restart

Xanthous

Thanks for the answer. But if someone like me is here and running inside a docker container. docker-compose restart will fix the issue. at least worked for me. – Eulau

@Eulau I used a variation of your command. docker restart <container-id> and it worked – Radian

Answer 2

"server null" indicates that no value was received for the server parameters while running your ZK CLI.
See if you can override your zookeeper server and port from what's specified in the config file by passing it through the command line.
./zkCli.sh -server hostname:port

Ursine