Login/Register
SchedulerStateManagerAdaptor failed to fetch data from zookeeper path in Heron Cluster
Asked Answered
apache-zookeeperschedulerheron
C

0

0

In Heron, I instantiated a SchedulerStateManagerAdaptor named adaptor using IStateManager and created a new Runtime config to saved this adaptor. Then, I created a ISchedulerClient object for invoking its schedulerClient.updateTopology function. Code as follows:

String stateMgrClass = Context.stateManagerClass(this.config); // get state manager instance
IStateManager stateMgr = null;
try {
     stateMgr = ReflectionUtils.newInstance(stateMgrClass);
     FileUtils.writeToFile(filename, "Create IStateManager object success...");
} catch (ClassNotFoundException | InstantiationException | IllegalAccessException e) {
     e.printStackTrace();
}
stateMgr.initialize(this.config);
stateManagerAdaptor = new SchedulerStateManagerAdaptor(stateMgr, 5000);

Config runtime = Config.newBuilder()
                .putAll(primaryRuntime)
                .put(Key.TOPOLOGY_NAME, Context.topologyName(config))
                .put(Key.SCHEDULER_STATE_MANAGER_ADAPTOR, stateManagerAdaptor)
                .build();

if (!schedulerClient.updateTopology(updateTopologyRequest)) {
     throw new TopologyRuntimeManagementException(String.format(
             "Failed to update " + topology.getName() +" with Scheduler, updateTopologyRequest=" + updateTopologyRequest));
}

However, when I tried to adaptor.getPhysicalPlan in scheduler.onUpdate function that was invoked by instantiated scheduler. This Exception happened: enter image description here What's more, I find the Exception in zookeeper log:

2018-07-07 21:01:09,855 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1044] - Closed socket connection for client /218.195.228.24:41348 which had sessionid 0x16469a49d550f23
2018-07-07 21:01:12,001 [myid:] - INFO  [SessionTracker:ZooKeeperServer@358] - Expiring session 0x16469a49d550f29, timeout of 4000ms exceeded
2018-07-07 21:01:12,002 [myid:] - INFO  [SessionTracker:ZooKeeperServer@358] - Expiring session 0x16469a49d550f28, timeout of 4000ms exceeded
2018-07-07 21:01:12,002 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@486] - Processed session termination for sessionid: 0x16469a49d550f29
2018-07-07 21:01:12,003 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@486] - Processed session termination for sessionid: 0x16469a49d550f28
2018-07-07 21:01:20,001 [myid:] - INFO  [SessionTracker:ZooKeeperServer@358] - Expiring session 0x16469a49d550f2a, timeout of 10000ms exceeded
2018-07-07 21:01:20,003 [myid:] - INFO  [SessionTracker:ZooKeeperServer@358] - Expiring session 0x16469a49d550f24, timeout of 10000ms exceeded
2018-07-07 21:01:20,003 [myid:] - INFO  [SessionTracker:ZooKeeperServer@358] - Expiring session 0x16469a49d550f23, timeout of 10000ms exceeded
2018-07-07 21:01:20,003 [myid:] - INFO  [SessionTracker:ZooKeeperServer@358] - Expiring session 0x16474d4e4bb0003, timeout of 10000ms exceeded
2018-07-07 21:01:20,004 [myid:] - INFO  [SessionTracker:ZooKeeperServer@358] - Expiring session 0x16474d4e4bb0004, timeout of 10000ms exceeded
2018-07-07 21:01:20,005 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@486] - Processed session termination for sessionid: 0x16469a49d550f2a
2018-07-07 21:01:20,005 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@486] - Processed session termination for sessionid: 0x16469a49d550f24
2018-07-07 21:01:20,006 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@486] - Processed session termination for sessionid: 0x16469a49d550f23
2018-07-07 21:01:20,006 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@486] - Processed session termination for sessionid: 0x16474d4e4bb0003
2018-07-07 21:01:20,006 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@486] - Processed session termination for sessionid: 0x16474d4e4bb0004
2018-07-07 21:01:23,779 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /218.195.228.24:41378
2018-07-07 21:01:23,779 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@896] - Connection request from old client /218.195.228.24:41378; will be dropped if server is in r-o mode
2018-07-07 21:01:23,779 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@942] - Client attempting to establish new session at /218.195.228.24:41378
2018-07-07 21:01:23,780 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /218.195.228.24:41380
2018-07-07 21:01:23,780 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@896] - Connection request from old client /218.195.228.24:41380; will be dropped if server is in r-o mode
2018-07-07 21:01:23,780 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@942] - Client attempting to establish new session at /218.195.228.24:41380
2018-07-07 21:01:23,783 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /218.195.228.24:41382
2018-07-07 21:01:23,783 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@896] - Connection request from old client /218.195.228.24:41382; will be dropped if server is in r-o mode
2018-07-07 21:01:23,783 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@942] - Client attempting to establish new session at /218.195.228.24:41382
2018-07-07 21:01:23,784 [myid:] - INFO  [SyncThread:0:ZooKeeperServer@687] - Established session 0x16474d4e4bb0005 with negotiated timeout 10000 for client /218.195.228.24:41378
2018-07-07 21:01:23,785 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /218.195.228.24:41384
2018-07-07 21:01:23,785 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@896] - Connection request from old client /218.195.228.24:41384; will be dropped if server is in r-o mode
2018-07-07 21:01:23,785 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@942] - Client attempting to establish new session at /218.195.228.24:41384
2018-07-07 21:01:23,786 [myid:] - INFO  [SyncThread:0:ZooKeeperServer@687] - Established session 0x16474d4e4bb0006 with negotiated timeout 10000 for client /218.195.228.24:41380
2018-07-07 21:01:23,787 [myid:] - INFO  [SyncThread:0:ZooKeeperServer@687] - Established session 0x16474d4e4bb0007 with negotiated timeout 10000 for client /218.195.228.24:41382
2018-07-07 21:01:23,789 [myid:] - INFO  [SyncThread:0:ZooKeeperServer@687] - Established session 0x16474d4e4bb0008 with negotiated timeout 10000 for client /218.195.228.24:41384
2018-07-07 21:01:38,005 [myid:] - INFO  [SessionTracker:ZooKeeperServer@358] - Expiring session 0x16469a49d550f70, timeout of 30000ms exceeded
2018-07-07 21:01:38,005 [myid:] - INFO  [SessionTracker:ZooKeeperServer@358] - Expiring session 0x16469a49d550f71, timeout of 30000ms exceeded
2018-07-07 21:01:38,005 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@486] - Processed session termination for sessionid: 0x16469a49d550f70
2018-07-07 21:01:38,005 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@486] - Processed session termination for sessionid: 0x16469a49d550f71
2018-07-07 21:02:19,897 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /218.195.228.24:41428
2018-07-07 21:02:19,899 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@942] - Client attempting to establish new session at /218.195.228.24:41428
2018-07-07 21:02:19,900 [myid:] - INFO  [SyncThread:0:ZooKeeperServer@687] - Established session 0x16474d4e4bb0009 with negotiated timeout 30000 for client /218.195.228.24:41428
2018-07-07 21:02:26,397 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /218.195.228.24:41436
2018-07-07 21:02:26,398 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@942] - Client attempting to establish new session at /218.195.228.24:41436
2018-07-07 21:02:26,399 [myid:] - INFO  [SyncThread:0:ZooKeeperServer@687] - Established session 0x16474d4e4bb000a with negotiated timeout 10000 for client /218.195.228.24:41436
2018-07-07 21:02:26,403 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@486] - Processed session termination for sessionid: 0x16474d4e4bb000a
2018-07-07 21:02:26,404 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1044] - Closed socket connection for client /218.195.228.24:41436 which had sessionid 0x16474d4e4bb000a
2018-07-07 21:02:29,672 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /218.195.228.43:40088
2018-07-07 21:02:29,675 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@942] - Client attempting to establish new session at /218.195.228.43:40088
2018-07-07 21:02:29,679 [myid:] - INFO  [SyncThread:0:ZooKeeperServer@687] - Established session 0x16474d4e4bb000b with negotiated timeout 10000 for client /218.195.228.43:40088
2018-07-07 21:02:30,480 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /218.195.228.28:57174
2018-07-07 21:02:30,493 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@942] - Client attempting to establish new session at /218.195.228.28:57174
2018-07-07 21:02:30,498 [myid:] - INFO  [SyncThread:0:ZooKeeperServer@687] - Established session 0x16474d4e4bb000c with negotiated timeout 10000 for client /218.195.228.28:57174
2018-07-07 21:02:31,233 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /218.195.228.19:52872
2018-07-07 21:02:31,235 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@942] - Client attempting to establish new session at /218.195.228.19:52872
2018-07-07 21:02:31,239 [myid:] - INFO  [SyncThread:0:ZooKeeperServer@687] - Established session 0x16474d4e4bb000d with negotiated timeout 10000 for client /218.195.228.19:52872
2018-07-07 21:02:34,002 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@486] - Processed session termination for sessionid: 0x16474d4e4bb0009
2018-07-07 21:02:34,009 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1044] - Closed socket connection for client /218.195.228.24:41428 which had sessionid 0x16474d4e4bb0009
2018-07-07 21:02:57,272 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /218.195.228.43:40100
2018-07-07 21:02:57,274 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@368] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x0, likely client has closed socket
    at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
    at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
    at java.lang.Thread.run(Thread.java:748)
2018-07-07 21:02:57,275 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1044] - Closed socket connection for client /218.195.228.43:40100 (no session established for client)
2018-07-07 21:02:57,276 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /218.195.228.43:40102
2018-07-07 21:02:57,277 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@942] - Client attempting to establish new session at /218.195.228.43:40102
2018-07-07 21:02:57,280 [myid:] - INFO  [SyncThread:0:ZooKeeperServer@687] - Established session 0x16474d4e4bb000e with negotiated timeout 10000 for client /218.195.228.43:40102
2018-07-07 21:02:57,357 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /218.195.228.43:40106
2018-07-07 21:02:57,358 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@896] - Connection request from old client /218.195.228.43:40106; will be dropped if server is in r-o mode
2018-07-07 21:02:57,358 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@942] - Client attempting to establish new session at /218.195.228.43:40106
2018-07-07 21:02:57,362 [myid:] - INFO  [SyncThread:0:ZooKeeperServer@687] - Established session 0x16474d4e4bb000f with negotiated timeout 30000 for client /218.195.228.43:40106
2018-07-07 21:02:58,936 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /218.195.228.43:40110
2018-07-07 21:02:58,938 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@942] - Client attempting to establish new session at /218.195.228.43:40110
2018-07-07 21:02:58,943 [myid:] - INFO  [SyncThread:0:ZooKeeperServer@687] - Established session 0x16474d4e4bb0010 with negotiated timeout 30000 for client /218.195.228.43:40110
2018-07-07 21:03:05,562 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /218.195.228.19:52884
2018-07-07 21:03:05,564 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@368] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x0, likely client has closed socket
    at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
    at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
    at java.lang.Thread.run(Thread.java:748)
2018-07-07 21:03:05,565 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1044] - Closed socket connection for client /218.195.228.19:52884 (no session established for client)
2018-07-07 21:03:05,568 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /218.195.228.19:52886
2018-07-07 21:03:05,569 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@942] - Client attempting to establish new session at /218.195.228.19:52886
2018-07-07 21:03:05,575 [myid:] - INFO  [SyncThread:0:ZooKeeperServer@687] - Established session 0x16474d4e4bb0011 with negotiated timeout 10000 for client /218.195.228.19:52886
2018-07-07 21:03:05,794 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /218.195.228.19:52888
2018-07-07 21:03:05,795 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@896] - Connection request from old client /218.195.228.19:52888; will be dropped if server is in r-o mode
2018-07-07 21:03:05,796 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@942] - Client attempting to establish new session at /218.195.228.19:52888
2018-07-07 21:03:05,798 [myid:] - INFO  [SyncThread:0:ZooKeeperServer@687] - Established session 0x16474d4e4bb0012 with negotiated timeout 30000 for client /218.195.228.19:52888
2018-07-07 21:03:43,958 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /218.195.228.24:41544
2018-07-07 21:03:43,960 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@942] - Client attempting to establish new session at /218.195.228.24:41544
2018-07-07 21:03:43,984 [myid:] - INFO  [SyncThread:0:ZooKeeperServer@687] - Established session 0x16474d4e4bb0013 with negotiated timeout 30000 for client /218.195.228.24:41544
2018-07-07 21:03:44,041 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@486] - Processed session termination for sessionid: 0x16474d4e4bb0013
2018-07-07 21:03:44,043 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1044] - Closed socket connection for client /218.195.228.24:41544 which had sessionid 0x16474d4e4bb0013
2018-07-07 21:03:53,413 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /218.195.228.28:57186
2018-07-07 21:03:53,415 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@368] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x0, likely client has closed socket
    at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
    at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
    at java.lang.Thread.run(Thread.java:748)

I don't know what's wrong with it! But when I invoke the adaptor.getPhysicalPlan in the file that created adaptor object, it worked. What reason caused this problem? Thanks for your help.

Colligate answered 5/7, 2018 at 9:45 Comment(4)
Does anyone know how to solve this problem? Or what is the cause of the problem? – Colligate
I just saw your comment on the zookeeper connection error post and I'm afraid I'll need to try this out before I can say anything with confidence. However, I have a feeling that the issue might be related to the NullPointerException that you're getting in your code. Try going through the exception chaining and see if you can fix the NullPointerException? I'll also try and dive into this. – Lakeshialakey
Thanks for your reply. I am now suspecting that the NullPointerException may be caused by Zookeeper's EndOfStreamException. So I am trying to solve the zookeeper problem first. This is a link to the question where you can see my zookeeper configuration and full zookeeper log content. I am sorry for not providing a separate description of the zookeeper problem, you can only pay attention to the content of zookeeper. – Colligate
I'll have a look at the zookeeper problem but yes, I agree that it could be either way i.e. the NullPointerException could be causing the zookeeper error or vice versa. Try attacking this from both angles and see if this works. Please do update your post when you fix it... :) – Lakeshialakey

© 2022 - 2024 — McMap. All rights reserved.