datastax opscenter agents not installing

Question

I'm running my own version of cassandra on 127.0.1.1. I changed the rpc_address and also the address to 127.0.1.1.

When I'm starting Opscenter I'm prompted to install agents and I'm doing so by the recommended Fix now option. But when I try to install it asks me for some Node ssh credentials. I don't know what this means.

What are the correct credentials to input here?

I tried adding a new user on Linux with root permissions and tried with that username too but it doesn't work. I manually tried to run the install_agent.sh too but it doesn't work.

Am I missing something?

EDIT:

address.yaml

stomp_interface: 127.0.1.1
agent_rpc_interface: "127.0.1.1"
cassandra-conf: /home/$username/Dropbox/Work/ITNow/olderVersions/cassandra2.11/apache-cassandra-2.1.12/conf/cassandra.yaml
stomp_port: 61620
jmx_host: 127.0.1.1
jmx_port: 7199

cassandra.yaml

http://textuploader.com/57ivn

EDIT2:

for username and password I'm using my datastax credentials. For the Private key, I'm using the key in /etc/ssh/ssh_host_rsa_key.pub as mentioned by @apesa. But still it can't install the agents.

Is my address.yaml file correct?

EDIT 3:

Like you can see in the following image, when thought Opscenter is not connected to the agents, I can still see the keyspaces and tables in cassandra.

For sudo netstat -p | grep 127.0.1.1 I get no output.

xyz@ubuntu$ ps -ef | grep datastax-agent
xyz@ubuntu$ ps -ef | grep cassandra

the output is this and this respectively. In short, yes I can see jars and I guess the environment variables too.

How did I start cassandra, you ask?

sudo ./bin/cassandra from the folder of cassandra. After that I'm using spark streaming to stream data from kafka to cassandra. But it shouldn't matter here.

Even if it is not connecting to the agent the opscenter should still show the cassandra instance. Do you see your test_Cluster? Yes! I do!

Finally, nodetool status gives me this:

Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address    Load       Tokens  Owns    Host ID                               Rack
UN  127.0.1.1  58.94 MB   256     ?       367dd1c6-291d-4e57-a2b6-e04d74154a6f  rack1

Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless

What I believe:

I am not sure, but maybe my address.yaml is not correct? I have updated my address.yaml above, maybe I need to change something? Changing the address.yaml gives me different outputs when I run ./datastax-agent. Thanks.

EDIT4:

Tried everything what @apesa suggested and here's the detailed outputs of the scripts. I still have the same problem.

version:

Opscenter: 5.2.4
datastax-agent: 5.2.4
Cassandra: 2.1.12

Output of sudo netstat -p | grep :9042

tcp        0      0 localhost:38155         ganguly:9042            ESTABLISHED 15907/python2.7 
tcp        0      0 localhost:38154         ganguly:9042            ESTABLISHED 15907/python2.7 
tcp        0      0 localhost:38153         ganguly:9042            ESTABLISHED 15907/python2.7 
tcp6       0      0 localhost:38176         ganguly:9042            ESTABLISHED 15951/java      
tcp6       0      0 localhost:38179         ganguly:9042            ESTABLISHED 15951/java      
tcp6       0      0 ganguly:9042            localhost:38154         ESTABLISHED 15720/java      
tcp6       0      0 ganguly:9042            localhost:38153         ESTABLISHED 15720/java      
tcp6       0      0 ganguly:9042            localhost:38179         ESTABLISHED 15720/java      
tcp6       0      0 ganguly:9042            localhost:38176         ESTABLISHED 15720/java      
tcp6       0      0 ganguly:9042            localhost:38155         ESTABLISHED 15720/java  

Output of sudo netstat -p | grep :7199

blank

Output of sudo netstat -p | grep :61620

blank

This is the console log on running sudo ./datastax-agent.

In short, this is the error line:

ERROR [clojure-agent-send-off-pool-0] 2016-02-04 11:33:26,274 Can't \
connect to Cassandra (All host(s) tried for query failed (tried:\
 /127.0.0.1:9042 \
(com.datastax.driver.core.TransportException: \
[/127.0.0.1:9042] Cannot connect))), retrying soon.

But I don't know why it's trying to connect to cassandra on 127.0.0.1 since cassandra is running on 127.0.1.1. What did I do wrong here?

Germanophobe

After looking at your cassandra.yaml you should change the seed provider to 127.0.1.1 so it is the same as the listen and rpc addresses. Also you may need to change the stomp interface in address.yaml to 127.0.1.1 as well. You should also look in the agent folder for the logs. – Finalize

Tried that now, still the same problem. :/ – Germanophobe

Are you sure cassandra is running? Also, what do the agent.log and opscenterd.log say? If you were trying to start the agent it will have logged the failure. – Finalize

@Finalize agent.log - textuploader.com/57vle and opscenterd.log - textuploader.com/57vcn. and yes I'm sure cassandra is running. – Germanophobe

In the agent.log I can see that it says Can't connect to Cassandra (All host(s) tried for query failed (tried: /127.0.0.1:9042.. but it should try 127.0.1.1 because I changed my rpc_address to this in cassandra.yaml. How to fix this? – Germanophobe

Thats your problem. When you changed your address did you restart everything? If your listen, rpc and seeds addresses all point to 127.0.1.1 and the stomp interface also points to 127.0.1.1 you should be fine. There are no other places to configure addressing. – Finalize

Yes I restarted everything! Do I need to do anything more than sudo ./usr/share/opscenter/bin/start_opscenter ? – Germanophobe

@Finalize I'm trying to manually installing using the ./usr/share/opscenter/agent/bin/install_agent.sh 127.0.1.1 but I still can't connect to my cassandra node. – Germanophobe

You have to start the agent from the agents bin directory. like xyz@ubuntu$ sudo ./datastax-agent – Finalize

still not working, look at my present agen.log - textuploader.com/57vxq – Germanophobe

my question is what ip is opscenter running on? is opscenter also on 127.0.1.1 or is opscenter on localhost? The ip address in the stomp_interface variable should be that of opscenter, not cassandra. – Crosscut

@Crosscut I'm running opscenter here 127.0.1.1:8888/opscenter but funnily it's also running on 127.0.0.1:8888/opscenter. – Germanophobe

Answer 1

There are a couple ways to install the DataStax agent from OpsCenter. Based on your IP Addr of 127.0.1.1 you must be running a Debian version of Linux. I know it says "optional" but I have never been able to run without a ssh private key. In which case you should install OpenSSH-server using the following command

xyz@ubuntu$ sudo apt-get install openssh-server

Then follow these instructions for setting up a SSH key that you will then use when providing your credentials to the "fix agent" dialog box. The username/password in Datastax for the DSE side is datastax/datastax unless you changed it. That is what you should put in for username / password and then also copy/paste your new SSH key inside the private key box. The ssh key will be found in

/etc/ssh/ssh_host_rsa_key.pub

Once you have all the credentials together go ahead and run the Install Agents tool. If it still fails to complete you should check the datastax-agent folder where you installed DSE and verify there is a jar file for the agent.. It should be something like

datastax-agent-5.2.4-standalone.jar

You should also see the folder structure for the Agent as well, bin, conf... etc directories should all be there. If everything is in place and your cassandra.yaml file is correct you should be able to get the agent running.

You can also do this manually by moving the correct directories and jar files in place and then restarting the agent(s).

EDIT:

After looking at your latest agent.log file I would say this. The broken pipe error in Java refers to a connection that is no longer up or unavailable. Where it is failing in the agent.log is where it is trying to communicate with the opscenter. One of your connections is not up or is mis configured. You also have another log for cassandra that is usually in /var/log/cassandra depending on how you installed cassandra. Comb through that and make sure it does not have errors in it during start up, warnings are fine but errors usually point to the problem.

From this point on you have the the following log files. Note the file locations may be different than yours depending on how you installed everything.

/var/log/cassandra
/var/log/datastax-agent
/var/log/OpsCenter

and you have the following commands to help sort out what is actually running and on what port is it bound. Take a look below, what do these commands tell you?

xyz@ubuntu$ sudo netstat -p | grep 127.0.1.1

You should see at least the following ports bound to your 127.0.1.1 addr: 9042, 8888, 61620 and several others.

Also look at the output of these two commands. If you don't get anything back try running them as sudo. The output will depend on how you started the jars

xyz@ubuntu$ ps -ef | grep datastax-agent
xyz@ubuntu$ ps -ef | grep cassandra

You should see all the loaded jar files as well as the environment variables that were set when starting cassandra and / or datastax-agent.

You might edit your post and explain how you started cassandra. ie as a service or from the command line?

Also, from OpsCenter you should be able to see the test_cluster at 127.0.1.1:8888. If your prompted to fix the agents just exit out of those dialog boxes until you're at the main screen showing the datacenter. Even if it is not connecting to the agent the opscenter should still show the cassandra instance. Do you see your test_Cluster?

Lastly, what do you get when you type this

xyz@ubuntu$ nodetool status

EDIT 2:

Based on your last edit it looks as if everything but the agent is up and running. So says nodetool, which is good. Also since you can see Keyspaces in OpsCenter you are definitely connected to a running cassandra instance.

So looking further it appears your datastax-agent is still not connecting. In looking at your address.yaml file I can see a couple possible issues. This is a simple standalone cassandra cluster with a single IP address of 127.0.1.1. So that 127.0.1.1 address is going to be used in all the config where an IP addr or host is involved. I would make the following changes to your address.yaml file.

stomp_interface: 127.0.1.1
cassandra-conf: /home/$username/Dropbox/Work/ITNow/olderVersions/cassandra2.11/apache-cassandra-2.1.12/conf/cassandra.yaml
stomp_port: 61620

Leave the JMX details out since they are not needed and will be sent by the opscenterd once connected. The agent_rpc_interface is also not needed in this install.

There is also cassandra-env.sh script that is found in the cassandra/conf dir. this script is executed when cassandra starts. look for the following line near the bottom of the script and uncomment it and add your host IP 127.0.1.1 in place of <public name>

# jmx: metrics and administration interface
#
# add this if you're having trouble connecting:
JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=127.0.1.1"

Lastly, there is the cassandra utility documented HERE You can pass several cli parameters to the cassandra shell when initializing. This particular one below is useful when you want to start/restart a cassandra cluster after changing the IP or hostname. You will need to restart cassandra once you have made these changes. In fact you shoudl restart everything in this order.. Cassandra, datastax-agent and then opscenter.

xyz@ubuntu$ bin/cassandra -Dcassandra.load_ring_state=false

Once you complete the changes and try starting the agent and it still fails, you are going to need more debugging details. Netstat returned false because of what we piped to the grep utility. try the following as the ports are probably bound to your hostname since your using 127.0.1.1. Run it for the following ports :9042, :7199 and :61620. You're looking for your hostname bound to all of these ports while cassandra is running.

xyz@ubuntu$ sudo netstat -p | grep :9042

Please also edit your question and add the versions of OpsCenter and Datastax-Agent your running. That matters.

Finalize

"check the datastax-agent folder where you installed DSE". I am not working with DSE. I am running the apache version of cassandra. I tried everything you mentioned and still it doesn't let me connect. Do I need to place the datastax-agent-5.2.4-standalone.jar somewhere specific? Right now it's in /usr/share/opscenter/agent/datastax-agent-5.2.4-standalone.jar – Germanophobe

I have also added my cassandra.yaml and address.yaml for better tracing. – Germanophobe

Please check my latest edit for the yaml codes. Thanks. – Germanophobe

hi! thanks for your comment, please check my latest edit, I've tried to answer all your queries. – Germanophobe

do you have any idea what to do next? – Germanophobe

I do, There are some environment variables and .sh scripts that are run during start up. It looks like you have some previously cached env variables that are keeping your agent from binding to the correct address:port. Give me a couple hours to get into my office and ill put it into an EDIT in my answer. I had the same issue the first time I changed from localhost to a static IP. – Finalize

I tried everything you mentioned, and I still have the same problem. :/ I have edited my question with the details, could you please check? – Germanophobe

It's trying to connect to cassandra on 127.0.0.1 now. that's what it says on the error. – Germanophobe

netstat says you have no stomp_connection and no JMX connection established. That is your broken pipe error.. Also I just re-read your cassandra.yaml file and noticed you have your seed provider set to 127.0.0.1, that should be set to 127.0.1.1 . If the seed provider is still set to localhost change it to 127.0.1.1 and restart cassandra and then run nodetool status and make sure cassandra is running, then run netstat on those 3 ports above. This should not be that difficult, – Finalize

hi! Sorry I didn't update my cassandra.yaml but at the time of posting the netstat outputs, I had changed the seed provided to 127.0.1.1. it looks like this now textuploader.com/57ivn – Germanophobe

Are you really using private key? I see that your EDIT2 is actually mentioning public key – Undertaking

@Undertaking well I was asked to use the key in /etc/ssh/ssh_host_rsa_key.pub and that's what I'm using. Do I need to use some other key? – Germanophobe

@Germanophobe 1. Do you really have SSH server running locally? If not forget about this. – Undertaking

@Germanophobe The OpenSSH install was for one reason and that is to allow OpsCenter to secure shell to the host and allow the installation of the Datastax-agent via OpsCenter. That is the only reason to have OpenSSH running. You already have the agent installed, you are having trouble getting it started. – Finalize

@Finalize I already have OpenSSH installed as you've advised. I also have the key from the file you mentioned. Anything that I'm missing here? – Germanophobe

@Undertaking yes I have OpenSSH-server installed already. Look at this answer, I followed the steps advised. – Germanophobe

@Germanophobe Can you try using private key of the user? – Undertaking

@Undertaking check this textuploader.com/52ai6. Check the error. This is the log for ./datastax-agent so the problem is not with the key. The problem is Exception running Jetty: java.net.BindException: Address already in use. I already have jetty running because I'm running spark monitoring UI jaceklaskowski.gitbooks.io/mastering-apache-spark/content/… which uses jetty. I don't know if this is the problem, and whether I should run the jetty instance for datastax-agent on a different port but maybe this log is a hint. – Germanophobe

Answer 2

You could try editing the address.yaml file of the datastax-agent. It should be located in /var/lib/datastax-agent/conf/. In this file you would need to add the following line: stomp_interface: Also make sure the stomp_port is set in the same file.

Reference : OpsCenter 5.0 User Guide

Sabiha

Hydrotherapy

Please check my latest edit for the yaml codes. It still doesn't work after editing my address.yaml. – Germanophobe