How to access a Zookeeper ensemble as a 'super' user via Zookeeper shell?

Question

Per the zookeeper docs, one can create a way to access the a Zookeeper ensemble as a "super" user. The directions for doing that are a little vague, both in terms of configuration and connection method. It does subtly suggest that this can only be done via a Java connection / API.

How do you connect to a Zookeeper ensemble as a "super" user via it's own CLI tools?

Luteolin

Answer 1

This tip totally worked! Thank you so much, Etienne! I was also able to try an alternative, skipACL=yes, which is another way of bypassing security, but that method is incredibly dangerous to use in the middle of the day. Here are my details:

• first of all, it's not skipACL=true, it's skipACL=yes, even though the specific wording of the boolean value is not mentioned in the stupid ZK docs page. Those docs are not just "a little vague"; I would call them quite weak in places.
• second of all, if you turn this on (and carefully restart all three ZK nodes, one at a time), it starts completely ignoring all ACLs. So sure, you can fix that one problem by doing a setAcl on that node you don't have permissions to... but if your cluster is live in the middle of the day, then all OTHER operations happening are ignoring ACLs entirely. And that includes if a component (like Hive) creates a brand-new chunk of ZK tree space -- the permissions on the parent node do not trickle down! So then after you turn skipACL back off (i.e., back to normal enforcement), you have random pieces of ZK that have the wrong permissions. Dang it.

So this method (superDigest) is far, far safer, since it allows you to be a super-user in just one ZK CLI connection, doing one little operation, while continuing to enforce normal ZK security rules in all other connections happening in real time in the rest of the cluster. I wish this were much easier to do, and I wish there were a way to perform the repairs with the built-in zkCli client that you already have, rather than downloading a separate client.

Also note that this special property zookeeper.DigestAuthenticationProvider.superDigest is only settable as a Java System Property Only, i.e., through the -D environment variable, as Etienne demonstrates here. Even though you can put skipACL into zoo.cfg, superDigest is only passable through the Java env variable -- don't ask me why. This detail is (not very clearly) mentioned on the ZK docs page.

Again, thank you, Etienne, for putting up these brief instructions, and thank you to Shlomi Noach for compiling a Go-based program with absolutely no dependencies, the pre-built binary of which I could download and use in the middle of a crisis. You two are generous and deserve a gold star today.

Clique

Thank you so much for your answer, I was struggling with the HMaster failure with NoAuth error past days, setting skipACL=yes correctly helped me. I had set it to true which made no effect. – Ironware

this saved my week ! always a property is set to true/false, but here it is set to yes..... and hardly written anywhere in the docs – Fosdick

@Jeff, where to set that SERVER_JVMFLAGS while we are in windows, any idea ? – Ketty

Answer 2

Zookeeper doc is indeed very vague...

Here how I did it

cd your_zookeeper_home
java -cp "./zookeeper-3.4.6.jar:./lib/slf4j-api-1.6.1.jar" org.apache.zookeeper.server.auth.DigestAuthenticationProvider super:password 

Insert your password instead of "password"

You will get a line like this :

super:password->super:DyNYQEQvajljsxlhf5uS4PJ9R28= 

Export your var or edit zkServer.sh :

SERVER_JVMFLAGS=-Dzookeeper.DigestAuthenticationProvider.superDigest=super:DyNYQEQvajljsxlhf5uS4PJ9R28=

restart zookeeper and you will be able to connect with super (I use this client https://github.com/outbrain/zookeepercli)

./zookeepercli --servers your_server  --auth_usr "super" --auth_pwd "password"  -c ls

Luciferase

This looks like it'd work. I don't have cluster to test against ATM, but the next time I do I'll verify this solution. Thanks! – Luteolin

In Windows , Use ; (semicolon) instead of : (colon) while executing that java command java -cp "./lib/zookeeper-3.6.3.jar;./lib/slf4j-api-1.7.25.jar" org.apache.zookeeper.server.auth.DigestAuthenticationProvider zookeeper:adminpassword – Ketty

@Etienne, where to set that SERVER_JVMFLAGS while in Windows environment, i cannot relate that zkserver.cmd as it has no relation with JVMFLAGS – Ketty

@Ketty not really familiar with Windows but you should be able de to do it with the set command : learn.microsoft.com/en-us/windows-server/administration/… – Luciferase