Login/Register
Zookeeper: java.io.IOException: No snapshot found, but there are log entries. Something is broken
Asked Answered
apache-kafkaapache-zookeeper
P

12

49

I have been working with Kafka 2.4.0 (2.11) and yesterday I had to forcefully terminate the process for some unknown reason. Since then I haven't been unable to start Zookeeper due to the following error:

[2020-01-11 11:12:43,783] ERROR Unexpected exception, exiting abnormally (org.apache.zookeeper.server.ZooKeeperServerMain)
java.io.IOException: No snapshot found, but there are log entries. Something is broken!
    at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:222)
    at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:240)
    at org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:290)
    at org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:450)
    at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:764)
    at org.apache.zookeeper.server.ServerCnxnFactory.startup(ServerCnxnFactory.java:98)
    at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:144)
    at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106)
    at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64)
    at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128)
    at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)

And as soon as I searched for this problem I found issue ZOOKEEPER-3513 reported, which may or may not explain the problem. However, what I'm finding strange is that if I delete the Kafka/Zookeeper directory and download it again from scratch, the problem persists. Does anyone know how I can solve this?

Thank you for your help

Photocopier answered 11/1, 2020 at 11:46 Comment(4)
You shouldn't need to re-download anything. There's no state stored outside of those data directories – Pyromorphite
what about the directory /tmp/zookeeper ? – Labionasal
Isn't that the dataDir? Besides, you should not store data on /tmp anyway – Pyromorphite
have you tried the workaround with snapshot.0 file described in issues.apache.org/jira/browse/ZOOKEEPER-3056? – Aerator
V
88

Check for the tmp/zookeeper folder on the drive where you have kafka folder (lets say D:/), and delete the folder tmp, which will create automatically for you once run the zookeeper again.

Villada answered 21/1, 2020 at 11:28 Comment(2)
this works for me, in Linux I used: rm -rf /tmp/zookeeper/* – Imes
I also had to delete broker files for this to work. deleted broker files from /tmp folder. – Subtend
U
14

On windows ->

Go to the tmp folder where the zookeeper details are stored and delete the existing log files

Directory path = d:\tmp\zookeeper\version-2

On Linux ->

Path = /tmp/zookeeper/version-2

And remove all the existing log files using rm -r log.1

The log files will be created automatically again and will resolve the issue.

Uda answered 7/7, 2021 at 9:37 Comment(1)
Linux location was the same for mac OS – Rory
A
10

Try changing your zookeeper data directory. Your zookeeper data directory is defined in zookeeper.properties (I think the default is /tmp/zookeeper).

Perhaps you're not deleting the correct zookeeper directory?

I had the same problem, and this solution worked.

NOTE: I'm experimenting with Kafka, and not using it in production. I have no idea what else the above does, apart from fix this error...

Alcinia answered 11/1, 2020 at 12:22 Comment(0)
D
6

I've faced the same issue with Zookeeper after updating from version 3.4.x to 3.5.6. As described here. I've:

  1. added empty snapshot.0 file in data directory
  2. added a property 'zookeeper.snapshot.trust.empty=true' to Zookeeper configuration file (default is zoo.cfg)
Dollhouse answered 9/3, 2020 at 7:1 Comment(2)
For me, it didn't work with empty snapshot.0, but with the one attached at the link that you posted. +1 though, that's the workaround. – Urbai
Download sanpshot.0 from the given link and paste it to dataDir. Now, the question is where exactly the data directory exists? I am using a mac and for me, the dataDir is located at /usr/local/var/run/zookeeper/data/version-2. You can check your dataDir inside zookeeper config file (zoo.cfg). Just add version-x to this dataDir. Please read this comment carefully. – Ironmonger
M
3

Faced same issue in macOS. Solution: In kafka dir, path cd /tmp/zookeeper/version-2 deleted the log.1 file. It worked for me

Miso answered 19/9, 2022 at 18:13 Comment(0)
C
2

I also faced this error. The above solution of deleting the zookeeper folder from tmp worked for me. For those who don't know how to find tmp folder on mac : Open a Finder window. Click on "Go" in the menu bar at the top of the screen. Select "Go to Folder" from the dropdown menu. In the dialog
box that appears, enter /tmp. Click the "Go" button.

This will open the /tmp directory where you can find the temporary files and folders. Then delete the zookeeper folder.

Calculated answered 14/6, 2023 at 6:27 Comment(0)
S
0

if you are on windows make sure you escape the location of the zookeeper temp directory.

dataDir=d:\tmp\zookeeper

Shaughn answered 21/1, 2020 at 4:57 Comment(1)
If you are not sure about any information to answer a question, you should ask them through comments. – Splice
X
0

Created a new dir for logs and configured the same path in zoo.cfg. It worked:)

Xeniaxeno answered 2/7, 2020 at 22:26 Comment(0)
P
0

I use macOS and my solution was to delete everything in the dataDir, the default value should be /usr/local/var/lib/zookeeper.

Pincers answered 6/10, 2020 at 2:43 Comment(0)
O
0

For those who are using docker, I'll share my experience:

I've been running zookeeper confluentinc/cp-zookeeper:5.2.1 as it follows:

 docker run \
    --network kafka-net --name=zookeeper \
    -e ALLOW_ANONYMOUS_LOGIN=yes \
    -e ZOOKEEPER_CLIENT_PORT=2181 \
    -v /tmp/zookeeper-data:/var/lib/zookeeper/data \
    -v /tmp/zookeeper-txn-logs:/var/lib/zookeeper/log \
    -p 2181:2182 confluentinc/cp-zookeeper:5.2.1

As expected, I can see a few files placed in /tmp/zookeeper-txn-logs and /tmp/zookeeper-data on host. After cleaning up /tmp/zookeeper-data and running again, I've got the error No snapshot found, but there are log entries.

In my case, I just had to purge the data on /tmp/zookeeper-txn-logs. For a dev/production environment, I'd recommend following the docs https://access.redhat.com/documentation/en-us/red_hat_amq/6.3/html/fabric_guide/ensemble-purgetxnlog

Oestrogen answered 10/2, 2021 at 15:47 Comment(0)
R
0

on mac it helped me to restart the computer but first delete the tmp/zookeper files

Retrospective answered 8/5, 2023 at 18:58 Comment(1)
Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. – Phytogenesis
F
0

Had the same issue right now. If you are using the docker-compose version, the zookeeper logs have their own volume.

For me, it was enough to delete the volume and then run ./install.sh again. The volume will be recreated by ./install.sh anyay

docker volume rm sentry-self-hosted_sentry-zookeeper-log

If you cannot find this volume, you might have to check if your name differs a bit.

List all the volumes and check for something similar:

docker volume ls
Forsooth answered 19/7 at 13:42 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.