Failed to start Sonar Webservice

Question

I am working on Sonarqube 5.1. It used to work fine earlier, later i got this no space left on device error.So i have deleted some unwanted files and restarted the service. Now i can see that the service status is running, but i cannot access the webpage(since sonar webservice is not started) I have checked the logs and i find the below error.

2015.08.03 10:58:56 INFO   es[o.elasticsearch.node]  [sonar-1438592314778] started
2015.08.03 10:59:03 INFO   es[o.e.gateway]  [sonar-1438592314778] recovered [6] indices into cluster_state
2015.08.03 11:00:44 WARN   es[o.e.indices.cluster]  [sonar-1438592314778] [sourcelines][0] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [sourcelines][0] failed to recover shard
at     org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:287) ~[elasticsearch-1.4.4.jar:na]
at  org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132) ~[elasticsearch-1.4.4.jar:na]
at  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:114 2) [na:1.8.0_40]
at  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_40]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_40]
Caused by: org.elasticsearch.index.translog.TranslogCorruptedException: translog corruption while reading from stream
at  org.elasticsearch.index.translog.ChecksummedTranslogStream.read(ChecksummedTranslogStream.java:70) ~[elasticsearch-1.4.4.jar:na]
at  org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:257) ~[elasticsearch-1.4.4.jar:na]
... 4 common frames omitted
Caused by: org.elasticsearch.ElasticsearchException: failed to read [sourceline][22c2d1d1-8e73-47c0-b9fa-f8f5cc96b93f_796]
at org.elasticsearch.index.translog.Translog$Index.readFrom(Translog.java:520) ~[elasticsearch-1.4.4.jar:na]
at  org.elasticsearch.index.translog.ChecksummedTranslogStream.read(ChecksummedTranslogStream.java:68) ~[elasticsearch-1.4.4.jar:na]
... 5 common frames omitted
Caused by: org.elasticsearch.ElasticsearchIllegalArgumentException: No version type match [105]
at org.elasticsearch.index.VersionType.fromValue(VersionType.java:307) ~[elasticsearch-1.4.4.jar:na]
at org.elasticsearch.index.translog.Translog$Index.readFrom(Translog.java:517) ~[elasticsearch-1.4.4.jar:na]
... 6 common frames omitted
2015.08.03 11:00:44 WARN   es[o.e.c.action.shard]  [sonar-1438592314778] [sourcelines][0] sending failed shard for [sourcelines][0], node[5eRgYcVUTNCNLjhhzdiHGA], [P], s[INITIALIZING], indexUUID [JaH5lnRnRVOFD95Nw28W5Q], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[sourcelines][0] failed to recover shard]; nested: TranslogCorruptedException[translog corruption while reading from stream]; nested: ElasticsearchException[failed to read [sourceline][22c2d1d1-8e73-47c0-b9fa-f8f5cc96b93f_796]]; nested: ElasticsearchIllegalArgumentException[No version type match [105]]; ]]
2015.08.03 11:00:44 WARN   es[o.e.c.action.shard]  [sonar-1438592314778] [sourcelines][0] received shard failed for [sourcelines][0], node[5eRgYcVUTNCNLjhhzdiHGA], [P], s[INITIALIZING], indexUUID [JaH5lnRnRVOFD95Nw28W5Q], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[sourcelines][0] failed to recover shard]; nested: TranslogCorruptedException[translog corruption while reading from stream]; nested: ElasticsearchException[failed to read [sourceline][22c2d1d1-8e73-47c0-b9fa-f8f5cc96b93f_796]]; nested: ElasticsearchIllegalArgumentException[No version type match [105]]; ]]

do i have to delete shards? if yes how to do that? how to recover the shard? is indexing wrong? where do i find [sourcelines] on my machine ? help me with this.

Thankyou in advance.

Papule

Answer 1

Your ES index seems to be corrupted.

All base information is stored in the DB. Do not delete or update it manually.

On the contrary, if you delete the ES indexes, they will simply be rebuilt at the next startup (which may take some time depending on the quantity of data you have).

To achieve that, delete the es directory in the data directory of your SQ instance.

Ukase

Note: from SonarQube 6.6 and onwards, the directory is named 'es5' and not 'es' anymore – Ukase

And with SonarQube 7 it is called 'es6'. – Evanne

Answer 2

We have faced this issues many times. We delete the es directory under $SONAR_HOME/data directory of SQ instance and restart SQ instance. It start the indexing again Note : Only problem you will face , during indexing , your SQ webserver will be not accessible. You can not browse your localhost:9000(it will keep loading unless indexing gets over). As our project is big , it takes couple of hours to complete the indexing.Once the complete indexing is done, you can browse your SQ webserver.

Bumbling

Answer 3

If u are using Database to store results of sonar analysis , then clear the database once and then start the sonar again because it sometimes mess up with old and new results stored in database.It it not doing the indexing of your files.

Bilski

Answer 4

Delete .recovery file inside the translog folder

Eg:/es/elasticsearch-1.7.1/data/[elasticsearch_clustername]/nodes/0/indices/[indexname]/2/translog/

Inessa

Answer 5

There is a way to specifically fix the corruption of elasticsearch's translog, which is affected here. The accepted answer results in high server overhead and seems to accept a lot of data loss. Take a look into my response here as a first step (similar to Balaji's response, but using an official tool for it): Elastic Search IndexShardGatewayRecoveryException

Backwoodsman