Login/Register
CouchDB: This database failed to load / No DB shards could be opened (logged in as admin)
Asked Answered
Solved authorizationcouchdbcouchappcouchdb-2.0fauxton
C

1

5

I am trying to regain access to my databases stored in CouchDB, but the error This database failed to load. gets shown instead of the databases (see attachments).

  • I started seeing all of these errors at once
  • It be some user rights misconfiguration (1. The error This database failed to load. is normal, when lacking privileges, 2. The actions to delete or replicate a DB are missing). However, I am logged as admin.

Screenshot of CouchDB Fauxton UI

Can you help me find the root cause of the error?

Simple PUT and DELETE requests curl -X PUT http://admin:[email protected]:5984/testdb:

[error] 2018-02-09T15:06:05.221393Z [email protected] <0.1112.0> e84f27d276 Request to create N=3 DB but only 2 node(s)
[error] 2018-02-09T15:06:05.237308Z [email protected] <0.1173.0> -------- ** System running to use fully qualified hostnames **
** Hostname localhost is illegal **

[notice] 2018-02-09T15:06:05.252506Z [email protected] <0.1112.0> e84f27d276 www.example.com:5984 130.211.68.93 undefined PUT /testdb 500 ok 47
[notice] 2018-02-09T15:06:16.286511Z [email protected] <0.1320.0> 890c465244 www.example.com:5984 130.211.68.93 undefined DELETE /testdb 500 ok 9
[error] 2018-02-09T15:06:16.288230Z [email protected] <0.1350.0> -------- ** System running to use fully qualified hostnames **
** Hostname localhost is illegal **

[error] 2018-02-09T15:06:16.288952Z [email protected] <0.1353.0> -------- ** System running to use fully qualified hostnames **
** Hostname localhost is illegal **

[error] 2018-02-09T15:06:16.289661Z [email protected] <0.1356.0> -------- ** System running to use fully qualified hostnames **
** Hostname localhost is illegal **

[error] 2018-02-09T15:06:16.290339Z [email protected] <0.1359.0> -------- ** System running to use fully qualified hostnames **
** Hostname localhost is illegal **

[error] 2018-02-09T15:06:16.290973Z [email protected] <0.1362.0> -------- ** System running to use fully qualified hostnames **
** Hostname localhost is illegal **

[error] 2018-02-09T15:06:16.291731Z [email protected] <0.1365.0> -------- ** System running to use fully qualified hostnames **
** Hostname localhost is illegal **

[error] 2018-02-09T15:06:16.292422Z [email protected] <0.1368.0> -------- ** System running to use fully qualified hostnames **
** Hostname localhost is illegal **

[error] 2018-02-09T15:06:16.292870Z [email protected] <0.1371.0> -------- ** System running to use fully qualified hostnames **
** Hostname localhost is illegal **

Starting CouchDB:

[info] 2018-02-09T14:54:12.420661Z [email protected] <0.7.0> -------- Application couch_log started on node '[email protected]'
[info] 2018-02-09T14:54:12.423091Z [email protected] <0.7.0> -------- Application folsom started on node '[email protected]'
[info] 2018-02-09T14:54:12.445498Z [email protected] <0.7.0> -------- Application couch_stats started on node '[email protected]'
[info] 2018-02-09T14:54:12.445617Z [email protected] <0.7.0> -------- Application khash started on node '[email protected]'
[info] 2018-02-09T14:54:12.450173Z [email protected] <0.7.0> -------- Application couch_event started on node '[email protected]'
[info] 2018-02-09T14:54:12.453404Z [email protected] <0.7.0> -------- Application ibrowse started on node '[email protected]'
[info] 2018-02-09T14:54:12.456050Z [email protected] <0.7.0> -------- Application ioq started on node '[email protected]'
[info] 2018-02-09T14:54:12.456173Z [email protected] <0.7.0> -------- Application mochiweb started on node '[email protected]'
[info] 2018-02-09T14:54:12.460566Z [email protected] <0.200.0> -------- Apache CouchDB 2.1.1 is starting.

[info] 2018-02-09T14:54:12.460656Z [email protected] <0.201.0> -------- Starting couch_sup
[info] 2018-02-09T14:54:12.523195Z [email protected] <0.200.0> -------- Apache CouchDB has started. Time to relax.

[info] 2018-02-09T14:54:12.523307Z [email protected] <0.200.0> -------- Apache CouchDB has started on http://127.0.0.1:5986/
[info] 2018-02-09T14:54:12.523449Z [email protected] <0.7.0> -------- Application couch started on node '[email protected]'
[info] 2018-02-09T14:54:12.523601Z [email protected] <0.7.0> -------- Application ets_lru started on node '[email protected]'
[info] 2018-02-09T14:54:12.540555Z [email protected] <0.7.0> -------- Application rexi started on node '[email protected]'
[info] 2018-02-09T14:54:12.546300Z [email protected] <0.273.0> -------- Opening index for db: _replicator idx: _design/_replicator sig: "3e823c2a4383ac0
c18d4e574135a5b08"
[info] 2018-02-09T14:54:12.550110Z [email protected] <0.287.0> -------- Opening index for db: _users idx: _design/_auth sig: "3e823c2a4383ac0c18d4e57413
5a5b08"
[error] 2018-02-09T14:54:12.570392Z [email protected] emulator -------- Error in process <0.302.0> on node '[email protected]' with exit value:
{badarg,[{ets,member,[mem3_openers,<<"myblogdb">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,486}]},{mem3_shards,l
oad_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,389}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,378}]},{mem3_sh
ards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,407}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,96}]},{fabric_doc_o
pen,go,3,[{file,"src/fabric_doc_open.erl"},{line,38}]},{couch_index_server,'-get_index/3-fun-0-',2,[{file,"src/couch_index_server.erl"},{line,68}]}]}

[error] 2018-02-09T14:54:12.570593Z [email protected] <0.251.0> -------- Error opening view group `sofa` from database `shards/e0000000-ffffffff/myblogd
b.1518183029`: {'EXIT',{{badmatch,{badarg,[{ets,member,[mem3_openers,<<"myblogdb">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards
.erl"},{line,486}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,389}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_
shards.erl"},{line,378}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,407}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shar
ds.erl"},{line,96}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,38}]},{couch_index_server,'-get_index/3-fun-0-',2,[{file,"src/couch_i
ndex_server.erl"},{line,68}]}]}},[{couch_mrview,get_info,2,[{file,"src/couch_mrview.erl"},{line,332}]},{couch_compaction_daemon,maybe_compact_view,3,[{f
ile,"src/couch_compaction_daemon.erl"},{line,253}]},{couch_compaction_daemon,maybe_compact_views,3,[{file,"src/couch_compaction_daemon.erl"},{line,227}]
},{couch_compaction_daemon,'-compact_loop/1-fun-0-',3,[{file,"src/couch_compaction_daemon.erl"},{line,141}]},{couch_server,'-all_databases/2-fun-0-',4,[
{file,"src/couch_server.erl"},{line,278}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,185}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},
{line,195}]},{couch_server,all_databases,2,[{file,"src/couch_server.erl"},{line,267}]}]}}
[error] 2018-02-09T14:54:12.576078Z [email protected] <0.331.0> -------- ** System running to use fully qualified hostnames **
** Hostname localhost is illegal **

[info] 2018-02-09T14:54:12.584026Z [email protected] <0.7.0> -------- Application mem3 started on node '[email protected]'
[info] 2018-02-09T14:54:12.584399Z [email protected] <0.7.0> -------- Application fabric started on node '[email protected]'
[info] 2018-02-09T14:54:12.605860Z [email protected] <0.7.0> -------- Application chttpd started on node '[email protected]'
[info] 2018-02-09T14:54:12.614513Z [email protected] <0.7.0> -------- Application couch_index started on node '[email protected]'
[info] 2018-02-09T14:54:12.614673Z [email protected] <0.7.0> -------- Application couch_mrview started on node '[email protected]'
[info] 2018-02-09T14:54:12.614809Z [email protected] <0.7.0> -------- Application couch_plugins started on node '[email protected]'
[notice] 2018-02-09T14:54:12.620187Z [email protected] <0.413.0> -------- Failed to ensure auth ddoc _users/_design/_auth exists for reason: read_failur
e
[error] 2018-02-09T14:54:12.620427Z [email protected] <0.251.0> -------- Error opening view group `_auth` from database `shards/40000000-5fffffff/_users
.1502869354`: {'EXIT',{{badmatch,{error,read_failure}},[{couch_mrview,get_info,2,[{file,"src/couch_mrview.erl"},{line,332}]},{couch_compaction_daemon,ma
ybe_compact_view,3,[{file,"src/couch_compaction_daemon.erl"},{line,253}]},{couch_compaction_daemon,maybe_compact_views,3,[{file,"src/couch_compaction_da
emon.erl"},{line,227}]},{couch_compaction_daemon,'-compact_loop/1-fun-0-',3,[{file,"src/couch_compaction_daemon.erl"},{line,141}]},{couch_server,'-all_d
atabases/2-fun-0-',4,[{file,"src/couch_server.erl"},{line,278}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,185}]},{filelib,do_fold_files2,8,
[{file,"filelib.erl"},{line,195}]},{couch_server,all_databases,2,[{file,"src/couch_server.erl"},{line,267}]}]}}
[error] 2018-02-09T14:54:12.640465Z [email protected] <0.251.0> -------- Error opening view group `_replicator` from database `shards/80000000-9fffffff/
_replicator.1502869354`: {'EXIT',{{badmatch,{error,read_failure}},[{couch_mrview,get_info,2,[{file,"src/couch_mrview.erl"},{line,332}]},{couch_compactio
n_daemon,maybe_compact_view,3,[{file,"src/couch_compaction_daemon.erl"},{line,253}]},{couch_compaction_daemon,maybe_compact_views,3,[{file,"src/couch_co
mpaction_daemon.erl"},{line,227}]},{couch_compaction_daemon,'-compact_loop/1-fun-0-',3,[{file,"src/couch_compaction_daemon.erl"},{line,141}]},{couch_ser
ver,'-all_databases/2-fun-0-',4,[{file,"src/couch_server.erl"},{line,278}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,185}]},{filelib,do_fol
d_files2,8,[{file,"filelib.erl"},{line,195}]},{couch_server,all_databases,2,[{file,"src/couch_server.erl"},{line,267}]}]}}
[notice] 2018-02-09T14:54:12.652087Z [email protected] <0.88.0> -------- config: [features] scheduler set to true for reason nil
[info] 2018-02-09T14:54:12.672254Z [email protected] <0.7.0> -------- Application couch_replicator started on node '[email protected]'
[info] 2018-02-09T14:54:12.681741Z [email protected] <0.7.0> -------- Application couch_peruser started on node '[email protected]'
[info] 2018-02-09T14:54:12.694711Z [email protected] <0.7.0> -------- Application ddoc_cache started on node '[email protected]'
[info] 2018-02-09T14:54:12.711264Z [email protected] <0.7.0> -------- Application global_changes started on node '[email protected]'
[info] 2018-02-09T14:54:12.711453Z [email protected] <0.7.0> -------- Application jiffy started on node '[email protected]'
[info] 2018-02-09T14:54:12.717159Z [email protected] <0.7.0> -------- Application mango started on node '[email protected]'
[info] 2018-02-09T14:54:12.721969Z [email protected] <0.7.0> -------- Application setup started on node '[email protected]'
[info] 2018-02-09T14:54:12.722104Z [email protected] <0.7.0> -------- Application snappy started on node '[email protected]'
[notice] 2018-02-09T14:54:17.623987Z [email protected] <0.472.0> -------- couch_replicator_clustering : cluster stable
[notice] 2018-02-09T14:54:17.626514Z [email protected] <0.617.0> -------- Started replicator db changes listener <0.835.0>
[notice] 2018-02-09T14:54:27.540971Z [email protected] <0.268.0> -------- rexi_buffer : cluster stable
[notice] 2018-02-09T14:54:27.541041Z [email protected] <0.265.0> -------- rexi_server : cluster stable

Listing DBs via CouchDB's Fauxton UI:

[notice] 2018-02-09T15:02:08.068878Z [email protected] <0.1176.0> 1558dcd7d9 www.example.com:5984 24.134.87.25 admin GET /_utils/index.html 304 ok 1
6
[notice] 2018-02-09T15:02:08.126185Z [email protected] <0.1176.0> 2e1ccbfb10 www.example.com:5984 24.134.87.25 admin GET /_utils/dashboard.assets/cs
s/styles.25bdeac6d2ce92b30461.css 304 ok 1
[notice] 2018-02-09T15:02:08.673483Z [email protected] <0.1176.0> a93b41dba2 www.example.com:5984 24.134.87.25 admin GET / 200 ok 4
[notice] 2018-02-09T15:02:08.738009Z [email protected] <0.1177.0> fe8dfd9e5c www.example.com:5984 24.134.87.25 admin GET /_session 200 ok 0
[notice] 2018-02-09T15:02:08.892143Z [email protected] <0.1177.0> a66af45255 www.example.com:5984 24.134.87.25 admin GET /_scheduler/jobs 200 ok 5
[notice] 2018-02-09T15:02:08.901744Z [email protected] <0.1176.0> e114044236 www.example.com:5984 24.134.87.25 admin GET /_all_dbs?_=1518188528333 2
00 ok 9
[notice] 2018-02-09T15:02:08.986915Z [email protected] <0.1178.0> b69c8dce91 www.example.com:5984 24.134.87.25 admin GET /_all_dbs?startkey=%22%22&e
ndkey=%22%E9%A6%99%22&limit=30&_=1518188528334 200 ok 2
[error] 2018-02-09T15:02:09.049365Z [email protected] <0.1176.0> be6ce99b2d req_err(1021143351) internal_server_error : No DB shards could be opened.
    [<<"fabric_util:get_shard/4 L185">>,<<"fabric:get_security/2 L146">>,<<"chttpd_auth_request:db_authorization_check/1 L91">>,<<"chttpd_auth_request:a
uthorize_request/1 L19">>,<<"chttpd:process_request/1 L293">>,<<"chttpd:handle_request_int/1 L231">>,<<"mochiweb_http:headers/6 L91">>,<<"proc_lib:init_
p_do_apply/3 L240">>]
[notice] 2018-02-09T15:02:09.049725Z [email protected] <0.1176.0> be6ce99b2d www.example.com:5984 24.134.87.25 undefined GET /_replicator?_=15181885
28335 500 ok 1
[error] 2018-02-09T15:02:09.056335Z [email protected] <0.1177.0> d07dad801b req_err(1021143351) internal_server_error : No DB shards could be opened.
    [<<"fabric_util:get_shard/4 L185">>,<<"fabric:get_security/2 L146">>,<<"chttpd_auth_request:db_authorization_check/1 L91">>,<<"chttpd_auth_request:a
uthorize_request/1 L19">>,<<"chttpd:process_request/1 L293">>,<<"chttpd:handle_request_int/1 L231">>,<<"mochiweb_http:headers/6 L91">>,<<"proc_lib:init_
p_do_apply/3 L240">>]
[notice] 2018-02-09T15:02:09.056610Z [email protected] <0.1177.0> d07dad801b www.example.com:5984 24.134.87.25 undefined GET /_users?_=1518188528336
 500 ok 1
[error] 2018-02-09T15:02:09.080553Z [email protected] <0.1178.0> 81dbed0cf9 req_err(1021143351) internal_server_error : No DB shards could be opened.
    [<<"fabric_util:get_shard/4 L185">>,<<"fabric:get_security/2 L146">>,<<"chttpd_auth_request:db_authorization_check/1 L91">>,<<"chttpd_auth_request:a
uthorize_request/1 L19">>,<<"chttpd:process_request/1 L293">>,<<"chttpd:handle_request_int/1 L231">>,<<"mochiweb_http:headers/6 L91">>,<<"proc_lib:init_
p_do_apply/3 L240">>]
[notice] 2018-02-09T15:02:09.080854Z [email protected] <0.1178.0> 81dbed0cf9 www.example.com:5984 24.134.87.25 undefined GET /functions?_=1518188528
340 500 ok 1
[error] 2018-02-09T15:02:09.099261Z [email protected] <0.1192.0> 856bef3f1a req_err(1021143351) internal_server_error : No DB shards could be opened.
    [<<"fabric_util:get_shard/4 L185">>,<<"fabric:get_security/2 L146">>,<<"chttpd_auth_request:db_authorization_check/1 L91">>,<<"chttpd_auth_request:a
uthorize_request/1 L19">>,<<"chttpd:process_request/1 L293">>,<<"chttpd:handle_request_int/1 L231">>,<<"mochiweb_http:headers/6 L91">>,<<"proc_lib:init_
p_do_apply/3 L240">>]
[notice] 2018-02-09T15:02:09.099639Z [email protected] <0.1192.0> 856bef3f1a www.example.com:5984 24.134.87.25 undefined GET /attribute-groups?_=151
8188528337 500 ok 2
[notice] 2018-02-09T15:02:09.105335Z [email protected] <0.1176.0> ec86ef466a www.example.com:5984 24.134.87.25 admin GET /myblogdb?_=1518188528341 2
00 ok 8
[error] 2018-02-09T15:02:09.182457Z [email protected] <0.1178.0> 07cd60fd59 req_err(1021143351) internal_server_error : No DB shards could be opened.
    [<<"fabric_util:get_shard/4 L185">>,<<"fabric:get_security/2 L146">>,<<"chttpd_auth_request:db_authorization_check/1 L91">>,<<"chttpd_auth_request:a
uthorize_request/1 L19">>,<<"chttpd:process_request/1 L293">>,<<"chttpd:handle_request_int/1 L231">>,<<"mochiweb_http:headers/6 L91">>,<<"proc_lib:init_
p_do_apply/3 L240">>]
[notice] 2018-02-09T15:02:09.182864Z [email protected] <0.1178.0> 07cd60fd59 www.example.com:5984 24.134.87.25 undefined GET /products?_=15181885283
43 500 ok 3
[error] 2018-02-09T15:02:09.183128Z [email protected] <0.1198.0> 9965ad9d2d req_err(1021143351) internal_server_error : No DB shards could be opened.
    [<<"fabric_util:get_shard/4 L185">>,<<"fabric:get_security/2 L146">>,<<"chttpd_auth_request:db_authorization_check/1 L91">>,<<"chttpd_auth_request:a
uthorize_request/1 L19">>,<<"chttpd:process_request/1 L293">>,<<"chttpd:handle_request_int/1 L231">>,<<"mochiweb_http:headers/6 L91">>,<<"proc_lib:init_
p_do_apply/3 L240">>]
[notice] 2018-02-09T15:02:09.183458Z [email protected] <0.1198.0> 9965ad9d2d www.example.com:5984 24.134.87.25 undefined GET /definitions?_=15181885
28339 500 ok 3
[error] 2018-02-09T15:02:09.183676Z [email protected] <0.1192.0> c29d349706 req_err(1021143351) internal_server_error : No DB shards could be opened.
    [<<"fabric_util:get_shard/4 L185">>,<<"fabric:get_security/2 L146">>,<<"chttpd_auth_request:db_authorization_check/1 L91">>,<<"chttpd_auth_request:a
uthorize_request/1 L19">>,<<"chttpd:process_request/1 L293">>,<<"chttpd:handle_request_int/1 L231">>,<<"mochiweb_http:headers/6 L91">>,<<"proc_lib:init_
Chaffer answered 9/2, 2018 at 15:39 Comment(0)
C
6

The error was caused by update to CouchDB 2.1.1. It's explained on the CouchDB website that the new version brings breaking changes and a line has to be modified in the config file.

So I had to change this file:

$ sudo vim /opt/couchdb/etc/vm.args

-name couchdb@localhost changed to -name [email protected]

Don't forget to move the DB shards to the new node and restart CouchDB.

This script is useful for migrating the DB shards to the new node:

#!/bin/bash
curl "http://admin:[email protected]:5986/_dbs/$1" > $1.json
cat $1.json | sed s/$2/$3/g > $1_edited.json
curl -X PUT "http://admin:[email protected]:5986/_dbs/$1" -d @$1_edited.json
Chaffer answered 9/2, 2018 at 16:47 Comment(2)
Hey Peter, I am facing a similar problem while setting up a cluster. As far as I understand we can not use [email protected] in cluster mode. The exact error i am getting is - Failed to ensure auth ddoc _users/_design/_auth exists for reason: read_failure. Any idea about this? Thanks :-) – Uella
I had to use the url :5984/_node/_local/_dbs/$1 instead of :5986/_dbs/$1 for it to work – Dispassion

© 2022 - 2024 — McMap. All rights reserved.