Login/Register
OrientDB corruption state in Nexus Repository version 3.2.0-01
Asked Answered
nexusorientdbnexus3
K

7

33

We are using Nexus Repository version 3.2.0-01 and started getting the following error in nexus.log. OrientDB has got corrupted. Nexus is no longer starting. 

2017-03-21 13:00:36,329+0000 INFO  [FelixStartLevel] *SYSTEM org.sonatype.nexus.internal.orient.DatabaseServerImpl - OrientDB version: 2.2.13
2017-03-21 13:00:36,348+0000 INFO  [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.OServer - OrientDB Server v2.2.13 is starting up...
2017-03-21 13:00:36,355+0000 INFO  [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.OServer - Databases directory: /c9/setup/apps/nexus/sonatype-work/nexus3/db
2017-03-21 13:00:36,647+0000 WARN  [FelixStartLevel] *SYSTEM com.orientechnologies - Not enough physical memory available for DISKCACHE: 1,873MB (heap=1,161MB direct=2,048MB). Set lower Maximum Heap (-Xmx setting on JVM) and restart OrientDB. Now running with DISKCACHE=256MB
2017-03-21 13:00:36,648+0000 INFO  [FelixStartLevel] *SYSTEM com.orientechnologies - OrientDB config DISKCACHE=256MB (heap=1,161MB direct=2,048MB os=1,873MB)
2017-03-21 13:00:36,735+0000 WARN  [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.OServer - Found ORIENTDB_ROOT_PASSWORD variable, using this value as root's password
2017-03-21 13:00:36,899+0000 INFO  [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.handler.OJMXPlugin - JMX plugin installed and active: 

profilerManaged=true
2017-03-21 13:00:36,901+0000 INFO  [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.OServer - OrientDB Studio available at $ANSI{blue 

http://localhost:2480/studio/index.html}
2017-03-21 13:00:36,902+0000 INFO  [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.OServer - $ANSI{green:italic OrientDB Server is active} v2.2.13.
2017-03-21 13:00:36,902+0000 INFO  [FelixStartLevel] *SYSTEM org.sonatype.nexus.internal.orient.DatabaseServerImpl - Activated
2017-03-21 13:00:36,904+0000 INFO  [FelixStartLevel] *SYSTEM org.sonatype.nexus.extender.NexusLifecycleManager - Start UPGRADE
2017-03-21 13:00:37,632+0000 ERROR [FelixStartLevel] *SYSTEM org.sonatype.nexus.orient.DatabaseInstanceImpl - Lifecycle operation start failed
com.orientechnologies.orient.core.exception.OSecurityAccessException: User or password not valid for database: 'config'^M
        DB name="config"
        at com.orientechnologies.orient.core.metadata.security.OSecurityShared.authenticate(OSecurityShared.java:237) [na:na]
        at com.orientechnologies.orient.core.metadata.security.OSecurityShared.authenticate(OSecurityShared.java:237) [na:na]
        at com.orientechnologies.orient.core.metadata.security.OSecurityExternal.authenticate(OSecurityExternal.java:62) [na:na]
        at com.orientechnologies.orient.core.metadata.security.OSecurityProxy.authenticate(OSecurityProxy.java:107) [na:na]
        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.open(ODatabaseDocumentTx.java:274) [na:na]
        at org.sonatype.nexus.orient.DatabaseManagerSupport.connect(DatabaseManagerSupport.java:142) [org.sonatype.nexus.orient:3.2.0.01]
        at org.sonatype.nexus.orient.DatabaseInstanceImpl.doStart(DatabaseInstanceImpl.java:56) [na:na]
        at org.sonatype.goodies.lifecycle.LifecycleSupport.start(LifecycleSupport.java:104) [org.sonatype.goodies.lifecycle:2.2.3]
        at org.sonatype.goodies.lifecycle.Lifecycles.start(Lifecycles.java:44) [org.sonatype.goodies.lifecycle:2.2.3]
        at org.sonatype.nexus.orient.DatabaseManagerSupport.createInstance(DatabaseManagerSupport.java:258) [org.sonatype.nexus.orient:3.2.0.01]
        at org.sonatype.nexus.orient.DatabaseManagerSupport.instance(DatabaseManagerSupport.java:232) [org.sonatype.nexus.orient:3.2.0.01]
        at org.sonatype.nexus.internal.orient.ConfigDatabase$ProviderImpl.get(ConfigDatabase.java:61) [org.sonatype.nexus.base:3.2.0.01]
        at org.sonatype.nexus.internal.orient.ConfigDatabase$ProviderImpl.get(ConfigDatabase.java:1) [org.sonatype.nexus.base:3.2.0.01]
        at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81) [com.google.inject:4.1.0]
        at com.google.inject.internal.BoundProviderFactory.provision(BoundProviderFactory.java:72) [com.google.inject:4.1.0]
        at com.google.inject.internal.ProviderInternalFactory$1.call(ProviderInternalFactory.java:65) [com.google.inject:4.1.0]
        at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:115) [com.google.inject:4.1.0]
        at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:133) [com.google.inject:4.1.0]
        at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:68) [com.google.inject:4.1.0]
        at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:63) [com.google.inject:4.1.0]
        at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:62) [com.google.inject:4.1.0]
        at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) [com.google.inject:4.1.0]
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) [com.google.inject:4.1.0]
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) [com.google.inject:4.1.0]
        at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194) [com.google.inject:4.1.0]
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) [com.google.inject:4.1.0]
        at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019) [com.google.inject:4.1.0]
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085) [com.google.inject:4.1.0]
        at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015) [com.google.inject:4.1.0]
         at org.eclipse.sisu.inject.LazyBeanEntry.getValue(LazyBeanEntry.java:81) [org.eclipse.sisu.inject:0.3.2]
        at org.eclipse.sisu.wire.BeanProviders.firstOf(BeanProviders.java:179) [org.eclipse.sisu.inject:0.3.2]
        at org.eclipse.sisu.wire.PlaceholderBeanProvider.lookup(PlaceholderBeanProvider.java:122) [org.eclipse.sisu.inject:0.3.2]
        at org.eclipse.sisu.wire.PlaceholderBeanProvider.get(PlaceholderBeanProvider.java:90) [org.eclipse.sisu.inject:0.3.2]
        at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81) [com.google.inject:4.1.0]
        at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:53) [com.google.inject:4.1.0]
        at com.google.inject.internal.ProviderInternalFactory$1.call(ProviderInternalFactory.java:65) [com.google.inject:4.1.0]
        at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:115) [com.google.inject:4.1.0]
        at org.eclipse.sisu.bean.BeanScheduler$Activator.onProvision(BeanScheduler.java:176) [org.eclipse.sisu.inject:0.3.2]
        at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:126) [com.google.inject:4.1.0]
        at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:68) [com.google.inject:4.1.0]
        at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:63) [com.google.inject:4.1.0]
        at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:45) [com.google.inject:4.1.0]
        at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019) [com.google.inject:4.1.0]
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085) [com.google.inject:4.1.0]
        at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015) [com.google.inject:4.1.0]
        at org.sonatype.nexus.upgrade.internal.ModelVersionStore.doStart(ModelVersionStore.java:82) [org.sonatype.nexus.upgrade:3.2.0.01]
        at org.sonatype.nexus.common.stateguard.StateGuardLifecycleSupport.start(StateGuardLifecycleSupport.java:67) [org.sonatype.nexus.common:3.2.0.01]
        at org.sonatype.nexus.upgrade.internal.ModelVersionStore$$EnhancerByGuice$$b30c182f.CGLIB$start$5(<generated>) [3.2.0-01:na]
        at org.sonatype.nexus.upgrade.internal.ModelVersionStore$$EnhancerByGuice$$b30c182f$$FastClassByGuice$$94ba58c4.invoke(<generated>) [3.2.0-01:na]
        at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228) [com.google.inject:4.1.0]
        at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:76) [com.google.inject:4.1.0]
        at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39) [org.sonatype.nexus.common:3.2.0.01]
        at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:191) [org.sonatype.nexus.common:3.2.0.01]
        at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:56) [org.sonatype.nexus.common:3.2.0.01]
        at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:77) [com.google.inject:4.1.0]
        at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:55) [com.google.inject:4.1.0]
        at org.sonatype.nexus.upgrade.internal.ModelVersionStore$$EnhancerByGuice$$b30c182f.start(<generated>) [3.2.0-01:na]
        at org.sonatype.nexus.upgrade.internal.UpgradeServiceImpl.doStart(UpgradeServiceImpl.java:80) [org.sonatype.nexus.upgrade:3.2.0.01]
        at org.sonatype.goodies.lifecycle.LifecycleSupport.start(LifecycleSupport.java:104) [org.sonatype.goodies.lifecycle:2.2.3]
        at org.sonatype.nexus.extender.NexusLifecycleManager.startComponent(NexusLifecycleManager.java:154) [org.sonatype.nexus.extender:3.2.0.01]
        at org.sonatype.nexus.extender.NexusLifecycleManager.to(NexusLifecycleManager.java:94) [org.sonatype.nexus.extender:3.2.0.01]
        at org.sonatype.nexus.extender.NexusContextListener.frameworkEvent(NexusContextListener.java:185) [org.sonatype.nexus.extender:3.2.0.01]
        at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1429) [org.apache.felix.framework-5.4.0.jar:na]
        at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308) [org.apache.felix.framework-5.4.0.jar:na]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]

Root cause Root cause of this was found to be an unclean shutdown of Nexus because of disk space getting full, this resulted in DB corruption. 

2017-03-17 07:09:49,480+0000 WARN  [elasticsearch[A82A95AC-5D396A2B-337B8736-5017E8FC-915232E4][local_transport][T#1]] *SYSTEM org.elasticsearch.cluster.action.shard - [A82A95AC-5D396A2B-337B8736-5017E8FC-915232E4] [cab22f445c1892acef64d17ac2da6d0086023e21][0] received shard failed for [cab22f445c1892acef64d17ac2da6d0086023e21][0], node[HBjARpI7RyqqqruVqEJ8Ig], [P], v[11], s[INITIALIZING], a[id=FmxOHS7nQcGNz_93VlTpKg], unassigned_info[[reason=ALLOCATION_FAILED], at[2017-03-17T07:09:49.101Z], details[failed recovery, failure IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to recover from translog]; nested: EngineException[failed to recover from translog]; nested: IndexFailedEngineException[Index failed for [component#36e3dec8de528c9b41c5c6ac3b18e3f7]]; nested: NotSerializableExceptionWrapper[No space left on device]; ]], indexUUID [fLN4hdE8QPKH7Pj5RpLtmw], message [engine failure, reason [index]], failure [NotSerializableExceptionWrapper[No space left on device]]
org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: No space left on device
at sun.nio.ch.FileDispatcherImpl.write0(Native Method) [na:1.8.0_121]
at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60) [na:1.8.0_121]
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) [na:1.8.0_121]
at sun.nio.ch.IOUtil.write(IOUtil.java:65) [na:1.8.0_121]
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211) [na:1.8.0_121]
at org.elasticsearch.common.io.Channels.writeToChannel(Channels.java:211) [na:na]
at org.elasticsearch.common.bytes.PagedBytesReference.writeTo(PagedBytesReference.java:132) [na:na]
at org.elasticsearch.index.translog.BufferingTranslogWriter.add(BufferingTranslogWriter.java:57) [na:na]

No backup We have not backed up Nexus VM, therefore we wont be able to restore it back. What is the possible way out - recreating the database - with least damage. Appreciate any help.

Knacker answered 22/3, 2017 at 12:31 Comment(0)
E
39

Just in case that actually matters for somebody else - the above did not help me, but the json export / import one.

In my case nexus runs in the official docker image, so 

docker exec -it nexus bash

Or however you access your nexus shell

cd /tmp
/opt/jdk1.8.0_141/bin/java -jar /opt/sonatype/nexus/lib/support/nexus-orient-console.jar

CONNECT plocal:/nexus-data/db/component admin admin

export database component-export
drop database
create database plocal:/nexus-data/db/component
import database component-export.json.gz

In your case the java-path could vary as also the data-path (nexus data) which we use to connect/create the database.

After that, restarting the service and your component database should be up and running again

Evers answered 31/8, 2017 at 16:47 Comment(2)
I had to use CREATE DATABASE PLOCAL:/usr/local/orientdb/databases/demo to create a database – Pusey
thanky very much for sharing, worked for me as well. solved the same corruption issue with component caused by full disk. also +1 @Markus, had to prefix plocal: as well, else I got incorrelt url exception – Catnap
H
29

We had similar issue with Nexus OSS verison 3.2.0-01 and OrientDB corrupted state because of unclean shutdown.

In our case the corrupted storage was component located in

$install-dir/sonatype-work/nexus3/db/component

2017-04-14 15:09:20,007+0200 ERROR [qtp18266400-74] *UNKNOWN com.google.common.eventbus.EventBus.reentrant - Could not dispatch event AssetUpdatedEvent{metadata=AttachedEntityMetadata{schema=asset, document=asset#18:91856{bucket:#13:3,format:maven2,last_updated:Fri Apr 14 15:09:19 CEST 2017,attributes:[5],component:null,name:org/apache/maven/plugins/maven-jar-plugin/maven-metadata.xml,size:743,content_type:application/xml,blob_ref:default@ECF4439E-D7F184F8-578BAB14-ACFC0FB8-5589B163:3fb95745-4aed-460f-b711-b5f5b74b837c,last_accessed:Fri Apr 14 15:09:19 CEST 2017} v466}, remoteNodeId=null} to subscriber org.sonatype.nexus.repository.maven.internal.group.MavenGroupFacet$$EnhancerByGuice$$a88d7327@83eca7 method [public void org.sonatype.nexus.repository.maven.internal.group.MavenGroupFacet.on(org.sonatype.nexus.repository.storage.AssetEvent)]
com.orientechnologies.orient.core.exception.OStorageException: Error during transaction commit
        DB name="component"
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.makeRollback(OAbstractPaginatedStorage.java:2210) [na:na]
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commit(OAbstractPaginatedStorage.java:1433) [na:na]
        at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit(OTransactionOptimistic.java:539) [com.orientechnologies.orientdb-core:2.2.13]
        at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:104) [com.orientechnologies.orientdb-core:2.2.13]
        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2756) [com.orientechnologies.orientdb-core:2.2.13]
        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2725) [com.orientechnologies.orientdb-core:2.2.13]
        at org.sonatype.nexus.repository.storage.StorageTxImpl.commit(StorageTxImpl.java:172) [org.sonatype.nexus.repository:3.2.0.01]
        at sun.reflect.GeneratedMethodAccessor105.invoke(Unknown Source) [na:na]

We managed to solve our problem with the following strategy

  1. Scheduling Nexus tasks, particullary:

    • Rebuild Maven repository metadata.
    • Rebuild repository index.
    • Compact blob store.

  2. Orient DB manipulation:

    • Rebuilding indexes.
    • Repairing database.

According Nexus support you can access Orient DB console with Nexus OSS verison 3.2.1-01. Everything is explained here. Make sure your Nexus application is properly shut down before you apply any operations on database.

Once in Orient DB console you connect to local database with

> CONNECT PLOCAL:${your_install_dir}/sonatype-work/nexus3/db/${database} admin admin

And you perform the following operations:

> REBUILD INDEX *
> REPAIR DATABASE --fix-graph
> REPAIR DATABASE --fix-links
> REPAIR DATABASE --fix-ridbags
> REPAIR DATABASE --fix-bonsai
> DISCONNECT

I am absolutely aware my answer probably won't solve your concrete problem but I still wanted to give you some directions because you don't have any backups.

Hearttoheart answered 17/4, 2017 at 10:38 Comment(3)
In our case when we ran out of space the wal files were corrupted, and we weren't able to connect to the databases from the orient DB console. – Polity
For me, this seemed to work at first, but pushing still gave a 500. Had to restore from backup in the end – Ferbam
this didnt work for me it was too currupt, followed this one with an export import and it worked fine #42952210 – Suellen
H
13

I had the same mistake:

Cannot open local storage '/nexus-data/db/config' with mode=rw

and when starting the nexus console (java -jar /opt/sonatype/nexus/lib/support/nexus-orient-console.jar) the following message appeared:

enter image description here

The reason was that Nexus was out of memory and had created some ".wal" files with 0 bytes in the folder "/nexus-data/db/config/". After I deleted them, everything worked again.

Many thanks also to yobert https://gist.github.com/marcelmaatkamp/123e8793e07a72a382d8d0e8d66bbd8f

Hoptoad answered 5/11, 2020 at 10:5 Comment(1)
Deleting both the db/config/*.wal files and the db/component/*.wal files fixed it for me. – Webbing
B
12

The database write ahead logs are corrupt.

Make a backup of $install-dir/sonatype-work/nexus3/db/component, and then remove files with extension .wal from that directory. Then try the startup again

if the issue still persist, try clearing all .wal files under the "DB" directory, will resolve the issue,

I did tried this method and it worked like charm.

regards vinay

Brunelle answered 27/4, 2019 at 4:27 Comment(0)
P
4

With Nexus 3.16 there is now a task: Create Repair - Reconcile component database from blob store

super slow, but it helps recreating a accidentally deleted blobstore still on disk.

Pusey answered 4/6, 2019 at 15:54 Comment(1)
That helped me restore some information in the DB, but before, I had to bring it to a minimal usable state, and only the export/create/import cycle helped me achieve that. – Chatterton
A
2

Deleting the cache folder (/opt/sonatype-work/nexus3/cache) solved the issue for me

Answerable answered 21/3, 2020 at 2:28 Comment(1)
If you are like me, just doing a new installation, I think this is the solution. – Adobe
O
1

Removing the file frozen.marker from the db folder solved it for us.

Outcross answered 3/6, 2022 at 13:57 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.