Unable to create a composite index, stuck at INSTALLED

Question

I'm unable to create an index. My Gremlin code is as follows:

usernameProperty = mgmt.getPropertyKey('username')
usernameIndex = mgmt.buildIndex('byUsernameUnique', Vertex.class).addKey(usernameProperty).unique().buildCompositeIndex()
mgmt.setConsistency(usernameIndex, ConsistencyModifier.LOCK)
mgmt.commit()

Shortly after I receive two errors:

18:04:57 ERROR com.thinkaurelius.titan.graphdb.database.management.ManagementLogger - Evicted [1@0a00009d2537-ip-10-0-0-1572] from cache but waiting too long for transactions to close. Stale transaction alert on: [standardtitantx[0x6549ce71]] 18:04:57 ERROR com.thinkaurelius.titan.graphdb.database.management.ManagementLogger - Evicted [1@0a00009d2537-ip-10-0-0-1572] from cache but waiting too long for transactions to close. Stale transaction alert on: [standardtitantx[0x2a2815cc], standardtitantx[0x025dc2c0]]

The status of the index is stuck at INSTALLED:

usernameIndex.getIndexStatus(usernameProperty)
==>INSTALLED

I read that a failed instance could cause the issue, but a check of running instances shows just one:

mgmt.getOpenInstances()
==>0a00009d3011-ip-10-0-0-1572(current)

I've also tried issuing a REGISTER_INDEX action, which also gets evicted from the transaction cache with a similar error message:

mgmt.updateIndex(usernameIndex, SchemaAction.REGISTER_INDEX).get()
mgmt.commit()

I've also tried restarting the server multiple times.

It seems like the registration process is simply timing out, causing an "eviction" from the transaction cache. I've waited 48 hours just to be sure it wasn't a slow process. Normal reads, writes, and associated commits to Titan do seem to be working correctly, I'm just not able to create this index. I'm stuck, is there something else I can try? Is there a way to extend the timeout on that transaction?

I'm running Titan 1.0.0 using a DynamoDB backend (setup with the AWS provided CloudFormation template).

EDIT: Here is the complete command I'm pasting into Gremlin with the addition of the awaitGraphStatus step suggested by @M-T-A:

mgmt = graph.openManagement();
usernameIndex = mgmt.getPropertyKey('usernameIndex');
mgmt.buildIndex('byUsername',Vertex.class).addKey(usernameIndex).unique().buildCompositeIndex();
// I have tried with and without a commit here: mgmt.commit();
mgmt.awaitGraphIndexStatus(graph, 'byUsername').status(SchemaStatus.REGISTERED).timeout(10, java.time.temporal.ChronoUnit.MINUTES).call();

This results in the following error:

java.lang.NullPointerException at com.thinkaurelius.titan.graphdb.database.management.GraphIndexStatusWatcher.call(GraphIndexStatusWatcher.java:52) at com.thinkaurelius.titan.graphdb.database.management.GraphIndexStatusWatcher.call(GraphIndexStatusWatcher.java:18) at java_util_concurrent_Callable$call.call(Unknown Source) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:110) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:114) at groovysh_evaluate.run(groovysh_evaluate:3) at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:215) at org.codehaus.groovy.tools.shell.Interpreter.evaluate(Interpreter.groovy:69) at org.codehaus.groovy.tools.shell.Groovysh.execute(Groovysh.groovy:185) at org.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:119) at org.codehaus.groovy.tools.shell.ShellRunner.work(ShellRunner.groovy:94)

I will also note that the routine to disable and delete an index is also failing.

mgmt = graph.openManagement()
theIndex = mgmt.getGraphIndex('byUsername')
mgmt.updateIndex(theIndex, SchemaAction.DISABLE_INDEX).get()
mgmt.commit()
mgmt.awaitGraphIndexStatus(graph, 'byUsername').status(SchemaStatus.DISABLED).timeout(10, java.time.temporal.ChronoUnit.MINUTES).call();
m = graph.openManagement()
i = m.getGraphIndex('byUsername')
m.updateIndex(i, SchemaAction.REMOVE_INDEX).get()
m.commit()

19:26:26 ERROR com.thinkaurelius.titan.graphdb.database.management.ManagementLogger - Evicted [1@ac1f3fa810472-ip-172-31-63-1681] from cache but waiting too long for transactions to close. Stale transaction alert on: [standardtitantx[0x2314cd97], standardtitantx[0x39f8adc0], standardtitantx[0x09de1b85]]

EDIT 2: Attempting to create a new property key and index within the same transaction DOES WORK! But does this mean that I can't create indexes on existing property keys??

graph.tx().rollback();
mgmt = graph.openManagement();
indexName = 'byUsernameTest2';
propertyKeyName = 'testPropertyName2';
propertyKey = mgmt.makePropertyKey(propertyKeyName).dataType(String.class).cardinality(Cardinality.SINGLE).make();
mgmt.buildIndex(indexName,Vertex.class).addKey(propertyKey).buildCompositeIndex();
mgmt.commit();
graph.tx().commit();
mgmt.awaitGraphIndexStatus(graph, indexName).status(SchemaStatus.REGISTERED).timeout(10, java.time.temporal.ChronoUnit.MINUTES).call();
mgmt.commit();

After a pause, this results in:

This management system instance has been closed

Trying to fetch the new index results in:

mgmt = graph.openManagement();
index = mgmt.getGraphIndex('byUsernameTest2');
propkey = mgmt.getPropertyKey('testPropertyName2');
index.getIndexStatus(propkey);

==>ENABLED

Appetizer

for more debug logging, add this to ./conf/log4j-console.properties log4j.logger.com.thinkaurelius.titan.graphdb.database.management=DEBUG – Homozygous

do you have a big dataset stored ? Did you try it on a almost-empty database? – Groceryman

Answer 1

You need to wait for the Titan and DynamoDB to get the index registered. You could do so by:

ManagementSystem.awaitGraphIndexStatus(graph, propertyKeyIndexName)
                    .status(SchemaStatus.REGISTERED)
                    .timeout(10, java.time.temporal.ChronoUnit.MINUTES) // set timeout to 10 min
                    .call();

The default timeout is usually not long enough, so you could increase it to 10 minutes, it usually does the trick with Dynamo backened.

Only when the index is in REGISTERED state, you could perform a reindex. ONce the reindex is done, you need to wait until it's ENABLED. by reusing the code sample above and changing the state to ENABLED.

For more info, see the docs.

Edit

Let me share the code that works with me on Berkeley and Dynamo DB backends all the time.

    graph.tx().rollback(); //Never create new indexes while a transaction is active
    TitanManagement mgmt=graph.openManagement();
    PropertyKey propertyKey=getOrCreatePropertyKeyIfNotExist(mgmt, propertyKeyName);
    String indexName = makePropertyKeyIndexName(propertyKey);

    if (mgmt.getGraphIndex(indexName)==null) { 
        mgmt.buildIndex(indexName, Vertex.class).addKey(propertyKey).buildCompositeIndex();
        mgmt.commit(); // you MUST commit mgmt
        graph.tx().commit(); // and commit the transaction too
        ManagementSystem.awaitGraphIndexStatus(graph, indexName).status(SchemaStatus.REGISTERED).call();
    }else { // already defined.
        mgmt.rollback();
        graph.tx().rollback();
    }

private static PropertyKey getOrCreatePropertyKeyIfNotExist(TitanManagement mgmt, String s) {
    PropertyKey key = mgmt.getPropertyKey(s);
    if (key != null)
        return key;
    else
        return mgmt.makePropertyKey(s).dataType(String.class).make();
}

private static String makePropertyKeyIndexName(PropertyKey pk) {
    return pk.name() + Tokens.INDEX_SUFFIX;
}

From the error that I saw, it seems like Titan couldn't get the index which means you're waiting for the index that is not even defined. Have a look at the line that causes the error here.

Make sure you're passing the right index name to the awaitGraphIndexStatus.

Indocile

Thanks for the reply. This is returning an error about ChronoUnit: No such property: ChronoUnit for class: groovysh_evaluate – Appetizer

The class has to be imported when using Gremlin console. I updated the answer for you. – Indocile

Adding that step results in a java.lang.NullPointerException. I've edited the question to include the exact commands I pasted into Gremlin. – Appetizer

Thanks but unfortunately it is not working. Error: No signature of method: groovysh_evaluate.getOrCreateIfNotExist(). I'm passing a string (name of the property) as the second arg since propertyKeyName isn't defined anywhere. I couldn't find this function in the Titan API docs. Does this work in a Gremlin shell for you? – Appetizer

Remember that this is Java code, you may want to change it a little bit to get it running in the Gremlin console ;) – Indocile

Some success! Please see edit 2. I'm able to create an index using your code only if I create the property key in the same transaction. Can you tell why I would not be able to create an index on an existing property key? – Appetizer

Instead of using makePropertyKey, use a method called something like getOrCreatePropertyKey or use my routine to fetch the property key. Will that work? – Indocile

@Appetizer this method – Indocile

I tried both techniques. If the property key does not already exist it will create it and the index correctly. If the key does exist it will fetch the key correctly but fail to create the index. It fails with the same "Evicted" message referenced in my post. – Appetizer

Answer 2

If you're running multiple Titan instances, you should be aware that they need to coordinate before the index will become available.

Furthermore, there are various subtleties around transaction management and under what circumstances transactions will be left open; I believe Titan 1.0.0 doesn't have the latest from TinkerPop in that regard. Have you tried creating the index immediately after boot?

Finally, the index creation process is different depending on whether or not the property keys being indexed have been used before. Are you attempting to index new keys, or existing ones?

Herschel

I'm running a single instance on a single AWS t2.micro instance (Linux). Yes, I've tried restarting the server and creating the index immediately with no luck. I've tried indexing with some existing vertices that use the property key. I've also deleted all vertices and tried creating the index. I get the same result every time. – Appetizer

For what it's worth, deleting all of the vertices won't affect the need for reindexing - the determination is made based on whether or not the property key is in the schema, and the schema in Titan is immutable. I'd definitely try to reproduce by creating an index on property keys that you've never used before (you have to create them in the same management transaction as the index, of course). – Herschel

Tried your recommendation and it also fails. I tried creating a brand new property key and an index on that prop key within the same transaction. Please see my Edit 2. – Appetizer

OK this DID in fact work. I have a working index! So why can't I create an index on an existing property key? Edit 2 is updated. – Appetizer

You should be able to create indexes on existing property keys, but as another answer mentioned you'll need to kick off a reindex before the index is usable. If you're still seeing stale transaction alerts, though, I'd start by trying to track those down. I don't believe the stale transaction cleanup logic will make the index "reindexable". – Herschel

The Evicted stale transaction alert is definitely the problem. That seems to cause the index to get stuck in the installed state during creation. This doesn't happen when creating the property key during the same transaction, only when looking up an existing property key. So the issue is how to solve the Evicted issue, which is where I'm currently stuck. – Appetizer