What's the meaning of NodeDiscoveryType as TOKEN_AWARE in Astyanax client?

Question

I found TOKEN_AWARE enum value in Astyanax client for Cassandra in com.netflix.astyanax.connectionpool.NodeDiscoveryType and am trying to understand what it does?

package com.netflix.astyanax.connectionpool;

public enum NodeDiscoveryType {
    /**
     * Discover nodes exclusively from doing a ring describe
     */
    RING_DESCRIBE,

    /**
     * Discover nodes exclusively from an external node discovery service
     */
    DISCOVERY_SERVICE,

    /**
     * Intersect ring describe and nodes from an external service. This solve
     * the multi-region ring describe problem where ring describe returns nodes
     * from other regions.
     */
    TOKEN_AWARE,

    /**
     * Use only nodes in the list of seeds
     */
    NONE
}

Suppose if I have 24 nodes cross colo cluster with 12 nodes in PHX colo/datacenter and 12 nodes in SLC colo/datacenter.

And I am connecting to Cassandra using Astyanax client as follows:

private CassandraAstyanaxConnection() {
    context = new AstyanaxContext.Builder()
                .forCluster(ModelConstants.CLUSTER)
                .forKeyspace(ModelConstants.KEYSPACE)
    .withConnectionPoolConfiguration(new ConnectionPoolConfigurationImpl("MyConnectionPool")
        .setPort(9160)
        .setMaxConnsPerHost(40)
        .setSeeds("cdb03.vip.phx.host.com:9160,cdb04.vip.phx.host.com:9160")
    )
    .withAstyanaxConfiguration(new AstyanaxConfigurationImpl()      
        .setCqlVersion("3.0.0")
        .setTargetCassandraVersion("1.2")
        .setDiscoveryType(NodeDiscoveryType.TOKEN_AWARE))
    .withConnectionPoolMonitor(new CountingConnectionPoolMonitor())
    .buildKeyspace(ThriftFamilyFactory.getInstance());

    context.start();
    keyspace = context.getEntity();

    emp_cf = ColumnFamily.newColumnFamily(
        ModelConstants.COLUMN_FAMILY, 
        StringSerializer.get(), 
        StringSerializer.get());
}

Can anyone explain me what the difference between TOKEN_AWARE of NodeDiscoveryType vs TOKEN_AWARE of ConnectionPoolType is?

Thanks for the help.

Updated Code

Below is the code I am using so far after making changes-

private CassandraAstyanaxConnection() {

    context = new AstyanaxContext.Builder()
    .forCluster(ModelConstants.CLUSTER)
    .forKeyspace(ModelConstants.KEYSPACE)
    .withConnectionPoolConfiguration(new ConnectionPoolConfigurationImpl("MyConnectionPool")
        .setPort(9160)
        .setMaxConnsPerHost(40)
        .setSeeds("cdb03.vip.phx.host.com:9160,cdb04.vip.phx.host.com:9160")
        .setLocalDatacenter("phx")
    )
    .withAstyanaxConfiguration(new AstyanaxConfigurationImpl()
        .setCqlVersion("3.0.0")
        .setTargetCassandraVersion("1.2")
        .setConnectionPoolType(ConnectionPoolType.TOKEN_AWARE))
    .withConnectionPoolMonitor(new CountingConnectionPoolMonitor())
    .buildKeyspace(ThriftFamilyFactory.getInstance());

    context.start();
    keyspace = context.getEntity();

    emp_cf = ColumnFamily.newColumnFamily(
        ModelConstants.COLUMN_FAMILY, 
        StringSerializer.get(), 
        StringSerializer.get());
}

You mentioned in your example that you will be using-

    .setDiscoveryType(NodeDiscoveryType.RING_DESCRIBE)
    .setConnectionPoolType(ConnectionPoolType.TOKEN_AWARE)

these two together right? But I believe TOKEN_AWARE ConnectionPoolType by default uses RING_DESCRIBE so it doesn't make sense to add it again. Am I right?

Correct me if I am wrong?

Megohm

techy i am little bit confused. In one post you are asking a question and in the next one you are answering the same question. – Guildroy

@abhi, I posted other question before I was aware of this thing. But before posting any thing I try those thing myself and with TOKEN_AWARE NODE DISCOVERY I always get all the nodes across the data center. – Megohm

Answer 1

When it comes to "node discovery" the relationship between TOKEN_AWARE for NodeDiscoveryType and TOKEN_AWARE for ConnectionPoolType is interrelated and somewhat confusing.

NodeDiscoveryType is determined as follows (and it -usually- isn't via setDiscoveryType()):

• If you've provided Seeds via setSeeds and ConnectionPoolType is TOKEN_AWARE then NodeDiscoveryType is RING_DESCRIBE.
• If you've provided Seeds via setSeeds and ConnectionPoolType is anything other than TOKEN_AWARE then your configured setDiscoveryType will be used. This is the only case in which your configured NodeDiscoveryType (via setDiscoveryType) will be used.
• If you did not provide Seeds via setSeeds AND ConnectionPoolType is TOKEN_AWARE then NodeDiscoveryType is TOKEN_AWARE.
• If you did not provide Seeds via setSeeds AND ConnectionPoolType is anything other than TOKEN_AWARE then NodeDiscoveryType is DISCOVERY_SERVICE.

Node Discovery

Now that we've determined how NodeDiscoveryType is set, let's see how it impacts actually discovering nodes. Node discovery boils down to which implementation of HostSupplier (i.e. Supplier<List<Host>>) is used.

• If NodeDiscoveryType (from above) is DISCOVERY_SERVICE then must use HostSupplier (via withHostSupplier).
• If NodeDiscoveryType (from above) is RING_DESCRIBE then use RingDescribeHostSupplier.
• If NodeDiscoveryType (from above) is TOKEN_AWARE and HostSupplier is set (via withHostSupplier) then use FilteringHostSupplier with RingDescribeHostSupplier.
• If NodeDiscoveryType (from above) is TOKEN_AWARE and no HostSupplier is set then use RingDescribeHostSupplier.

RingDescribe and using the local DC

Based on the configuration you've supplied you'll end up with RingDescribeHostSupplier. RingDescribeHostSupplier allows connections to all nodes in the ring unless you've specified a datacenter. So, when setting up your AstyanaxContext using ConnectionPoolConfigurationImpl you might want to setLocalDatacenter with the desired DC. That will ensure that hosts from the other dc's are not in the connection pool and that your requests are local.

.withConnectionPoolConfiguration(new ConnectionPoolConfigurationImpl("MyConnectionPool")
        .setPort(9160)
        .setMaxConnsPerHost(40)
        .setLocalDatacenter("phx")
        .setSeeds("cdb03.vip.phx.host.com:9160,cdb04.vip.phx.host.com:9160")
    )

ConnectionPoolType

You also might want to set ConnectionPoolType to TOKEN_AWARE. When that value is left unset, it will default to ROUND_ROBIN (using the nodes from the node discovery work described above). TOKEN_AWARE ConnectionPoolType will "keep track of which hosts have which tokens and attempt to direct traffic intelligently".

I'd do something like this for Astyanax configuration, unless you are providing a HostSupplier.

.withAstyanaxConfiguration(new AstyanaxConfigurationImpl()      
        .setDiscoveryType(NodeDiscoveryType.RING_DESCRIBE)
        .setConnectionPoolType(ConnectionPoolType.TOKEN_AWARE)
    )

Pool Optimizations

Another consideration would be optimizing the pool usage with Astyanax "latency awareness" on ConnectionPoolConfigurationImpl, but YMMV on the settings. e.g. :

.setLatencyScoreStrategy(new SmaLatencyScoreStrategyImpl(10000,10000,100,0.50))
// The constructor takes:
//  UpdateInterval: 10000 : Will resort hosts per token partition every 10 seconds
//  ResetInterval: 10000 : Will clear the latency every 10 seconds
//  WindowSize: 100 : Uses last 100 latency samples
//  BadnessThreshold: 0.50 : Will sort hosts if a host is more than 100% 

See Astyanax Configuration

TLDR;

In summary, set NodeDiscoveryType to RING_DESCRIBE (if you aren't using a HostSupplier) and ConnectionPoolType to TOKEN_AWARE. Additionally, use setLocalDatacenter to keep requests local to the dc and consider the latency awareness settings.

Argumentation

@TechGeeky said- Ok cool. With my current setup right now for Cassandra database in production, I have cross colo cluster with 24 nodes, 12 nodes in SLC and 12 nodes in PHX. And replication factor is of 4 so that means 2 copies of same data will be there in each colo. Now with the current configuration of Astyanax that I am using currently which will autodiscover the nodes only for each datacenter right? Currently it is PHX. – TechGeeky – Argumentation

@TechGeeky said- continuation from above.... Suppose I am looking for rowKey 123 then it will go and get the data from two replicas in PHX colo, right? Then what will happen in that case, suppose two nodes in PHX that will contain the data for rowKey 123 are down? Then it will go and get the data from SLC colo replicas nodes? Am I correct? – TechGeeky – Argumentation

That is my understanding, but I haven't proved it. I'm in the middle of learning/coding/deploying this stuff as well, so I'll report back (as my setup will be quite similar). Also, I'm going to kill this post and switch it back from community wiki... cause I'm an idiot (it probably doesn't warrant being wiki style). – Argumentation

Thanks m4tty for the suggestion. Let me know what do you find out with the use case in which the data present in two nodes is down. – Megohm

One thing to note: The "latency" settings should not be set on the Connection Pool Config object via .setLatencyAwareXX methods, as Astyanax has changed. Instead the latency values are supplied to the constructor of SmaLatencyScoreStrategyImpl (or whichever strategy is used) and this object should be supplied to the .setLatencyScoreStrategy(). I've changed the example code to reflect that. – Argumentation

It seems that the BadnessThreshold setting for badness for 100% should be 2, see line 183 of AbstractLatencyScoreStrategyImpl – Webster

Answer 2

Implementation wise ConnectionPoolType.TOKEN_AWARE

Connection pool that partitions connections by the hosts which own the token being operated on. When a token is not available or an operation is known to span multiple tokens (such as a batch mutate or an index query) host pools are picked using round robin.

Where as NodeDiscoveryType.TOKEN_AWARE

Re-discover the ring describe information in a multi data-center scenario.

Guildroy