Items set with spymemcached cannot be fetched with php memcached

Question

I am using spymemcached. I set a couple of items. Then I run a php script, however then I cannot get all those items using php memcached. PHP-Memcached can only partially retrieve those items.

I cannot change php's hashing algorithm or distribution strategy. In our system we are using default hashing (which is jenkin's one-at-a-time according to php.net documentation). And distribution strategy is modulo for php-memcached. I have read that spymemcached uses consistent hashing. Is there any way by which I can use modulo hashing in spymemcached.

In other words how can I make spymemcached's set operations or any other store operations compatible with php-memcached's get operations?

If spymemcached is not able to do that, are there any other memcached client in java that will allow me to do so?

Help will not only be appreciated, it will also be rewarded a bounty.

Java code:

public static void main(String [] args) {
    List<InetSocketAddress> addrs = new ArrayList<>();
    addrs.add(new InetSocketAddress("10.90.12.87", 11211));
    addrs.add(new InetSocketAddress("10.90.12.87", 11311));
    try {
        MemcachedClient memcache = new MemcachedClient(addrs);
        memcache.add("foo", 0, "bar");
        memcache.add("sample", 0, "key");
        memcache.add("try", 0, "another");
        memcache.add("ax-spadg-list", 0, "3045,6645");
    } catch (IOException ex) {
        Logger.getLogger(CategoryDataOperator.class.getName()).log(Level.SEVERE, null, ex);
    }
    System.out.println("Done");
}

PHP code:

<?php
$mem = new Memcached();
$mem->addServer('10.90.12.87', 11211);
$mem->addServer('10.90.12.87', 11311);
var_dump $mem->get('foo');
var_dump($mem->get('try'));
var_dump($mem->get('sample'));
var_dump($mem->get('ax-spadg-list'));

Diazotize

You might be interested to read a bit here: Java Memcached Client. – Wolsky

I am already using it. However items set with spymemcached are giving me trouble while retrieving them using php memcached. Answer for which I am seeking – Diazotize

That I have perfectly understood, but see the user who answered there he wrote that he is the author of Spymemcachedn so probably if you ping him you get some more infos. He perhaps knows about that problem. – Wolsky

yes, but I don't know how to contact him. His profile has no contact information. Plus I had commented twice, but he has not gotten back at it. – Diazotize

Sorry for multi edition but I didn't read correctly your question and requirement that php client can't change. I offer you 3 solutions. – Chokedamp

Answer 1

The problem is about Hash, the default php-memcached hash is

(Jenkins one-at-a-time) item key hashing algorithm

whereas the list of spymemcached hashes are:

• NATIVE_HASH: simply Native hash (String.hashCode()). does not match with default php-memcached Memcached::HASH_DEFAULT
• CRC_HASH => Memcached::HASH_CRC
• FNV1_64_HASH => Memcached::HASH_FNV1_64
• FNV1A_64_HASH => Memcached::HASH_FNV1A_64
• FNV1_32_HASH => Memcached::HASH_FNV1_32
• FNV1A_32_HASH => Memcached::HASH_FNV1A_32
• KETAMA_HASH => "MD5-based hash algorithm used by ketama." So maybe Memcached::HASH_MD5 but anyway is not Memcached::HASH_DEFAULT

So there is not direct matching between the two libs if you can't change PHP client configuration or extend spymemcached lib.

SOLUTION 1: If you look on history (you can have an example with php client hash modification).

SOLUTION 2: Else you can create a JenkinHash class (I copy past the Xmemcached code: https://code.google.com/p/xmemcached/source/browse/trunk/src/main/java/net/rubyeye/xmemcached/HashAlgorithm.java?r=801#176 [But take in consideration the Xmemcached Licenses and keep the author/license inside the source code])

import net.spy.memcached.HashAlgorithm;

import java.io.UnsupportedEncodingException;

public class JenkinsHash implements HashAlgorithm {
    @Override
    public long hash(String k) {
        try {
            int hash = 0;
            for (byte bt : k.getBytes("utf-8")) {
                hash += (bt & 0xFF);
                hash += (hash << 10);
                hash ^= (hash >>> 6);
            }
            hash += (hash << 3);
            hash ^= (hash >>> 11);
            hash += (hash << 15);
            return hash;
        } catch (UnsupportedEncodingException e) {
            throw new IllegalStateException("Hash function error", e);
        }
    }
}

then:

import net.spy.memcached.*;

import java.io.IOException;
import java.net.InetSocketAddress;
import java.util.ArrayList;
import java.util.List;
import java.util.logging.Level;
import java.util.logging.Logger;

public class Main {

    public static void main(String[] args) throws IOException {
        List<InetSocketAddress> addrs = new ArrayList<InetSocketAddress>();
        addrs.add(new InetSocketAddress("127.0.0.1", 11211));
        addrs.add(new InetSocketAddress("172.28.29.22", 11211));
        try {
            ConnectionFactory connectionFactory = new ConnectionFactoryBuilder()
                .setProtocol(ConnectionFactoryBuilder.Protocol.TEXT)
                .setHashAlg(new JenkinsHash())
                .setLocatorType(ConnectionFactoryBuilder.Locator.ARRAY_MOD).build();
            MemcachedClient memcache = new MemcachedClient(connectionFactory, addrs);
            memcache.add("foo", 0, "bar2");
            memcache.add("sample", 0, "key");
            memcache.add("try", 0, "another");
            memcache.add("ax-spadg-list", 0, "3045,6645");
        } catch (IOException ex) {
            Logger.getLogger(Main.class.getName()).log(Level.SEVERE, null, ex);
        }
        System.out.println("Done");
    }
}

With php script:

<?php

$memcached = new Memcached();
$memcached->addserver('127.0.0.1', 11211);
$memcached->addserver('172.28.29.22', 11211);
var_dump($memcached->get('foo'));
var_dump($memcached->get('try'));
var_dump($memcached->get('sample'));
var_dump($memcached->get('ax-spadg-list'));

test:

$ echo "flush_all" | nc 172.28.29.22 11211 && echo "flush_all" | nc 127.0.0.1 11211
OK
OK
$ php mem.php 
bool(false)
bool(false)
bool(false)
bool(false)

RUN JAVA

$ php mem.php 
string(4) "bar2"
string(7) "another"
string(3) "key"
string(9) "3045,6645"

SOLUTION 3: use https://code.google.com/p/xmemcached/ with ONE_AT_A_TIME hash algorithm

import net.rubyeye.xmemcached.HashAlgorithm;
import net.rubyeye.xmemcached.MemcachedClient;
import net.rubyeye.xmemcached.MemcachedClientBuilder;
import net.rubyeye.xmemcached.XMemcachedClientBuilder;
import net.rubyeye.xmemcached.exception.MemcachedException;
import net.rubyeye.xmemcached.impl.ArrayMemcachedSessionLocator;

import java.io.IOException;
import java.net.InetSocketAddress;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.TimeoutException;

public class Main {

    public static void main(String[] args) throws IOException, InterruptedException, MemcachedException, TimeoutException {
        List<InetSocketAddress> addrs = new ArrayList<InetSocketAddress>();
        addrs.add(new InetSocketAddress("127.0.0.1", 11211));
        addrs.add(new InetSocketAddress("172.28.29.22", 11211));
        MemcachedClientBuilder builder = new XMemcachedClientBuilder(addrs);
        builder.setSessionLocator(new ArrayMemcachedSessionLocator(HashAlgorithm.ONE_AT_A_TIME));
        MemcachedClient memcachedClient = builder.build();
        memcachedClient.set("foo", 0, "bar2");
        memcachedClient.set("sample", 0, "key");
        memcachedClient.set("try", 0, "another");
        memcachedClient.set("ax-spadg-list", 0, "3045,6645");
        memcachedClient.shutdown();
        System.out.println("Done");
    }
}

Chokedamp

Took me long to get back at you. I tried xmemcached solution. However, it still doesn't work. It tries to locate item on only one server. I set items from php, and try to get it in java. Throws ArrayIndexOutOfBounds exception! Is distribution strategy messed up? – Diazotize

cannot post here. Code will look messed up. Can we chat somewhere? I am stuck in this project for way too long. And it seems only you can help me. Could you please send me your email-id or any other means of contacting you? – Diazotize

MemcachedClientBuilder builder = new xmemcachedClientBuilder(AddrUtil.getAddresses( "127.0.0.1:11211 127.0.0.1:11311" )); builder.setSessionLocator(new ArrayMemcachedSessionLocator(HashAlgorithm.ONE_AT_A_TIME)); MemcachedClient memcachedClient = builder.build(); System.out.println("Trying to get 1 milllion items"); System.out.println(memcachedClient.get("foo")); System.out.println(memcachedClient.get("bar")); System.out.println(memcachedClient.get("yin")); – Diazotize

Tomorrow for chating i can't today – Chokedamp

May use gist or something else :) – Chokedamp

let us continue this discussion in chat – Chokedamp

Answer 2

Hash algorithms that spymemcached supports are here: https://github.com/couchbase/spymemcached/blob/master/src/main/java/net/spy/memcached/DefaultHashAlgorithm.java

You should be able to change the hash algorithm by using a ConnectionFactory to create your MemcachedClient. Do something like this:

ConnectionFactoryBuilder builder = new ConnectionFactoryBuilder();
builder.setHashAlgorithm(HashAlgorithm.CRC_HASH);
ConnectionFactory factory = builder.build();
MemcachedClient client = new MemcachedClient(Arrays.asList(new InetSocketAddr("localhost", 11211)), factory);

Shantung

I am showing cannot find symbol: CRC32. Is there any other jar or something I need to add for CRC32? – Diazotize

I edited the answer to include one of the hash types included in the file above. The code above should be used as a reference. I didn't actually compile it. Note in the file other possible hash algorithms are NATIVE_HASH, FNV1_64_HASH, KETAMA_HASH, etc. – Shantung

Thanks for that. However HashAlgorithm is an interface, so it didn't have fields you mentioned. I used DefaultHashAlgorithm.CRC_HASH. However I tried all algorithms. Nothing worked :( – Diazotize

in php memcached hashing is set to default that is jenkin's one by one as specified on php.net. I don't know what is default hashing in spymemcached. also php memcached has modulo as well as consistent as default distribution strategy. modulo is default. I chose ConnectionFactoryBuilder.Locator.ARRAY_MOD. But still no luck – Diazotize

Answer 3

Re: Kakawait (and also to Shades88)

Solution #2 is incorrect because xmemcached didn't properly port the original C code of the Jenkins hash, which uses an unsigned. Fixing this will also solve the ArrayIndexOutOfBoundsException that Shades88 was seeing.

public class JenkinsHash implements HashAlgorithm {
    @Override
    public long hash(String k) {
        try {
            int hash = 0;
            for (byte bt : k.getBytes("utf-8")) {
                hash += (bt & 0xFF);
                hash += (hash << 10);
                hash ^= (hash >>> 6);
            }
            hash += (hash << 3);
            hash ^= (hash >>> 11);
            hash += (hash << 15);

            // the hash variable in the original C code is a uint32.
            // convert the java signed int to an "unsigned",
            // represented via a long:
            return hash & 0xFFFFFFFFl;
        } catch (UnsupportedEncodingException e) {
            throw new IllegalStateException("Hash function error", e);
        }
    }
}

// Unit test
public class JenkinsHashTest {
    @Test
    public void testHash() throws Exception {
        JenkinsHash j = new JenkinsHash();
        Properties p = new Properties();

        // This file contains k/v mappings,
        // with values generated by the reference C code
        p.load(new FileReader("src/test/resources/jenkinsHashTest.dat"));

        for (Entry<Object, Object> entry : p.entrySet()) {
            long result = j.hash((String)entry.getKey());
            // Print out hash mismatches
            if (result != Long.parseLong((String)entry.getValue())) {
                System.out.println("Key: " + (String)entry.getKey());
                System.out.println("Expected Hash Value: " + Long.parseLong((String)entry.getValue()));
                System.out.println("Actual Hash Value: " + result);
            }
            assertEquals(result, Long.parseLong((String)entry.getValue()));
        }
    }
}

The test data file is to compare the Java code against the C code. Build the C code then hash a bunch of random words and map them in a file, like this:

jenkinsHashTest.dat:

sausage=2834523395
blubber=1103975961
pencil=3318404908
cloud=670342857
moon=2385442906
water=3403519606
computer=2375101981
school=1513618861
network=2981967937
hammer=1218821080

... add as many as you want

Danyluk

Yes, I figured it out a while back and just used abs function on hash variable. Will typecasting it to long will make it a bit faster? I will try this as well. – Diazotize

The code I added converts the hash variable back to unsigned, which is what you need for equivalence with the C code. Using abs() will prevent the ArrayIndexOutOfBoundsException but will give different hash results, which may cause a compatibility problem between Java and PHP. The original C code I looked at is here: en.wikipedia.org/wiki/Jenkins_hash_function – Danyluk

Last note: abs() will not cause a memcached problem until your Java code actually hits a key that hashes to a negative number. That's when the PHP incompatibility will occur. – Danyluk

Fix is now in xmemcached 2.0.0 (github.com/killme2008/xmemcached/releases/tag/xmemcached-2.0.0) – Danyluk