CookieManager for multiple threads

Question

I am trying to make multiple connections via threads.

But every connection seems to override the other's cookies, resulting in the connections using the wrong cookies.

inside the threaded class's constructor:

    manager = new CookieManager();
    manager.setCookiePolicy(CookiePolicy.ACCEPT_ALL);
    CookieHandler.setDefault(manager);

Any way to manage the cookies per thread or per class?

New failed try:

Now every thread is using it's own index, yet they still seem to override each other cookie-wise. Any ideas?

public class threadedCookieStore implements CookieStore, Runnable {
    CookieStore[] store = new CookieStore[1000];
    int index;

    public threadedCookieStore(int new_index) {
        index = new_index;
        // get the default in memory cookie store
        store[index] = new CookieManager().getCookieStore();


        // todo: read in cookies from persistant storage
        // and add them store

        // add a shutdown hook to write out the in memory cookies
        Runtime.getRuntime().addShutdownHook(new Thread(this)); 
    }

    public void run() {
        // todo: write cookies in store to persistent storage
    }

    public void add(URI uri, HttpCookie cookie) {
        store[index].add(uri, cookie);
    }

    public List<HttpCookie> get(URI uri) {
        return store[index].get(uri);
    }

    public List<HttpCookie> getCookies() {
        return store[index].getCookies();
    }

    public List<URI> getURIs() {
        return store[index].getURIs();
    }

    public boolean remove(URI uri, HttpCookie cookie) {
        return store[index].remove(uri, cookie);
    }

     public boolean removeAll()  {
         return store[index].removeAll();
    }
}

Within the class:

threadedCookieStore cookiestore = new threadedCookieStore(index);

manager = new CookieManager(cookiestore,CookiePolicy.ACCEPT_ALL);
manager.setCookiePolicy(CookiePolicy.ACCEPT_ALL);
CookieHandler.setDefault(manager);

Systematology

I can't believe how bad the java.net team are at writing APIs. Why can't we just set the cookie manager per-connection? :( You'd think they would have learned their mistake after AuthenticationHandler. – Brunhild

A better solution can be found here – Ylem

Answer 1

Thanks everyone.

I upvoted all the answers, yet none had a complete solution.

Since google'ing this problem leads to this page here, I'll post the complete solution and accept my own answer:

HowTo:

1 Extend CookieHandler to SessionCookieManager

this is based on How to use different cookies for each connection using HttpURLConnection and the CookieManager in Java , nivs describes it correctly, doesn't provide a full solution tho. So most/all credit goes to him, I'm just making the complete HowTo. The SessionCookieManager is based on Java 's source code http://docs.oracle.com/javase/7/docs/api/java/net/CookieManager.html

import java.io.IOException;
import java.net.CookieHandler;
import java.net.CookiePolicy;
import java.net.CookieStore;
import java.net.HttpCookie;
import java.net.URI;
import java.util.Collections;
import java.util.Comparator;
import java.util.List;
import java.util.Map;

public class SessionCookieManager extends CookieHandler
{
    private CookiePolicy policyCallback;


    public SessionCookieManager() {
        this(null, null);
    }

    private final static SessionCookieManager ms_instance = new SessionCookieManager();

    public static SessionCookieManager getInstance()
    {
        return ms_instance;
    }

    private final static ThreadLocal<CookieStore> ms_cookieJars = new ThreadLocal<CookieStore>() {
        @Override
        protected synchronized CookieStore initialValue() { return new InMemoryCookieStore(); }
    };

    public void clear()
    {
        getCookieStore().removeAll();
    }
    public SessionCookieManager(CookieStore store,
                         CookiePolicy cookiePolicy)
    {
        // use default cookie policy if not specify one
        policyCallback = (cookiePolicy == null) ? CookiePolicy.ACCEPT_ALL //note that I changed it to ACCEPT_ALL
                                                : cookiePolicy;

        // if not specify CookieStore to use, use default one

    }
    public void setCookiePolicy(CookiePolicy cookiePolicy) {
        if (cookiePolicy != null) policyCallback = cookiePolicy;
    }

    public CookieStore getCookieStore() {
        return ms_cookieJars.get();
    }

    public Map<String, List<String>>
        get(URI uri, Map<String, List<String>> requestHeaders)
        throws IOException
    {
        // pre-condition check
        if (uri == null || requestHeaders == null) {
            throw new IllegalArgumentException("Argument is null");
        }

        Map<String, List<String>> cookieMap =
                        new java.util.HashMap<String, List<String>>();
        // if there's no default CookieStore, no way for us to get any cookie
        if (getCookieStore() == null)
            return Collections.unmodifiableMap(cookieMap);

        List<HttpCookie> cookies = new java.util.ArrayList<HttpCookie>();
        for (HttpCookie cookie : getCookieStore().get(uri)) {
            // apply path-matches rule (RFC 2965 sec. 3.3.4)
            if (pathMatches(uri.getPath(), cookie.getPath())) {
                cookies.add(cookie);
            }
        }

        // apply sort rule (RFC 2965 sec. 3.3.4)
        List<String> cookieHeader = sortByPath(cookies);

        cookieMap.put("Cookie", cookieHeader);
        return Collections.unmodifiableMap(cookieMap);
    }


    public void
        put(URI uri, Map<String, List<String>> responseHeaders)
        throws IOException
    {
        // pre-condition check
        if (uri == null || responseHeaders == null) {
            throw new IllegalArgumentException("Argument is null");
        }


        // if there's no default CookieStore, no need to remember any cookie
        if (getCookieStore() == null)
            return;

        for (String headerKey : responseHeaders.keySet()) {
            // RFC 2965 3.2.2, key must be 'Set-Cookie2'
            // we also accept 'Set-Cookie' here for backward compatibility
            if (headerKey == null
                || !(headerKey.equalsIgnoreCase("Set-Cookie2")
                     || headerKey.equalsIgnoreCase("Set-Cookie")
                    )
                )
            {
                continue;
            }

            for (String headerValue : responseHeaders.get(headerKey)) {
                try {
                    List<HttpCookie> cookies = HttpCookie.parse(headerValue);
                    for (HttpCookie cookie : cookies) {
                        if (shouldAcceptInternal(uri, cookie)) {
                            getCookieStore().add(uri, cookie);
                        }
                    }
                } catch (IllegalArgumentException e) {
                    // invalid set-cookie header string
                    // no-op
                }
            }
        }
    }


    /* ---------------- Private operations -------------- */

    // to determine whether or not accept this cookie
    private boolean shouldAcceptInternal(URI uri, HttpCookie cookie) {
        try {
            return policyCallback.shouldAccept(uri, cookie);
        } catch (Exception ignored) { // pretect against malicious callback
            return false;
        }
    }


    /*
     * path-matches algorithm, as defined by RFC 2965
     */
    private boolean pathMatches(String path, String pathToMatchWith) {
        if (path == pathToMatchWith)
            return true;
        if (path == null || pathToMatchWith == null)
            return false;
        if (path.startsWith(pathToMatchWith))
            return true;

        return false;
    }


    /*
     * sort cookies with respect to their path: those with more specific Path attributes
     * precede those with less specific, as defined in RFC 2965 sec. 3.3.4
     */
    private List<String> sortByPath(List<HttpCookie> cookies) {
        Collections.sort(cookies, new CookiePathComparator());

        List<String> cookieHeader = new java.util.ArrayList<String>();
        for (HttpCookie cookie : cookies) {
            // Netscape cookie spec and RFC 2965 have different format of Cookie
            // header; RFC 2965 requires a leading $Version="1" string while Netscape
            // does not.
            // The workaround here is to add a $Version="1" string in advance
            if (cookies.indexOf(cookie) == 0 && cookie.getVersion() > 0) {
                cookieHeader.add("$Version=\"1\"");
            }

            cookieHeader.add(cookie.toString());
        }
        return cookieHeader;
    }


    static class CookiePathComparator implements Comparator<HttpCookie> {
        public int compare(HttpCookie c1, HttpCookie c2) {
            if (c1 == c2) return 0;
            if (c1 == null) return -1;
            if (c2 == null) return 1;

            // path rule only applies to the cookies with same name
            if (!c1.getName().equals(c2.getName())) return 0;

            // those with more specific Path attributes precede those with less specific
            if (c1.getPath().startsWith(c2.getPath()))
                return -1;
            else if (c2.getPath().startsWith(c1.getPath()))
                return 1;
            else
                return 0;
        }
    }
}

Note that in my case I changed the default value of CookiePolicy to ACCEPT_ALL

2 In global scope, before running any threads, call:

CookieHandler.setDefault(SessionCookieManager.getInstance());

3 When your thread is finished, call inside of it:

SessionCookieManager.getInstance().clear();

again: not my idea, just putting it together. All credit goes to Java and https://stackoverflow.com/users/1442259/nivs

Systematology

Answer 2

Thanks, I tried to use your answer, but it was based on an old version of CookieManager (probably why you had to use ACCEPT_ALL) and referenced the package-private InMemoryCookieStore so it inspired me to the final solution. Should have been obvious to all of us before: a ThreadLocal CookieStore proxy class.

CookieHandler.setDefault(new CookieManager(new ThreadLocalCookieStore(), null));

with

import java.net.CookieManager;
import java.net.CookieStore;
import java.net.HttpCookie;
import java.net.URI;
import java.util.List;

public class ThreadLocalCookieStore implements CookieStore {

    private final static ThreadLocal<CookieStore> ms_cookieJars = new ThreadLocal<CookieStore>() {
        @Override
        protected synchronized CookieStore initialValue() {
            return (new CookieManager()).getCookieStore(); /*InMemoryCookieStore*/
            }
    };

    @Override
    public void add(URI uri, HttpCookie cookie) {
        ms_cookieJars.get().add(uri, cookie);
    }

    @Override
    public List<HttpCookie> get(URI uri) {
        return ms_cookieJars.get().get(uri);
    }

    @Override
    public List<HttpCookie> getCookies() {
        return ms_cookieJars.get().getCookies();
    }

    @Override
    public List<URI> getURIs() {
        return ms_cookieJars.get().getURIs();
    }

    @Override
    public boolean remove(URI uri, HttpCookie cookie) {
        return ms_cookieJars.get().remove(uri, cookie);
    }

    @Override
    public boolean removeAll() {
        return ms_cookieJars.get().removeAll();
    }
}

Seems to be working like a charm for me

Ammonium

Answer 3

Based on the answers in this thread, I created another very simple ThreadLocalCookieStore implementation and pushed it to GitHub (also providing it as a Maven dependency there):

public class ThreadLocalCookieStore implements CookieStore {
    private final static ThreadLocal<CookieStore> stores = new ThreadLocal<CookieStore>() {
        @Override protected synchronized CookieStore initialValue() { 
            return (new CookieManager()).getCookieStore(); //InMemoryCookieStore 
        }
    };

    @Override public void add(URI uri, HttpCookie cookie) { getStore().add(uri,cookie); }
    @Override public List<HttpCookie> get(URI uri) { return getStore().get(uri); }
    @Override public List<HttpCookie> getCookies() { return getStore().getCookies(); }
    @Override public List<URI> getURIs() { return getStore().getURIs(); }
    @Override public boolean remove(URI uri, HttpCookie cookie) { return getStore().remove(uri,cookie); }
    @Override public boolean removeAll() { return getStore().removeAll(); }
    @Override public int hashCode() { return getStore().hashCode(); }

    protected CookieStore getStore() { return stores.get(); }
    public void purgeStore() { stores.remove(); }
}

Without much code it becomes very simple to set the cookie store, with any policy value, e.g.:

CookieHandler.setDefault(new java.net.CookieManager(
    new ThreadLocalCookieStore(), CookiePolicy.ACCEPT_ALL));

In addition the dependency features a small sevlet @WebFilter, to separate cookie stores on multiple serlvet requests if required.

Forebode

Answer 4

You could install a CookieHandler which manages ThreadLocal CookieManager instances.

Disobedient

Don't know where to start here. Any examples? – Systematology

@Systematology - Create a CookieHandler where the methods delegate to a CookieHandler instance which is maintained in a ThreadLocal. – Disobedient

Answer 5

The ThreadLocal CookieStore by DavidBlackledge is imo the best way to go. For the sake of memory efficiency I'm providing here a simple implementation of a regular CookieStore so you don't have to instantiate a whole CookieManager for each thread (assuming you have more than just a few).

/**
 * @author lidor
 * A simple implementation of CookieStore
 */
public class CookieJar implements CookieStore {

private Map<URI, List<HttpCookie>> jar;
private List<HttpCookie> freeCookies;

public CookieJar() {
    jar = new HashMap<URI, List<HttpCookie>>();
    freeCookies = new ArrayList<HttpCookie>();
}

@Override
public void add(URI uri, HttpCookie cookie) {
    if (uri != null) {
        if (!jar.containsKey(uri))
            jar.put(uri, new ArrayList<HttpCookie>());
        List<HttpCookie> cookies = jar.get(uri);
        cookies.add(cookie);
    } else {
        freeCookies.add(cookie);
    }
}

@Override
public List<HttpCookie> get(URI uri) {
    Log.trace("CookieJar.get (" + this + ") called with URI " + uri + " (host=" + uri.getHost() + ")");
    List<HttpCookie> liveCookies = new ArrayList<HttpCookie>();
    if (jar.containsKey(uri)) {
        for (HttpCookie cookie : jar.get(uri)) {
            if (!cookie.hasExpired())
                liveCookies.add(cookie);
        }
    }
    for (HttpCookie cookie : getCookies()) {
        if (cookie.getDomain().equals(uri.getHost()))
            if (!liveCookies.contains(cookie))
                liveCookies.add(cookie);
    }
    return Collections.unmodifiableList(liveCookies);
}

@Override
public List<HttpCookie> getCookies() {
    List<HttpCookie> liveCookies = new ArrayList<HttpCookie>();
    for (URI uri : jar.keySet())
        for (HttpCookie cookie : jar.get(uri)) {
            if (!cookie.hasExpired())
                liveCookies.add(cookie);
        }
    for (HttpCookie cookie : freeCookies) {
        if (!cookie.hasExpired())
            liveCookies.add(cookie);
    }
    return Collections.unmodifiableList(liveCookies);
}

@Override
public List<URI> getURIs() {
    return Collections.unmodifiableList(new ArrayList<URI>(jar.keySet()));
}

@Override
public boolean remove(URI uri, HttpCookie cookie) {
    if (jar.containsKey(uri)) {
        return jar.get(uri).remove(cookie);
    } else {
        return freeCookies.remove(cookie);
    }
}

@Override
public boolean removeAll() {
    boolean ret = (jar.size() > 0) || (freeCookies.size() > 0);
    jar.clear();
    freeCookies.clear();
    return ret;
}
}

So if you have this CookieJar then you can change the ms_cookieJars declaration to this:

private final static ThreadLocal<CookieStore> ms_cookieJars = new ThreadLocal<CookieStore>() {
        @Override
        protected synchronized CookieStore initialValue() { 
            return new CookieJar();
        }
    };

Baldridge

Answer 6

You can set a different path for the cookies. Thus it will not be overwritten.

http://docs.oracle.com/javase/6/docs/api/java/net/HttpCookie.html#setPath%28java.lang.String%29

Triarchy

sounds promising. Since I am not using single Cookies, where would I set this? – Systematology

Instead of default CookieManager constructor, use the one that accepts CookieStore. You can define your own CookieStore to achive this. The CookieManager javadoc provides the details. docs.oracle.com/javase/6/docs/api/java/net/CookieStore.html – Triarchy

by defining a new class with the required methods? Do you know of an already made one? Would I have to extend the CookieStore or what would be the right approach? – Systematology

CookieStore is an interface. You need to implement and have your own CookieStore. – Triarchy

You can check this page for custom cookie store implementation docs.oracle.com/javase/tutorial/networking/cookies/custom.html – Triarchy

Please check out the edit of my question. I tried to use your suggestion but am unable to make it work. – Systematology

I downvoted as the setPath method seems to be for the actual cookie domain and path, not where the cookie is persisted in Java. So if there's two threads both getting the same URL, each cookie with same path will still overwrite. – Stork

Answer 7

How about a ThreadLocal CookieManager? Same idea as some of the other answers but seems to require less code:

public class ThreadLocalCookies extends CookieManager {

    private static CookiePolicy s_policy = null;
    private static ThreadLocal<CookieManager> s_impl =
        new ThreadLocal<CookieManager>() {
            @Override protected CookieManager initialValue() {
                if (null == s_policy) {
                    throw new IllegalStateException("Call install() first");
                }
                return new CookieManager(null, s_policy);
            }
        };

    public static void install() {
        install(CookiePolicy.ACCEPT_ALL);
    }

    public static void install(CookiePolicy policy) {
        s_policy = policy;
        CookieHandler.setDefault(new ThreadLocalCookies());
    }

    public static void clear() {
        s_impl.set(new CookieManager(null, s_policy));
    }

    @Override
    public Map<String, List<String>>
    get(URI uri, Map<String, List<String>> requestHeaders) 
    throws IOException {
        return s_impl.get().get(uri, requestHeaders);
    }

    @Override
    public void put(URI uri, Map<String,List<String>> responseHeaders)
    throws IOException {
        s_impl.get().put(uri, responseHeaders);
    }
}

Nessie