ZooKeeper for Java/Spring Config?

Question

Are there any well documented use cases of Apache ZooKeeper being used to distribute configuration of Java applications, and in particular Spring services?

Like many users of cloud services I have a requirement to change the configuration of a variable amount of Java services, preferably at run-time without needing to restart the services.

UPDATE

Eventually I ended up writing something that would load a ZooKeeper node as a properties file, and create a ResourcePropertySource and insert it into a Spring context. Note that this will not reflect changes in the ZooKeeper node after the context has started.

public class ZooKeeperPropertiesApplicationContextInitializer implements ApplicationContextInitializer<ConfigurableApplicationContext> {
    private static final Logger logger = LoggerFactory.getLogger(ZooKeeperPropertiesApplicationContextInitializer.class);

    private final CuratorFramework curator;
    private String projectName;
    private String projectVersion;

    public ZooKeeperPropertiesApplicationContextInitializer() throws IOException {
        logger.trace("Attempting to construct CuratorFramework instance");

        RetryPolicy retryPolicy = new ExponentialBackoffRetry(10, 100);
        curator = CuratorFrameworkFactory.newClient("zookeeper", retryPolicy);
        curator.start();
    }

    /**
     * Add a primary property source to the application context, populated from
     * a pre-existing ZooKeeper node.
     */
    @Override
    public void initialize(ConfigurableApplicationContext applicationContext) {
        logger.trace("Attempting to add ZooKeeper-derived properties to ApplicationContext PropertySources");

        try {
            populateProjectProperties();
            Properties properties = populatePropertiesFromZooKeeper();
            PropertiesPropertySource propertySource = new PropertiesPropertySource("zookeeper", properties);
            applicationContext.getEnvironment().getPropertySources().addFirst(propertySource);

            logger.debug("Added ZooKeeper-derived properties to ApplicationContext PropertySources");
            curator.close();
        } catch (IOException e) {
            logger.error("IO error attempting to load properties from ZooKeeper", e);
            throw new IllegalStateException("Could not load ZooKeeper configuration");
        } catch (Exception e) {
            logger.error("IO error attempting to load properties from ZooKeeper", e);
            throw new IllegalStateException("Could not load ZooKeeper configuration");
        } finally {
            if (curator != null && curator.isStarted()) {
                curator.close();
            }
        }
    }

    /**
     * Populate the Maven artifact name and version from a property file that
     * should be on the classpath, with values entered via Maven filtering.
     * 
     * There is a way of doing these with manifests, but it's a right faff when
     * creating shaded uber-jars.
     * 
     * @throws IOException
     */
    private void populateProjectProperties() throws IOException {
        logger.trace("Attempting to get project name and version from properties file");

        try {
            ResourcePropertySource projectProps = new ResourcePropertySource("project.properties");
            this.projectName = (String) projectProps.getProperty("project.name");
            this.projectVersion = (String) projectProps.getProperty("project.version");
        } catch (IOException e) {
            logger.error("IO error trying to find project name and version, in order to get properties from ZooKeeper");
        }
    }

    /**
     * Do the actual loading of properties.
     * 
     * @return
     * @throws Exception
     * @throws IOException
     */
    private Properties populatePropertiesFromZooKeeper() throws Exception, IOException {
        logger.debug("Attempting to get properties from ZooKeeper");

        try {
            byte[] bytes = curator.getData().forPath("/distributed-config/" + projectName + "/" + projectVersion);
            InputStream in = new ByteArrayInputStream(bytes);
            Properties properties = new Properties();
            properties.load(in);
            return properties;
        } catch (NoNodeException e) {
            logger.error("Could not load application configuration from ZooKeeper as no node existed for project [{}]:[{}]", projectName, projectVersion);
            throw e;
        }

    }

}

Swinney

What kind of config changes do you want to push? Basic stuff like strings/ints/booleans? Or object wiring/DI changes? – Sidhu

To be frank, the most common change would be logging levels, although system admins have expressed an interest in things liek connection URLs for data stores. – Swinney

Could you please provide your ResourcePropertySource and context.xml examples? Or do you use ConfigurableApplicationContext? I can't get context started because of properties absence. Thanks in advance! – Raconteur

Sorry Anton, I don't have access to the code any more. – Swinney

read this: dzone.com/articles/spring-cloud-config-part-3-zookeeper-backend – Countrified

Answer 1

I was at an Apache Camel talk from James Strachen last week and he mentioned using ZooKeeper under the covers for their Java-based server in the cloud as the source of configuration info.

I've seen a talk from Adrian Colyer (CTO of SpringSource) about runtime config change in Spring, but does Spring support this today?

In my opinion, if you're starting from a typically architected Spring application, I don't see you having an easy job retrofitting dynamic config changes on top of it.

Conte

Spring allows JMX config changes, but the bit I'm struggling with is that JMX seems to be point-to-point, and that I'd have to know how many servers were running, and the connect to each server in turn an apply a change. Any ideas if there's a way around this? – Swinney

Actually a Zookeeper->JMX integration would be pretty interesting. The retrofit using JMX would be relatively easy using @Managed*. – Sidhu

Answer 2

You should consider Spring Cloud Config:

http://projects.spring.io/spring-cloud/

Spring Cloud Config Centralized external configuration management backed by a git repository. The configuration resources map directly to Spring Environment but could be used by non-Spring applications if desired.

Source code available here:

https://github.com/spring-cloud/spring-cloud-config

Sample application here:

https://github.com/spring-cloud/spring-cloud-config/blob/master/spring-cloud-config-sample/src/main/java/sample/Application.java

Armor

does Spring Cloud Config cause an application to reread the configuration when it is changed? – Rustler

@Rustler yes you can configure this (using the spring cloud bus and @RefreshScope) – Toddtoddie

after the changes - you should do POST to your rest services using localhost:xxxx/actuator/refresh and this would reflect the changes made in config file – Brit

Answer 3

I created a set of spring beans integration zookeeper and springframework as propertyplaceholderconfigurer, in github: https://github.com/james-wu-shanghai/spring-zookeeper.git you can take a look.

Entrain

Answer 4

Not spring in particular but for java generally, there is a CXF implementation of the distributed OSGI standard that uses ZooKeeper as the discovery server to push updated bundles down to the container : http://cxf.apache.org/dosgi-discovery.html.

Unequivocal

Answer 5

Zookeeper can be very nicely leveraged with higher abstraction using Curator APIs for configuration management in distributed applications. To get started just follow these two steps.

STEP 1 : Start zookeper server and then start zookeeper cli and create some znodes. Znodes are nothing but UNIX like files which contain values, and name of files depict property name.
To create/fetch/update properties use these commands on zookeeper cli.

create /system/dev/example/port 9091
get /system/dev/example/port
set /system/dev/example/port 9092

To fetch these properties in java program refer this code snippet.

import java.util.HashMap;
import java.util.Map;

import org.apache.curator.framework.CuratorFramework; 
import org.apache.curator.framework.CuratorFrameworkFactory; 
import org.apache.curator.retry.ExponentialBackoffRetry;
public class App 
{
    public static void main( String[] args ) throws Exception
    {
         final String ZK = "localhost:2181"; 

         final Map<String, String> data = new HashMap<String, String>();         
         CuratorFramework client = CuratorFrameworkFactory.newClient(ZK, new ExponentialBackoffRetry(100, 3)); 
         client.start();
         System.out.println(new String(client.getData().forPath("/system/dev/example/port")));
       }  
}

Cattan

Answer 6

After finding a suggestion to use a FactoryBean to populate a regular PropertyPlaceholderConfigurer I've built this:

package fms;

import org.apache.zookeeper.KeeperException;
import org.apache.zookeeper.WatchedEvent;
import org.apache.zookeeper.Watcher;
import org.apache.zookeeper.ZooKeeper;
import org.apache.zookeeper.data.Stat;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.config.AbstractFactoryBean;

import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.util.Properties;

public class ZkPropertiesFactoryBean extends AbstractFactoryBean<Properties> implements Watcher {
    private Logger LOGGER = LoggerFactory.getLogger(ZkPropertiesFactoryBean.class);
    private String zkConnect;
    private String path;
    private int timeout = 1000;

    @Override protected Properties createInstance() throws Exception {
        long start = System.currentTimeMillis();
        Properties p = new Properties();
        p.load(new ByteArrayInputStream(loadFromZk()));
        double duration = (System.currentTimeMillis() - start)/1000d;
        LOGGER.info(String.format("Loaded %d properties from %s:%s in %2.3f sec", p.size(), zkConnect, path, duration));
        return p;
    }

    @Override public Class<Properties> getObjectType() {
        return Properties.class;
    }

    private byte[] loadFromZk() throws IOException, KeeperException, InterruptedException {Stat stat = new Stat();
        ZooKeeper zk = new ZooKeeper(zkConnect, timeout, this);
        return zk.getData(path, false, stat);
    }

    @Override public void process(WatchedEvent event) {}

    public void setPath(String path) {this.path = path;}

    public void setZkConnect(String zkConnect) {this.zkConnect = zkConnect;}
}

In the spring-config.xml you create the beans as follows:

<bean id="zkProperties" class="fms.ZkPropertiesFactoryBean" p:zkConnect="localhost:2181" p:path="/app/zk-properties"/>
<context:property-placeholder properties-ref="zkProperties"/>

Neckerchief

How to configure properties in zookeeper...from where this path property being loaded [p:path="/app/zk-properties"/] ... cloud you please elaborate – Sessler