Communication between 2 nodes in a cluster

Question

I am trying to figure out how this will work out:

1. client-server communication via NIO/BIO
2. Server-server communication (replication, membership etc) via JGroups (replication of data can be a pain?)

What are your suggestion, pros/cons of using such an architecture, why go for jgroups, any alternatives?

Should I go with native NIO or use some libraries like Mina or Netty?

Update

1. No REST, it's not a web server. I am building a dynamo like distributed database.
2. activeMQ is like JMS I think, where I will have single point of failure. JGroups solves that issue, right?

Ornamented

if you are not going to implement NIO on your own, Mina is ok. JGroups is good as well. The design looks good to me. – Kalie

any tradeoff b/w native NIO and mina? – Ornamented

It depends on your server(s) the amount of your permanent connections you need to achieve and so on. Good custom NIO framework could be 3k lines or so, need some protocol over that as well. If you are new to NIO, while not hard, you'd be better off w/ Mina. We have custom built NIO impl. that works alongside tomcat (and hijacks channels and doesn't ever returns them back). But we also use NIO for simple things like APN. BIO is an option but you need 2 threads per connection and I'd pick Mina over any day. – Kalie

okay, so MINA it is, also, is there a configuration in MINA where I can switch to BIO if I want to. Some of my use cases need BIO. – Ornamented

@zengr, as of now I think BIO is unsupported. Yet you get Future object you can await on. – Kalie

Have you considered Terracotta or Coherence? – Deprived

no, I think they are complete solutions, I am trying to implement one by my self. – Ornamented

Answer 1

I wonder why you you want to implement binary protocol for client-server communication? My advice is to go with REST or WebServices. It has huge advantages compared to proprietary binary protocol.

About server-server communication... There are huge number of choices and I wonder why you have chosen JGroups. Can you elaborate on this? I will advice you to go with some messaging solution like JMS. There are a lot of available implementation like ActiveMQ. It will nicely decouple servers from each other.

If you need to maintain shared configuration information, naming, have distributed synchronization between servers you can look at Apache ZooKeeper project.

If your servers are deployed at Amazon EC2 instances, than I think the best solution is to use SQS and/or SNS services for server-server communication.

You can use Terracotta for replication of your server's state (still I will recommend you to make your servers as stateless as possible).

But it's really hard to advice anything more concrete because you have not described neither your system nor requirements to it. I think it would be very helpful!

Zaffer

so you offer long time polls to implement push? I have no issues w/ JGroups on my own. – Kalie

@bestsss: the original question tells nothing about requirements, so I just listed my preferences. You can use pull or server-push for client-server communication. But if binary protocol is also OK, than JMS is also good option for client-server communication (if @Ornamented really needs server push). Choice between JGroups and JMS is highly depends on system requirement. For most system I saw - JMS is the way to go, because it provides reliable and persistent messaging. – Zaffer

1. No REST, it's not a web server. I am building a dynamo like distributed database. 2. activeMQ is like JMS I think, where I will have single point of failure. JGroups solves that issue? – Ornamented

@Easy Angel, client-server, imo, means push from the server (otherwise I'd see no reason for a custom client). I have no positive experience w/ JMS (I don't disregard it, though), and if a server fails process/respond JGroups msg I consider it dead as dodo and not part of the cluster. For anything I might need a true persistence/durable state, there is a transactional database. – Kalie

@zengr, JGroups will solve that ("single point of failure") issue but not alone. JGroups w/o local multicast requires TCP stack and configuration (I also have no xp w/ the latest JGroups versions) – Kalie

@zengr: Every single standalone server will be point of failure unless you will cluster it. I'm sure that JGroups is not an exception. In case of ActiveMQ you can setup MasterSlave replication in order to reach high availability and fault tolerance: activemq.apache.org/masterslave.html – Zaffer

@zengr: The fact that it's not web-server does not mean, that you can't expose REST API. As an example you can look at CouchDB: couchdb.apache.org . In java there are a lot of ways to expose REST API within normal Java SE application (you can embed jetty; look at Restlet as an example: wiki.restlet.org/docs_2.0/13-restlet/275-restlet/…) – Zaffer