Decouple programs using queues

Question

In his talk at the 54:53 minute mark, Rich Hickey is talking about the usage of queues as a mean to decouple dependent program parts. Can you give me an example on how to deouple the following piece of Java-pseudo-code in order to improve it's design and/or flexibility:

// Warning: Java-pseudo-code ahead
class Job {
    public void doRun(A a) {
        saveObjectToDatabase(a);

        B b = computeB(a);
        saveObjectToDatabase(b);

        C c = computeC(b);
        logToFile(c);
    }
}

saveObjectToDatabase and saveObjectToDatabase can be seen as a method with side-effects, whereas computeB's and computeC's output only depend on a.

I know this question is rather vague/broad. I would like to get a feeling on how to leverage queueing mechanisms without massively complicating my program and still making sure it does the right thing in the right order. Any pointers into the right direction are appreciated.

Gula

It's not very clear. The example you're showing has a self-contained method, which doesn't use any external dependency. There is nothing to decouple as the Job isn't coupled to anything. – Garrotte

My guess is that @Gula would like to get different threads doing the computeB() and computeC() methods, using queues to move units of work between the different threads. – Handicraftsman

Check out Enterprise Integration Patterns. – Microgamete

Thanks for the book link, will take a look at it. – Gula

Answer 1

Well, it's not a very good example, but (in the most straight-forward design) you'd basically have two queues, and (depending on the amount of data involved) you might omit the database.

A first process would receive your a objects from the "outside world" and enqueue them in queue 1. A second process would dequeue objects from queue 1, perform computeB, and enqueue the results onto queue 2. A third process would dequeue objects from queue 2, perform computeC, and log the result or whatever.

Depending, as I said, on the amount of data involved (and maybe a few other factors) the "objects" passed in the queues could either be your actual a and b objects or else just tokens/keys to find the data in the database.

The queues themselves could be implemented several ways. It's possible to implement a queue with a database, eg, though the details get kind of messy. The "processes" could be Java tasks within a single Java process or could be separate OS processes, possibly even on separate machines.

When you use "pipes" on Unix you are effectively using queues in this fashion.

Sewage

Answer 2

This is exactly the principle used by a java library I am using. The idea is to have components assigned to individual tasks in the programs (the logger is a perfect example). Now each component need to run independently from the others, either as a thread either as an event handler.

In the case of event-driven, each component notify which types of events\messages he want to listen to. You have a dispatcher which gather incoming messages and insert them in the receivers' queue. The receiver process, and eventually generate new messages. And Etc...

In your case, something like this:

class SaveObjectHandler{
//
void handle(Event e, Object o){
  if(e instanceof SaveEvent)
      saveObjectToDatabase(o);
}

};

class TransformObject{
//
 void handle(Event e,Object o){
   if(e instanceof TransformEvent){
      B result = compute(o);
      send(new SaveEvent(),result)
   }

 }

};

class Logger{

   void handle(Event e, Object o){
      if(o instanceof B)
        //perform computeC
        logEvent((B)o);
   }

};

};

The library in question is SEDA.

Joplin

The link to the library is down. – Dinar

Answer 3

For the sake of completeness, I'd like to add some more information to Hot Licks' answer:

I've been doing more research on this topic and finally came to the conclusion, that untangling the method is the way to go. I'll be using the kafka terminology of producers/consumers/topics. For more information see The Log: What every software engineer should know about real-time data's unifying abstraction and in particular this graphic:

Regarding my specific question of the example posted, there are two ways to solve it:

Solution 1

• Consumer 1:
  • consume from topic a
  • save to database.
• Consumer 2:
  • consume from topic a
  • compute b
  • save to database.
• Consumer 3: consume from topic a
  • compute b
  • compute c
  • save to database

This has the disadvantage of computing b twice. In pseudo-code:

class ConsumerA  {
    public void consume(A a) {
        saveObjectToDatabase(a);
    }
}

class ConsumerB  {
    public void consume(A a) {
        B b = computeB(a);
        saveObjectToDatabase(b);
    }
}

class ConsumerLog  {
    public void consume(A a) {
        B b = computeB(a);
        C c = computeC(b);
        logToFile(c);
    }
}

Solution 2

• Consumer 1:
  • consume from topic a
  • save to database.
• Consumer 2:
  • consume from topic a
  • compute b, save to database
  • publish b to a separate topic b.
• Consumer 3:
  • consume from topic b
  • compute c
  • logToFile c

In pseudo-code:

class ConsumerA  {
    public void consume(A a) {
        saveObjectToDatabase(a);
    }
}

class ConsumerB  {
    public void consume(A a) {
        B b = computeB(a);
        saveObjectToDatabase(b);
        publish(b); // republish computed information to another topic b
    }
}

class ConsumerLog  {
    public void consume(B b) {
        C c = computeC(b);
        logToFile(c);
    }
}

Gula

Answer 4

I'm afraid with saveObject methods having side-effect you can't decouple it well or at least not easily.

But lets say you need write to database some objects fast. My opinion is that the fastest way with Relational DB should be to save the objects to a queue by multiple clients and than pick them up by one or two pretty fast writers pushing the data to the database as fast as they can.

Resendez