Process Complex Events

Question

I have a stream of discrete events coming into my system and I need to apply rules based on the content of each event. Also, I want to apply complex event processing on these stream events.

Constraints 1. These rules are user supplied and will change dynamically. 2. I do not want to restart my system whenever a rule is applied. 3. HA 4. Only mature open source solutions

Possible ways.. 1. Run Esper CEP inside storm bolt
2. Have drools run inside Storm bolt

• Would this do single event rule processing as well as complex events? Would Rule changes will require my storm to restart ?

• Can I dynamically insert rules into storm bolts?

Other Solutions? Spark?

Pietism

Answer 1

Both Drools and Esper can dynamically inject rules and do not require a restart of Storm or the JVM.

Both Esper and Drools Fusion can handle complex events, generally, with different tradeoffs.

In the same distributed stream processing solution domain are Spark streaming, Storm, Yahoo S4 and Apache Samza. Any of these can be used to host Esper/Drools. For Storm I think since it replays tuples you'd want to either de-duplicate before sending to Drools/Esper or design your continuous queries expecting possible duplicate deliveries.

Paleozoic

Could you point me to some resources where Spark Streaming and Esper can be use together? – Derive