How does multi-threading works in Cadence/Temporal workflow?

Question

In Cadence/Temporal workflow programming:

• Native threading library is not allowed. E.g. in Java, threads must be created through Async.procedure or Async.function and in Golang, threads must be created through workflow.Go. So why?
• Is there any racing condition like using native threading? E.g. Hashtable or ConcurrentHashMap should be used instead of HashMap for thread-safty?

Bimbo

Answer 1

Summary:

Workflow execution must be deterministic. This is required for history replay to rebuild thread states. In order to be deterministic, Cadence/Temporal controls the thread scheduling in a cooperative manner(instead of preemptive like most OS does):

• Only one workflow thread can be running at any point of time
• Only when the current executing workflow thread blocked on something else, it will yield and let a next workflow thread to run.
• The order of "next workflow thread" is deterministic.

Therefore:

• Native threading library is never allowed in workflow code, as Cadence/Temporal will lose the control for determinism
• Racing condition that we usually run into will never happen because of the cooperative multi-threading. HashMap is safe to use in workflow code.

More Details

Cadence/Temporal SDKs have a DeterministicRunner to manipulate the thread execution. E.g. Java SDK, Golang SDK. This deterministic runner will decide which workflow thread to run in the right order, and one at a time. For each decision task, it will execute in loop until "all threads are blocked" -- RunUntilAllBlocked/ExecuteUntilAllBlocked.

Async.procedure/ Async.function/ workflow.Go will create a new thread and add to the list in the deterministicRunner, so that the executing will be controlled.

Because only one thread can be executed at anytime, most racing condition that we are running into in regular code will not happen.

However, this doesn't mean there is no racing condition at all. In some cases, there will still be conditions to cause some deadlock.

In other words, being cooperative doesn't mean no racing condition or deadlock. It just mean the deadlock situations are much less. And there is no racing condition to cause dirty read like preemptive thread scheduling.

Deadlock example

If threadA grabs lockA and wait for an activity, then it yield to threadB, then threadB grabs lockB and wait for an activity;

After the activity, threadA will try to get lockB before releasing lockA, threadB will try to get lockA before releasing lockA;

Now they will run into deadlock when the activities are completed.

Others

Using iWF will keep you away from these complicated concepts. iWF provides a nice abstraction on top of Cadence/Temporal but keep the same power.

More reference

https://community.temporal.io/t/how-does-workflow-thread-synchronization-work/504

Bimbo