Can one monitor STM's contention level?

Question

Is there any way to poll whether Clojure's STM transactions are being retried, and at what rate?

Kktp

See also: #4792697 – Herewith

Answer 1

You can observe the history count of a ref which will indicate that there is contention on it:

user=> (def my-ref (ref 0 :min-history 1))
#'user/my-ref
user=> (ref-history-count my-ref)
0
user=> (dosync (alter my-ref inc))
1
user=> (ref-history-count my-ref)
1

The history count does not directly represent contention. Instead it represents the number of past values that have been maintained in order to service concurrent reads.

The size of the history is limited by min and max values. By default those are 0 and 10, respectively, but you can change them when creating the ref (see above). Since min-history is 0 by default, you won't usually see ref-history-count return non-zero values, unless there is contention on the ref.

See more discussion on history count here: https://groups.google.com/forum/?fromgroups#!topic/clojure/n_MKCoa870o

I don't think there is any way, provided by clojure.core, to observe the rate of STM transactions at the moment. You can of course do something similar to what @Chouser did in his history stress test:

(dosync
    (swap! try-count inc)
    ...)

i.e. increment a counter inside the transaction. The increment will happen every time the transaction is tried. If try-count is larger than 1, the transaction was retried.

Richma

Never thought of using the history count, or atoms within txs. Thanks for the suggestions! – Kktp

Answer 2

By introducing named dosync blocks and commit counts (the times a named dosync has succeeded), one can quite easily keep track of the times threads have retried a given transaction.

(def ^{:doc "ThreadLocal<Map<TxName, Map<CommitNumber, TriesCount>>>"}
  local-tries (let [l (ThreadLocal.)]
                (.set l {})
                l))

(def ^{:doc "Map<TxName, Int>"}
  commit-number (ref {}))

(def history ^{:doc "Map<ThreadId, Map<TxName, Map<CommitNumber, TriesCount>>>"}
  (atom {}))

(defn report [_ thread-id tries]
  (swap! history assoc thread-id tries))

(def reporter (agent nil))

(defmacro dosync [tx-name & body]
  `(clojure.core/dosync
    (let [cno# (@commit-number ~tx-name 0)
          tries# (update-in (.get local-tries) [~tx-name] update-in [cno#] (fnil inc 0))]
      (.set local-tries tries#)
      (send reporter report (.getId (Thread/currentThread)) tries#))
    ~@body
    (alter commit-number update-in [~tx-name] (fnil inc 0))))

Given the following example...

(def foo (ref {}))

(def bar (ref {}))

(defn x []
  (dosync :x ;; `:x`: the tx-name.
          (let [r (rand-int 2)]
            (alter foo assoc r (rand))
            (Thread/sleep (rand-int 400))
            (alter bar assoc (rand-int 2) (@foo r)))))

(dotimes [i 4]
  (future
   (dotimes [i 10]
     (x))))

...@history evaluates to:

;; {thread-id {tx-name {commit-number tries-count}}}
{40 {:x {3 1, 2 4, 1 3, 0 1}}, 39 {:x {2 1, 1 3, 0 1}}, ...}

Kktp

Answer 3

This additional implementation is substantially simpler.

;; {thread-id retries-of-latest-tx}
(def tries (atom {}))

;; The max amount of tries any thread has performed
(def max-tries (atom 0))

(def ninc (fnil inc 0))

(def reporter (agent nil))

(defn report [_ tid]
  (swap! max-tries #(max % (get @tries tid 0)))
  (swap! tries update-in [tid] (constantly 0)))

(defmacro dosync [& body]
  `(clojure.core/dosync
    (swap! tries update-in [(.getId (Thread/currentThread))] ninc)
    (commute commit-id inc)
    (send reporter report (.getId (Thread/currentThread)))
    ~@body))

Kktp