Enable/Disable Hardware Lock Elision

Question

I am using glibc 2.24 version. It has lock elision path included for pthread_mutex_lock implementation with Transactional Synchronization Extensions such as _xbegin() and _xend(). The hardware is supposed to support lock elision as hle CPU flag is for Hardware Lock Elision I think. The processor I am using is Intel(R) Xeon(R) Gold 6130 with Skylake architecture.

First I wanted to disable Lock elision but when I run the program that uses pthread_mutex_lock, with perf stat -T to monitor transactional cycles, I got 0. I assume this means pthread_mutex_lock does not use transactional path at all. After googling I found that it might be necessary to use export GLIBC_TUNABLES=glibc.elision.enable=1 first, to enable lock elision but after this step I still don't see any transactions with perf.

On the other hand when I include _xbegin(); and _xend(); directly in the process, I get some number of transactional cycles with perf stat -T, which should mean that I am looking for the right counters with perf, hopefully.

So any advice to how I can enable lock elision would be helpful. Or am I checking it incorrectly?

Update for TSX I'm using this two instructions in the main function, just like this:

_xbegin();
_xend();

I'm not sure which library it needs, I already have tens of them included. For compilation I'm using the following flags: -O3 -march=native -lpthread that are relevant for this example.

For locks I have mutex:

pthread_mutex_t * mutex;
mutex = (pthread_mutex_t *) malloc(10 * sizeof(pthread_mutex_t));
for(int k=0; k<10; k++){
    pthread_mutex_init(&mutex[k], NULL);
  }

Maybe for elision I should initialize it differently?

Foliage

I don't have a GLIBC recent enough to test, but this link contains the patch to the manual. It contains the attribute bit for the kind of the mutex to make it always (or never) use elision. It also says that mutex without a kind set uses the PTHREAD_MUTEX envar. It is old though. – Changchangaris

Would you please post a minimal example and show us how you compile and link it? – Repercussion

@FlorianWeimer Hi, I updated the post, is this what you meant? – Foliage

@MargaretBloom hi, I went through the link. I can't compile program if I use PTHREAD_MUTEX_INIT_NP or PTHREAD_MUTEX_ELISION_NP type. maybe it needs another gcc flag? I also tried running "PTHREAD_MUTEX=elision" directly from shell, does it need anything else? because this still does not seem to work. – Foliage

@AnaKhorguani I had the same problem compiling, I checked my GLIBC version and it wasn't recent enough. Unfortunately, I cannot update it right now. – Changchangaris

@MargaretBloom Well thank you anyway. Just a weird thing is that this patch is around 6 years ago, and I tried with glibc 2.27 version now which is from last year, and still PTHREAD_MUTEX_ELISION_NP does not compile. gcc version is 7.4 – Foliage

According to Intel, your processor should support TSX-NI, but check for both hle and rtm in /proc/cpuinfo and make sure it's not been disabled using a microcode update (because it's buggy according to erratum SKX88). Also, glibc.elision.enable is supported starting with glibc 2.27. You can check your glibc version using ldd --version. Then we can see what to do. – Inflammable

@HadiBrais Both hle and rtm flags are presented. How can I check if it was disabled with microcode update? In Bugs it shows these: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds. Current version of libc is 2.24, I will try to see if I can update it to 2.27 – Foliage

If both hle and rtm flags are supported, then TSX-NI should work on your processor. I think the problem is with your glibc version. glibc.elision.enable has no effect on versions older than 2.27. If I remember corrcetly, older versions of glibc have to be compiled with --enable-lock-elision=yes for lock elision to work. – Inflammable

Just to clarify, glibc before 2.27 have to be compiled with --enable-lock-elision=yes, not your code. In contrast, starting with glibc 2.27, your code has to be run with export GLIBC_TUNABLES=glibc.elision.enable=1. – Inflammable

Ok I see. So either I should recompile existing glibc version with --enable-lock-elision=yes, or I should upgrade it to 2.27 and then run export GLIBC_TUNABLES=glibc.elision.enable=1 with it, right? – Foliage

@HadiBrais Thanks for the clarifications, I tried with glibc 2.27 version and it seems that export GLIBC_TUNABLES=glibc.elision.enable=1 works with that. Even thought I get 0 cycles for elision, I get up to 1% transactional cycles but all of them are aborted. This should be a "problem" of the program. – Foliage

Nice. I cannot comment on the "0 cycles for elision" because I don't what the code exactly is. Also it would be nice to know what events you're using. Note that there are performance events that you can use to determine why a transaction got aborted. – Inflammable

yes, what I'm doing is a very general observation on the program. In the first place, I just wanted to determine if the program was benefiting from this new lock elision feature. I'm using perf stat -T command which is sufficient so far, but I agree, I found several native counters mentioned for closer monitoring. – Foliage

Your edit (that the env var works with glibc 2.27) looks like an answer. You should probably post it as such instead of just an edit to the question. – Macready

@PeterCordes Ok, I did so. I decided to update the post since it was HadiBrais suggestion that I checked and saw that it worked. But I agree that adding it as an answer is clearer. – Foliage

Answer 1

In earlier versions of glibc, before 2.27, support for TSX could only be controlled using a compile-time flag called enable-lock-elision. I don't know which versions have enable-lock-elision enabled or disabled, but that's how it used to work¹. So if you wanted to enable/dsiable TSX, you'll have to compile glibc yourself and use that flag accordingly. Starting with glibc 2.27, that compile-time option was removed and replaced by a run-time option called glibc.elision.enable. That is, glibc is always compiled with support for TSX, but TSX will only be used if the environment variable glibc.elision.enable is set to 1 before running the application (e.g., by executing export GLIBC_TUNABLES=glibc.elision.enable=1).

Before 2.27, glibc.elision.enable doesn't exist and so it has no effect. Whether TSX is used depends on the compile-time flag enable-lock-elision. You're using 2.24. So the easiest solution would be to move to 2.27 or a more recent version.

Note that all current Intel processors that support TSX appear to have the same bug that "using Intel TSX Instructions May Lead to Unpredictable System Behavior" according to the respective specification updates. For some processors, Intel has released microcode updates to actually disable TSX. However, the implementation on your processor continues to be enabled.

---

Footnotes:

(1) According to this bug report, lock elision has been disabled in glibc starting with 2.23. I think this is done by disabling enable-lock-elision when building glibc, but I've not verified this by looking at the code. But this is consistent with your observation that it is disabled in 2.24.

Foliage

If this answers your question, you can/should accept the answer by clicking on the check mark below the voting buttons. – Inflammable

Ok, but it was in fact your answer :) – Foliage

That's OK, you can go ahead and accept the answer. I've added more information if you don't mind, just to make the answer more complete. – Inflammable

@HadiBrais yes sure, this is a way more comprehensive answer. Thank you very much. – Foliage

"this bug report," please mention that that wasn't considered a bug in glibc – Proselytize

Answer 2

To add to the answer from Ana, I think the glibc may have custom variables on the OS intended for. At least on RHEL, "RHEL_GLIBC_TUNABLES="glibc.elision.enable=1" was already introduced in glibc 2.17. It is possible that OP's distro also has a similar environmental variable for enabling HLE at runtime.

%define glibcsrcdir glibc-2.17-c758a686
%define glibcversion 2.17
%define glibcrelease 317%{?dist}
...
# If the architecture has elision support in glibc then it should be listed
# here to enable elision for default pthread mutexes and rwlocks. The elision
# is not enabled automatically and each process has to opt-in to elision via
# the environment variable RHEL_GLIBC_TUNABLES by setting it to enabled e.g.
# RHEL_GLIBC_TUNABLES="glibc.elision.enable=1".
%define elisionarches x86_64 %{power64}
...
%ifarch %{elisionarches}
    --enable-lock-elision=yes \
%endif
...

According to glibc.spec of glibc 2.23.1 for Fedora 24, elisions has been disabled at compile-time since glibc 2.20 except on s390/s390x systems. This is reaffirmed with the glibc.spec of glibc 2.24. for Fedora 25.

# Enable lock elision support for these architectures
#
# At the moment lock elision is disabled on x86_64 until there's a CPU that
# would actually benefit from enabling it.  Intel released a microcode update
# to disable HLE and RTM at boot and the Fedora kernel now applies it early
# enough that keeping lock elision enabled should be harmless, but we have
# disabled it anyway as a conservative measure.
%define lock_elision_arches s390 s390x
...
%ifarch %{lock_elision_arches}
        --enable-lock-elision \
%endif
...
# Disable rwlock elision if --enable-lock-elision is not used.
Patch0052: glibc-disable-rwlock-elision.patch
...
* Tue Sep 30 2014 Siddhesh Poyarekar <[email protected]> - 2.20.90-6
- Sync with upstream master.
- Disable more Intel TSX usage in rwlocks (#1146967).
- Enable lock elision again on s390 and s390x.
- Enable Systemtap SDT probes for all architectures (#985109).

while it's not conclusive to say that the OP's Linux distro runs on the same glibc 2.24 as Fedora 25 does, it is likely that the generic glibc 2.24 has elision disabled as per Patch0052. That's why OP couldn't see performance gain with "export GLIBC_TUNABLES=glibc.elision.enable=1"

source:
http://rpmfind.net/linux/RPM/centos/7.9.2009/x86_64/Packages/glibc-2.17-317.el7.x86_64.html
http://rpm.pbone.net/info_idpl_48715292_distro_fedora_24_com_glibc-2.23.1-12.fc24.x86_64.rpm.html
http://rpm.pbone.net/resultsb_dist_103_size_26149846_name_glibc-2.24-10.fc25.src.rpm.html

update:
SLES12 has its own glibc versions. SP2 comes with glibc 2.22, which is different from the glibc 2.22 from Fedora 25.

SLES12 SP2	Fedora 25
\	Patch0052: glibc-disable-rwlock-elision.patch (since 2.20.90)
elision=--enable-lock-elision if [ "$dirname" = "noelision" ]; then elision=--disable-lock-elision fi	\

source:
https://www.suse.com/support/kb/doc/?id=000019071 http://rpm.pbone.net/info_idpl_55157770_distro_opensuse_com_glibc-2.24-3.29.x86_64.rpm.html

Decomposed

Also note that all x86 CPUs with updated microcode have HLE disabled. Only RTM is still available on any CPUs, if the OS allows. Has Hardware Lock Elision gone forever due to Spectre Mitigation?/ – Macready