Segmentation Fault when Using SentenceTransformer Inside Docker Container

Question

Edit: After test on different machines it is Apple M1 M2 specific bug.

I am trying to run a Flask application inside a Docker container on Apple Silicon M2 (could be an issue), where I use the SentenceTransformer model to encode sentences. However, when I call the encode method on the model, the application crashes with a segmentation fault.

Here's the relevant code:

from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-MiniLM-L6-v2')

#Our sentences we like to encode
sentences = ['This framework generates embeddings for each input sentence',
    'Sentences are passed as a list of string.', 
    'The quick brown fox jumps over the lazy dog.']

#Sentences are encoded by calling model.encode()
sentence_embeddings = model.encode(sentences)

Source: https://www.sbert.net/docs/quickstart.html

after using import faulthandler and faulthandler.enable() The error traceback is:

Fatal Python error: Segmentation fault

Thread 0x0000ffff640ff1a0 (most recent call first):
  File "/usr/local/lib/python3.11/threading.py", line 331 in wait
  File "/usr/local/lib/python3.11/threading.py", line 629 in wait
  File "/usr/local/lib/python3.11/site-packages/tqdm/_monitor.py", line 60 in run
  File "/usr/local/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
  File "/usr/local/lib/python3.11/threading.py", line 1002 in _bootstrap

Current thread 0x0000ffffa6814020 (most recent call first):
  File "/usr/local/lib/python3.11/site-packages/transformers/activations.py", line 78 in forward
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527 in _call_impl
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518 in _wrapped_call_impl
  File "/usr/local/lib/python3.11/site-packages/transformers/models/bert/modeling_bert.py", line 452 in forward
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527 in _call_impl
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518 in _wrapped_call_impl
  File "/usr/local/lib/python3.11/site-packages/transformers/models/bert/modeling_bert.py", line 551 in feed_forward_chunk
  File "/usr/local/lib/python3.11/site-packages/transformers/pytorch_utils.py", line 240 in apply_chunking_to_forward
  File "/usr/local/lib/python3.11/site-packages/transformers/models/bert/modeling_bert.py", line 539 in forward
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527 in _call_impl
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518 in _wrapped_call_impl
  File "/usr/local/lib/python3.11/site-packages/transformers/models/bert/modeling_bert.py", line 612 in forward
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527 in _call_impl
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518 in _wrapped_call_impl
  File "/usr/local/lib/python3.11/site-packages/transformers/models/bert/modeling_bert.py", line 1022 in forward
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527 in _call_impl
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518 in _wrapped_call_impl
  File "/usr/local/lib/python3.11/site-packages/sentence_transformers/models/Transformer.py", line 66 in forward
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527 in _call_impl
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518 in _wrapped_call_impl
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/container.py", line 215 in forward
  File "/usr/local/lib/python3.11/site-packages/sentence_transformers/SentenceTransformer.py", line 165 in encode
  File "<stdin>", line 1 in <module>

Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, scipy._lib._ccallback_c, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._isolve._iterative, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.linalg._flinalg, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.spatial._ckdtree, scipy._lib.messagestream, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.spatial.transform._rotation, scipy.ndimage._nd_image, _ni_label, scipy.ndimage._ni_label, scipy.optimize._minpack2, scipy.optimize._group_columns, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize.__nnls, scipy.optimize._highs.cython.src._highs_wrapper, scipy.optimize._highs._highs_wrapper, scipy.optimize._highs.cython.src._highs_constants, scipy.optimize._highs._highs_constants, scipy.linalg._interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.optimize._direct, scipy.integrate._odepack, scipy.integrate._quadpack, scipy.integrate._vode, scipy.integrate._dop, scipy.integrate._lsoda, scipy.special.cython_special, scipy.stats._stats, scipy.stats.beta_ufunc, scipy.stats._boost.beta_ufunc, scipy.stats.binom_ufunc, scipy.stats._boost.binom_ufunc, scipy.stats.nbinom_ufunc, scipy.stats._boost.nbinom_ufunc, scipy.stats.hypergeom_ufunc, scipy.stats._boost.hypergeom_ufunc, scipy.stats.ncf_ufunc, scipy.stats._boost.ncf_ufunc, scipy.stats.ncx2_ufunc, scipy.stats._boost.ncx2_ufunc, scipy.stats.nct_ufunc, scipy.stats._boost.nct_ufunc, scipy.stats.skewnorm_ufunc, scipy.stats._boost.skewnorm_ufunc, scipy.stats.invgauss_ufunc, scipy.stats._boost.invgauss_ufunc, scipy.interpolate._fitpack, scipy.interpolate.dfitpack, scipy.interpolate._bspl, scipy.interpolate._ppoly, scipy.interpolate.interpnd, scipy.interpolate._rbfinterp_pythran, scipy.interpolate._rgi_cython, scipy.stats._biasedurn, scipy.stats._levy_stable.levyst, scipy.stats._stats_pythran, scipy._lib._uarray._uarray, scipy.stats._statlib, scipy.stats._sobol, scipy.stats._qmc_cy, scipy.stats._mvn, scipy.stats._rcont.rcont, regex._regex, sklearn.__check_build._check_build, sklearn.utils._isfinite, sklearn.utils.murmurhash, sklearn.utils._openmp_helpers, sklearn.utils._logistic_sigmoid, sklearn.utils.sparsefuncs_fast, sklearn.preprocessing._csr_polynomial_expansion, sklearn.preprocessing._target_encoder_fast, sklearn.utils._vector_sentinel, sklearn.feature_extraction._hashing_fast, sklearn.utils._random, sklearn.utils._seq_dataset, sklearn.metrics.cluster._expected_mutual_info_fast, sklearn.metrics._dist_metrics, sklearn.metrics._pairwise_distances_reduction._datasets_pair, sklearn.utils._cython_blas, sklearn.metrics._pairwise_distances_reduction._base, sklearn.metrics._pairwise_distances_reduction._middle_term_computer, sklearn.utils._heap, sklearn.utils._sorting, sklearn.metrics._pairwise_distances_reduction._argkmin, sklearn.metrics._pairwise_distances_reduction._argkmin_classmode, sklearn.metrics._pairwise_distances_reduction._radius_neighbors, sklearn.metrics._pairwise_fast, sklearn.linear_model._cd_fast, sklearn._loss._loss, sklearn.utils.arrayfuncs, sklearn.svm._liblinear, sklearn.svm._libsvm, sklearn.svm._libsvm_sparse, sklearn.utils._weight_vector, sklearn.linear_model._sgd_fast, sklearn.linear_model._sag_fast, scipy.io.matlab._mio_utils, scipy.io.matlab._streams, scipy.io.matlab._mio5_utils, sklearn.datasets._svmlight_format_fast, charset_normalizer.md, yaml._yaml, sentencepiece._sentencepiece, PIL._imaging (total: 163)
Segmentation fault

Some points:

The Docker container has ample memory allocated. I've tried updating the libraries (torch, transformers, and sentence-transformers). The same code works perfectly outside the Docker environment. How can I resolve this segmentation fault when running the code inside Docker?

Here is a pip list of actual versions.

Package               Version
--------------------- ---------
blinker               1.6.3
certifi               2023.7.22
charset-normalizer    3.3.0
click                 8.1.7
filelock              3.12.4
Flask                 3.0.0
fsspec                2023.9.2
huggingface-hub       0.17.3
idna                  3.4
itsdangerous          2.1.2
Jinja2                3.1.2
joblib                1.3.2
MarkupSafe            2.1.3
mpmath                1.3.0
networkx              3.1
nltk                  3.8.1
numpy                 1.26.0
packaging             23.2
Pillow                10.0.1
pip                   23.2.1
PyYAML                6.0.1
regex                 2023.10.3
requests              2.31.0
safetensors           0.4.0
scikit-learn          1.3.1
scipy                 1.11.3
sentence-transformers 2.2.2
sentencepiece         0.1.99
setuptools            65.5.1
sympy                 1.12
threadpoolctl         3.2.0
tokenizers            0.14.1
torch                 2.1.0
torchvision           0.16.0
tqdm                  4.66.1
transformers          4.34.0
typing_extensions     4.8.0
urllib3               2.0.6
Werkzeug              3.0.0
wheel                 0.41.2

For more clarification, here is the most important part of my Dockerfile.

FROM python:3.11

RUN pip install --upgrade pip
RUN pip install Flask==3.0.0 sentence-transformers==2.2.2

Aggregate

Answer 1

Couldn't find out why newer python versions are failing, but python:3.6.15 tag works just fine on mac arm64 apple silicon M1.

Edit (better alternative based in @jackson-bierfeldt comment): Downgrading PyTorch to 2.0.1 solved it. So, put something like torch==2.0.* to requirements.txt, no need to downgrade Python itself.

Aggregate

I've just run into the same issue. Downgrading PyTorch to 2.0.1 solved it. So, put something like torch==2.0.* to requirements.txt, no need to downgrade Python itself. (I'm running with FROM python:3.11-slim.) – Epigraph

@Epigraph You just saved my entire day. Adding torch==2.0.* to my requirements file worked for my project, also running on python:3.11-slim – Mf

I had the same issue on a M2 Mac, if I use conda works fine, but inside docker I get the segmentation fault. If I downgrade torch to 2.0.*, it works on local docker and in conda, but then is broken in the target aws ec2 instance where this application is deployed, any thoughts? – Internalcombustion

Answer 2

Encountered the same issue on an M2 Pro 13.4.1 developing inside the official Python 3.11.4 Docker image.

I was encountering the issue on both PyTorch versions 2.1.0 and 2.0.1+cpu, so the other fixes stated did not apply in my situation.

I was able to resolve the segmentation faults by switching to the official PyTorch Docker image. I was able to use latest, which at the time of this post is the following image. While not super ideal, it did the trick in my case

Appel

Downside here is that the official PyTorch installation I linked seems to be running on Python 3.10.13 – Appel

Answer 3

Encountered this issue on an MB Pro 2021 M1 14.0 and python 3.9 with pytorch 2.1.0. In the end I was able to fix it by updating the dependencies: Upgrade PyTorch from 2.1.0 to 2.1.1.

Fordham