OCI runtime exec failed: exec failed: unable to start container process: open /dev/pts/0: operation not permitted: unknown [closed]

Question

I have read many links similar to my issue, but none of them were helping me to resolve the issue.

Similar Links:

1. Failed to exec into the container due to permission issue after executing 'systemctl daemon-reload'
2. OCI runtime exec failed: exec failed: unable to start container process: open /dev/pts/0: operation not permitted: unknown
3. CI runtime exec failed: exec failed: unable to start container process: open /dev/pts/0: operation not permitted: unknown
4. OCI runtime exec failed: exec failed: unable to start container process: open /dev/pts/0: operation not permitted: unknown
5. Fail to execute docker exec
6. OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "open /proc/self/fd: no such file or directory": unknown

Problem Description:

I have created a new Kubernetes cluster using Kubespray. When I wanted to execute some commands in one of containers I faced to the following error:

Executed Command

kubectl exec -it -n rook-ceph rook-ceph-tools-68d847b88d-7kw2v -- sh

Error:

OCI runtime exec failed: exec failed: unable to start container process: open /dev/pts/1: operation not permitted: unknown command terminated with exit code 126

I have also logged in to the node, which runs the pod, and try executing the container using docker exec command, but the error was not changed.

Workarounds:

• As I have found, the error code (126) implies that the permissions are insufficient, but I haven't faced this kind of error (like executing sh) in Docker or Kubernetes.

• I have also checked whether SELinux is enabled or not (as it has been said in the 3rd link).

  apt install policycoreutils
  sestatus
  # Output
  SELinux status:                 disabled
  

• In the 5th link, it was said to check whether you have updated the kernel, and I didn't upgrade anything on the nodes.

  id; stat /dev/pts/0
  # output
  uid=0(root) gid=0(root) groups=0(root)
    File: /dev/pts/0
    Size: 0               Blocks: 0          IO Block: 1024   character special file
  Device: 18h/24d Inode: 3           Links: 1     Device type: 88,0
  Access: (0600/crw-------)  Uid: (    0/    root)   Gid: (    5/     tty)
  Access: 2022-08-21 12:01:25.409456443 +0000
  Modify: 2022-08-21 12:01:25.409456443 +0000
  Change: 2022-08-21 11:54:47.474457646 +0000
   Birth: -
  

• Also tried /bin/sh instead of sh or /bin/bash, but not worked and the same error occurred.

Can anyone help me to find the root cause of this problem and then solve it?

Shimmer

kubectl run debug --image=alpine -i --tty -- sh can you check if this command work – Developing

@Developing It works, but it doesn't work for executing the existing pods like rook-ceph-tools-68d847b88d-7kw2v, I've mentioned it here. – Shimmer

is there any volume mount? is the image support your platform? k describe node node_name | grep "kubernetes.io/arch" – Developing

docker inspect image my-image | grep Architecture the node and image architecture should match, if not then you need to rebuild image – Developing

docker build --platform linux/x86-64 -t demo-image . or platform linux/amd64 – Developing

Can you edit the question to include a minimal reproducible example; what is the application code required to reproduce the issue? Do you think this is a duplicate of the existing question you linked to? Or do you think this is about the way you installed the cluster (in which case another site like Server Fault may be more appropriate)? – Ot

@Developing Yes! they are the same and the architecture is amd64. – Shimmer

@DavidMaze No it is not depend on the way I have installed the cluster. I was just for describing what has happened to me. No, it's not the duplicate question, since non of them work properly and the problem is still unresolved. – Shimmer

Try kubectl exec -i -n rook-ceph rook-ceph-tools-68d847b88d-7kw2v -- sh without the -t. You might get a running shell without tty. Just test at empty prompt. E.g. date and exit. If it works, you have hit a bug in runc and have to update to v1.1.4 – Lassie

Answer 1

This issue may relate to docker, first drain your node.

kubectl drain <node-name>

Second, SSH to the node and restart docker service.

systemctl restart docker.service

At the end try to execute your command.

Sequent

If you are using Docker Compose, bringing your containers down and back up again also solves the problem. My question is what might be causing Docker to do this? – Kabob

@Kabob I'm not sure this is exactly Docker related, but this solution fix his problem. – Sequent

looks like this was a bug in runc 1.1.3 that has since been fixed. i upgraded containerd on my machine and haven't seen this issue since. github.com/opencontainers/runc/pull/3554 – Kabob

@Kabob Definitely not just docker. I'm getting this with Podman under runc. I'm on Almalinux 8 which has runc 1.1.3. Hopefully runc 1.1.4 shows up soon. I thought this was some bug in Cockpit's Podman plugin, but guess not.... – Czerny