[SOLVED]: Execute Commands on Kubernetes Pods with root access

In cloud-native world, we follow the principal of least privileges for microservices in production environment, which in layman term means, root owned operations are not allowed. But in development environment, it becomes inevitable to execute few root owned commands to verify and making some quick changes, one of such use case is to edit some conf files under /etc directory

In this article, we will see how to become root in a container on a pod without redeploying the pod. We will be considering, docker and containerd as container runtime

Existing command execution behavior

When we try to execute some root executable command, we will be getting permission denied, as the container in a pod is running as non root user

]# kubectl exec -it -n ckey-second ckey2-ckey-0 bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
Defaulting container name to ckey2-ckey.
Use 'kubectl describe pod/ckey2-ckey-0 -n ckey-second' to see all of the containers in this pod.
bash-4.2$ hostname
ckey2-ckey-0
bash-4.2$ id 
uid=1000(keycloak) gid=1000(keycloak) groups=1000(keycloak)
bash-4.2$ ls -l /etc/securetty 
-rw-------. 1 root root 221 Apr 1 2020 /etc/securetty
bash-4.2$ cat /etc/securetty 
cat: /etc/securetty: Permission denied

Execute command as root User with Docker as Container Runtime

Identify worker Node

• Identify the worker node where the pod is running, but using -o wide in the get pods command

[root@ctrl-01 task]# kubectl get pods -n ckey-second -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
ckey2-ckey-0 2/2 Running 0 2d17h 192.106.194.109 worker-15 <none> <none>
ckey2-master-realm-configuration-job-k5xc8 0/1 Completed 0 2d17h 192.106.128.84 worker-17 <none> <none>
cmdb-mariadb-0 2/2 Running 0 2d18h 192.106.119.142 worker-06 <none> <none>

Identify the container ID

• Login to the worker node and identify the container ID for the corresponding pod
• Container ID can be identified by greping the pod name after listing all the available containers
• In my case, since it is a multi-container pod, i can see multiple containers and in the last column of the output you can see the container name in human readable format

[root@worker-15 cloud-user]# docker ps|grep ckey2-ckey-0
9adfd9047177 <trimmed> "/bin/sh -c 'exec /b…" 2 days ago Up 2 days k8s_cbura-sidecar_ckey2-ckey-0_ckey-second_72ccc622-fc96-4682-b347-bc9d4fc8f97a_0
b2194fdc637e <trimmed> "/opt/keycloak/csf-d…" 2 days ago Up 2 days k8s_ckey2-ckey_ckey2-ckey-0_ckey-second_72ccc622-fc96-4682-b347-bc9d4fc8f97a_0
edeab232b339 <trimmed> "/pause"               2 days ago Up 2 days k8s_POD_ckey2-ckey-0_ckey-second_72ccc622-fc96-4682-b347-bc9d4fc8f97a_0

Login to the container as Root

• After identifying the container id of the container corresponding to the pod. Access the container as root user by executing the below  docker command

[root@worker-15 cloud-user]# docker exec -u 0 -it b2194fdc637e bash
bash-4.2# hostname
ckey2-ckey-0
bash-4.2# id
uid=0(root) gid=0(root) groups=0(root),1000(keycloak) 
bash-4.2# cat  /etc/securetty 
<trimmed>

Execute command as root User with Containerd / Podman as Container Runtime

• Identify the worker node where the pod is running

Identify the container

• Login to the worker node and identify the container ID  for the corresponding pod's container using ctr command
• We need to identify the container grepping the image (in my case keyclock) of the container

[root@wrk-04 cloud-user]# /usr/bin/ctr -n k8s.io containers ls|grep keycloak
c6ab6dccb1a90484d5e1a5b6bfb80be899ae4380069ab250a04c04e5aee50559 <trimmed>keycloak/keycloak-ha:17.0.1-1-6525 io.containerd.runc.v2

Identify the PID for the Container

• Identify the PID of the container by mentioning the container id

[wrk-04]# /usr/bin/ctr -n k8s.io task list | grep c6ab6dccb1a90484d5e1a5b6bfb80be899ae4380069ab250a04c04e5aee50559
c6ab6dccb1a90484d5e1a5b6bfb80be899ae4380069ab250a04c04e5aee50559 16268 RUNNING

Login to the pod using the PID

Approach 1 using nsenter

• Login to the pod using nsenter as root using below command

[root@wrk-04 cloud-user]# /usr/bin/nsenter -t 16268 -n -m -u -p /bin/sh
sh-4.2# id
uid=0(root) gid=0(root) groups=0(root)
sh-4.2# hostname
ckey2-ckey-0
sh-4.2# cat /etc/securetty 
<trimmed>

Approach 2 using ctr

• Below ctr command can be used access the container as root

[root@wrk-04 cloud-user]# ctr -n k8s.io task exec -t -exec-id 0 --user 0 c6ab6dccb1a90484d5e1a5b6bfb80be899ae4380069ab250a04c04e5aee50559 sh
sh-4.2# hostname
sdl-secum-ckey-0
sh-4.2# id
uid=0(root) gid=0(root) groups=0(root),1(bin),1000(keycloak)
sh-4.2# cat /etc/securetty
<trimmed>