SESSION ID: CSCO-R05

Lateral Movements in Kubernetes

#RSAC

Yossi Weizman
Senior Security Researcher
Microsoft
#RSAC

Disclaimer

Presentations are intended for educational purposes only and do not replace independent professional
judgment. Statements of fact and opinions expressed are those of the presenters individually and,
unless expressly stated to the contrary, are not the opinion or position of RSA Conference™ or any other
co-sponsors. RSA Conference does not endorse or approve, and assumes no responsibility for, the
content, accuracy or completeness of the information presented.
Attendees should note that sessions may be audio- or video-recorded and may be published in various
media, including print, audio and video formats without further notice. The presentation template and
any media capture are subject to copyright protection.
© 2023 RSA Conference LLC or its affiliates. The RSA Conference logo and other trademarks are proprietary. All rights reserved.

2
#RSAC

Agenda

• Kubernetes – Overview & identities

• Inner-cluster lateral movement
• Cluster-to-cloud lateral movement
• Cross-cloud lateral movement
• Mitigations & Detections
• Key takeaways

3
Kubernetes:
Overview & identities

#RSAC
#RSAC

Kubernetes – overview

Containers - A unit of software that packages the code and all its dependencies. The
executable package is called image. At runtime, containers run as isolated software in
the host OS. Containers run by a container engine such as Docker or ContainerD

Kubernetes - container-orchestration system. Basically, manages a cluster of

computers (nodes), each one is running a Container engine. Enables to manage and
scale containerized environments.

5
#RSAC

K8s control plane

API Server scheduler

etcd Controller manager

Node 1 Node 2 Node 3

Container engine Container engine Container engine

Container A Container C Container E

Container B Container D Container F

Kubelet Kubelet Kubelet

#RSAC

K8s control plane

In managed clusters:
Fully managed API Server scheduler
by the cloud provider
etcd Controller manager

Node 1 Node 2 Node 3

Container engine Container engine Container engine

Container A Container C Container E

Container B Container D Container F

Kubelet Kubelet Kubelet

#RSAC

K8s control plane

API Server scheduler

etcd Controller manager

Node 1 Node 2 Node 3

Pod A Pod C Pod E

Pod B Pod D Pod F

#RSAC

Identity types in Kubernetes

Three main areas:

• How users (or applications) from outside the cluster authenticate with the cluster.

• How workloads in the cluster authenticate within the cluster.

• How workloads in the cluster authenticate with resources in the cloud outside the
cluster.

9
#RSAC

Identity types in Kubernetes

Three main areas:

• How users (or applications) from outside the cluster authenticate with the cluster.

• How workloads in the cluster authenticate within the cluster.

• How workloads in the cluster authenticate with resources in the cloud outside the
cluster.

10
#RSAC

Kubernetes AuthN & AuthZ

• Service account = Identity of an application in Kubernetes

• RBAC – Role-based access control
• Roles = set of rules (permissions)
• Role binding = assignment of a role to a user\service account

11
#RSAC

Kubernetes AuthN & AuthZ

• Service account = Identity of an application in Kubernetes

• RBAC – Role-based access control
• Roles = set of rules (permissions)
• Role binding = assignment of a role to a user\service account

12
#RSAC

Kubernetes AuthN & AuthZ

• Service accounts are mounted to pods, allowing them to authenticate with the API
server.

13
#RSAC

K8s control plane

API Server scheduler

etcd Controller manager

Node 1 Node 2 Node 3

Pod A Pod C Pod E

Pod B Pod D Pod F

#RSAC

K8s control plane

API Server scheduler

etcd Controller manager

Node 1 Node 2 Node 3

Pod A Pod C Pod E

Pod B Pod D Pod F

Inner-cluster lateral movement

#RSAC
#RSAC

Inner-cluster lateral movement

Let’s assume a pod is compromised

17
#RSAC

K8s control plane

API Server scheduler

etcd Controller manager

Node 1 Node 2 Node 3

Pod A Pod C Pod E

Pod B Pod D Pod F

#RSAC

K8s control plane

API Server scheduler

etcd Controller manager

Node 1 Node 2 Node 3

Pod A Pod C Pod E

Pod B Pod D Pod F

#RSAC

K8s control plane

API Server scheduler

etcd Controller manager

Node 1 Node 2 Node 3

Pod A Pod C Pod E

Pod B Pod D Pod F

#RSAC

K8s control plane

API Server scheduler

etcd Controller manager

Node 1 Node 2 Node 3

Pod A Pod C Pod E

Pod B Pod D Pod F

#RSAC

K8s control plane

API Server scheduler

etcd Controller manager

Node 1 Node 2 Node 3

Pod A Pod C Pod E

Pod B Pod D Pod F

#RSAC

Inner-cluster lateral movement

How can attackers leverage a compromised pod for cluster takeover?

23
#RSAC

K8s control plane

API Server scheduler

etcd Controller manager

Node 1 Node 2 Node 3

Pod’s A service
account token

Pod A Pod C Pod E

Pod B Pod D Pod F

Kubelet’s
credentials
#RSAC

Inner-cluster lateral movement

How can attackers leverage a compromised pod for cluster takeover?

Good news – it becomes more difficult:

• Read secrets permission isn’t enough
• Newer versions of K8s don’t create long-lived SA tokens as secret objects

• Node takeover ≠ cluster takeover

• Node authorizer + NodeRestriction admission controller limit the permissions of the nodes’ identities

25
#RSAC

Inner-cluster lateral movement

How can attackers leverage a compromised pod for cluster takeover?

Good news – it becomes more difficult:

• Read secrets permission isn’t enough
• Newer versions of K8s don’t create long-lived SA tokens as secret objects

• Node takeover ≠ cluster takeover

• Node authorizer + NodeRestriction admission controller limit the permissions of the nodes’ identities

However, common misconfigurations still allow it.

Let’s see an example

26
#RSAC

Inner-cluster lateral movement

Example: Self-update permissions

• We’ve observed cases in which applications had permissions to update

themselves.

• This allows the applications to change their own configuration. For

example: update their configuration to be privileged, change their
service account, schedule to specific node(s).

• Effectively, this can lead to cluster takeover

27
#RSAC

K8s control plane

API Server scheduler

etcd Controller manager

Change the
application
configuration

Node 1 Node 2 Node 3

Pod A Pod C Pod E

Pod B Pod D Pod F

#RSAC

K8s control plane

API Server scheduler

etcd Controller manager

Node 1 Node 2 Node 3

Pod 𝑨𝑨∗𝟏𝟏 Pod C Pod E

Pod B Pod D Pod F

#RSAC

K8s control plane

API Server scheduler

etcd Controller manager

Node 1 Node 2 Node 3

Pod 𝑨𝑨∗𝟏𝟏 Pod C Pod E

Pod B Pod D Pod F

Pod 𝑨𝑨∗𝟐𝟐
#RSAC

K8s control plane

API Server scheduler

etcd Controller manager

Node 1 Node 2 Node 3

Pod 𝑨𝑨∗𝟏𝟏 Pod C Pod E

Pod B Pod D Pod F

Pod 𝑨𝑨∗𝟐𝟐 Pod 𝑨𝑨∗𝟑𝟑

#RSAC

K8s control plane

API Server scheduler

etcd Controller manager

Node 1 Node 2 Node 3

Pod 𝑨𝑨∗𝟏𝟏 Pod C Pod E

Pod B Pod D Pod F

Pod 𝑨𝑨∗𝟐𝟐 Pod 𝑨𝑨∗𝟑𝟑

#RSAC

Inner-cluster lateral movement

Other sensitive permissions (partial list):

Permission Allows to
Create pod\controllers Use a privileged service account in a new pod

Update controller Change the configuration to use a privileged service

account
Create secrets & get\list secret Create a new long-lived token for SA and read its
value
Create serviceaccounts/token Create a short-lived token for SA

33
Cluster-to-cloud lateral movement

#RSAC
#RSAC

Cluster-to-cloud lateral movement

• Workloads in Kubernetes may need access to cloud resources (for example: cloud
storage \ cloud secret store etc.)

• Managed clusters use cloud services for the cluster operation.

• How do workloads in Kubernetes authenticate with the cloud?

35
#RSAC

Cluster-to-cloud lateral movement

We’ll go over the following methods:

1. Storing cloud identity credential on the node

2. Direct access to IMDS
3. Indirect access to IMDS
4. Using OIDC (identity federation)

36
#RSAC

Cluster-to-cloud lateral movement

Storing cloud identity credential on the node

• Used to be the default authentication method in AKS in the past.

• In this method, each Kubernetes node stores a file with service principal (SPN) credentials.
SPNs are Azure application identities.

• By default, this SPN has Contributor role for the node resource group.

• Users can bring their own SPN or grant more permissions to the SPN if their applications need
access to more cloud resources. For example: add permissions to a cloud storage.

37
#RSAC

Cluster-to-cloud lateral movement

Storing cloud identity credential on the node

• Used to be the default authentication method in AKS in the past.

• In this method, each Kubernetes node stores a file with service principal (SPN) credentials.
SPNs are Azure application identities.

• By default, this SPN has Contributor role for the node resource group.

• Users can bring their own SPN or grant more permissions to the SPN if their applications need
access to more cloud resources. For example: add permissions to a cloud storage.

 Access to the node’s FS = Contributor permission on the resource group

38
#RSAC

Kubernetes Service Account K8s control plane

Azure Service principal API Server scheduler

etcd Controller manager

Node 1 Node 2 Node 3

Pod A Pod C Pod E

Pod B Pod D Pod F

#RSAC

Kubernetes Service Account K8s control plane

Azure Service principal API Server scheduler

etcd Controller manager

Node 1 Node 2 Node 3

Pod A Pod C Pod E

Pod B Pod D Pod F

#RSAC

Kubernetes Service Account K8s control plane

Azure Service principal API Server scheduler

etcd Controller manager

“Pod create” operation

In the pod config: mount the SPN

into the container

Node 1 Node 2 Node 3

Pod A Pod C Pod E

Pod B Pod D Pod F

#RSAC

Kubernetes Service Account K8s control plane

Azure Service principal API Server scheduler

etcd Controller manager

Node 1 Node 2 Node 3

Backdoor
pod

Pod A Pod C Pod E

Pod B Pod D Pod F

#RSAC

Kubernetes Service Account K8s control plane

Azure Service principal API Server scheduler

etcd Controller manager

Node 1 Node 2 Node 3

Backdoor
pod

Pod A Pod C Pod E

Pod B Pod D Pod F

#RSAC

Cluster-to-cloud lateral movement

Direct access to IMDS

• The metadata service is a special endpoint that is accessible to VMs, allowing them to retrieve
information about the VM.

• Implemented by all major cloud providers.

• Metadata service allows retrieving tokens for the cloud identity that is attached to the VM:
– Azure: 169.254.169.254/metadata/identity/oauth2
– AWS: 169.254.169.254/latest/meta-data/iam/security-credentials
– GCP: metadata.google.internal/computeMetadata/v1/instance/service-accounts

• Querying the metadata service doesn’t require any authentication.

44
#RSAC

Cluster-to-cloud lateral movement

Direct access to IMDS

• In managed K8s clusters, the nodes are VM which can access to their metadata service.

• By default, pods can access to the metadata service of their nodes.

• Thus, pods can acquire tokens of cloud identities attached to the nodes.

• The permissions of the identities depend on the cloud provider and the specific environment.

45
#RSAC

Cluster-to-cloud lateral movement

Direct access to IMDS - AKS

• AKS uses several managed identities to operate the cluster.

• Users can change the default permissions of those identities, or alternatively attach additional
managed identities to the nodes.

46
#RSAC

Cluster-to-cloud lateral movement

Direct access to IMDS - AKS

47
#RSAC

Cluster-to-cloud lateral movement

Direct access to IMDS - EKS

• EKS uses EC2 Roles for the Kubernetes nodes.

• By default, the EC2 role has the policies:

– AmazonEC2ContainerRegistryReadOnly - Pull permissions to the container registry.
– AmazonEKSWorkerNodePolicy - Read permissions to the compute environment (EC2, VPC etc.)
– AmazonEKS_CNI_Policy – Attach network interfaces and IPs to VMs

• Users can add more policies, if their containers require access to cloud resources.

48
#RSAC

Cluster-to-cloud lateral movement

Direct access to IMDS - GKE

• GKE uses IAM service accounts to authenticate with the cloud.

• By default, all the VMs in a project, including the Kubernetes nodes, share a default SA.

• This SA has Editor role for the project.

• While the access scope limits the permissions, they are still powerful by default:

49
#RSAC

Cluster-to-cloud lateral movement

Direct access to IMDS

How does lateral movement from the cluster to the cloud would look like?

50
#RSAC

K8s control plane

API Server scheduler

etcd Controller manager

Node 1 Node 2 Node 3

Pod A Pod C Pod E

Pod B Pod D Pod F

#RSAC

K8s control plane

API Server scheduler

etcd Controller manager

Node 1

Metadata
Pod A
service

Pod B
#RSAC

K8s control plane

API Server scheduler

etcd Controller manager

Node 1

Get token
Metadata
Pod A
service

Pod B
#RSAC

K8s control plane

API Server scheduler

etcd Controller manager

Node 1

Returns cloud
identity token
Metadata
Pod A
service

Pod B
#RSAC

K8s control plane

API Server scheduler

etcd Controller manager

List storage accounts \

storage buckets
Node 1

Metadata
Pod A
service

Pod B
#RSAC

K8s control plane

API Server scheduler

etcd Controller manager

List secrets from secret store

(Key Vault, KMS, etc)

Node 1

Metadata
Pod A
service

Pod B
#RSAC

K8s control plane

API Server scheduler

etcd Controller manager

Get credentials of
Kubernetes clusters
Node 1

Metadata
Pod A
service

Pod B
#RSAC

Cluster-to-cloud lateral movement

Direct access to IMDS

The problem:
• Pods can freely access to their node’s cloud identities.
• All pods share the same cloud identities (the node’s identities).

What we want:
• Allocate a specific identity to each pod (that needs access to cloud resources) with the minimal
needed permissions.
• Make sure pods can only acquire tokens for their own identities.

58
#RSAC

Cluster-to-cloud lateral movement

Indirect access to IMDS

1. Users allocate different identities to the various applications in the cluster.

2. When pods query IMDS, the traffic is intercepted and redirected to a local server in the cluster.

3. The local server is K8s-aware, thus can identify the querying pod.

4. The local server queries IMDS on behalf of the pod and request the pod-specific identity.

5. This concept was implemented in Azure by AAD Pod Identity [recently deprecated].

59
#RSAC

K8s control plane

API Server scheduler

etcd Controller manager

1. Requests to the metadata server are intercepted and

sent to the NMI pod (by modifying the IPTables of the
node).
2. NMI recognizes the querying pod and retrieves its
Node 1
token from the metadata server.
3. NMI send back the token to the pod

Pod A
Metadata 3
1
service 2 3
Pod B

Local server
(NMI)
#RSAC

K8s control plane

API Server scheduler

etcd Controller manager

Limitations
1. Works only for Linux containers (uses IPTables).
2. Not supported by all Kubernetes network configuration
(Prone to ARP poisoning) Node 1

Pod A
Metadata 3
1
service 2 3
Pod B

Local server
(NMI)
#RSAC

Cluster-to-cloud lateral movement

Using OIDC (identity federation)

• Implemented by all major cloud providers:

– Azure: AAD workload identity
– AWS: IAM roles for service accounts (IRSA)
– GCP: Workload identity

• The Kubernetes cluster is used as an OIDC identity provider (IdP).

• Trust relation is created such as the cloud identity service (e.g. AAD, AWS IAM, GCP IAM) trusts the
service accounts issued by the K8s cluster.

• This trust relation allows applications in the cluster to exchange a K8s service account token with a cloud
identity token.

62
#RSAC

Cluster-to-cloud lateral movement

Using OIDC (identity federation)

Cluster’s
Kubelet Pod Cloud identity service OIDC endpoint

Projects a signed service

account token to the pod

Sends the token to the cloud

identity service and request
to exchange it with cloud
identity token
Validates the service account
token

Returns a cloud identity token

63
#RSAC

Cluster-to-cloud lateral movement

Using OIDC (identity federation) - GCP

• In GCP, there’s a unified identity pool for the entire project.

• Meaning, there’s a single binding of a K8s service account (namespace + SA name) to a cloud identity.

Cluster A Cluster B

Namespace: Namespace:
monitoring monitoring

Service account: Service account:

sa-1 sa-1

MyCloudApp

gcloud iam service-accounts add-iam-policy-binding [email protected] \

--role roles/iam.workloadIdentityUser \
--member "serviceAccount:My-GCP-Project.svc.id.goog[monitoring/sa-1]"
#RSAC

Cluster-to-cloud lateral movement

Using OIDC (identity federation) - GCP
Cluster C

Cluster A Cluster B

Namespace: Namespace:
monitoring monitoring

Service account: Service account:

sa-1 sa-1

MyCloudApp
#RSAC

Cluster-to-cloud lateral movement

Using OIDC (identity federation) - GCP
Cluster C
Namespace: Create new namespace: monitoring
Monitoring
Create new Service Account: sa-1
Service account:
sa-1

Cluster A Cluster B

Namespace: Namespace:
monitoring monitoring

Service account: Service account:

sa-1 sa-1

MyCloudApp
#RSAC

Cluster-to-cloud lateral movement

Using OIDC (identity federation) - GCP
Cluster C
Namespace: Create new namespace: monitoring
Monitoring
Create new Service Account: sa-1
The security boundary is the Service account:
project, not the cluster! sa-1

Cluster A Cluster B

Namespace: Namespace:
monitoring monitoring

Service account: Service account:

sa-1 sa-1

MyCloudApp
Cross-cloud lateral movement

#RSAC
#RSAC

Cross-cloud lateral movement

Possible cross-cloud lateral movement scenarios:

• Multi-cloud Kubernetes clusters
• Multi-cloud workload authentication
• Multi-cloud supply-chain attack

69
#RSAC

Cross-cloud lateral movement

Possible cross-cloud lateral movement scenarios:

• Multi-cloud Kubernetes clusters
• Multi-cloud workload authentication
• Multi-cloud supply-chain attack

70
#RSAC

Cross-cloud lateral movement

Azure AWS

Pull images

Azure ACR EKS

71
#RSAC

Cross-cloud lateral movement

Azure AWS

Push image

Pull images

Azure ACR EKS

72
#RSAC

Cross-cloud lateral movement

Azure AWS

Push image

Cluster-to-
cloud lateral
movement

Pull images

Azure ACR EKS

73
#RSAC

Cross-cloud lateral movement

Azure AWS

AKS

Push image

Cluster-to-
cloud lateral
movement

Pull images

Azure ACR EKS

74
#RSAC

What have we seen so far?

• Inner-cluster lateral movement:

– Example: Self-update permissions that lead to cluster-takeover
– Additional permissions can lean to cluster takeover (partial list)

• Cluster-to-cloud lateral movement:

– Storing cloud identity credential on the node
– Direct access to IMDS
– Indirect access to IMDS
– Using OIDC (identity federation)

• Cross-cloud lateral movement:

– Multi-cloud supply-chain attack

75
Detections & Mitigations

#RSAC
#RSAC

Detections

Kubernetes control plane Cloud provider control plane

Monitor suspicious activity in the cluster using K8s Monitor suspicious activity of cloud identities used by K8s
Audit log (kube-audit). Examples: workloads\nodes. Examples:
1. Deployment of abnormal images 1. Abnormal behavior of cloud identities. Usually, the cloud
identities used by the workloads have a consistent
2. Pods with suspicious configurations behavior.
(sensitive volume mounts, privileged etc.)
2. Suspicious access to sensitive cloud services (e.g. storage,
3. Reconnaissance activity secret store etc.)
(for example: SelfSubjectRulesReview API call).
4. Sensitive API calls, such as “get secret”

77
#RSAC

Detections

Kubernetes control plane Cloud provider control plane

Monitor suspicious activity in the cluster using K8s Monitor suspicious activity of cloud identities used by K8s
Audit log (kube-audit). Examples: workloads\nodes. Examples:
1. Deployment of abnormal images 1. Abnormal behavior of cloud identities. Usually, the cloud
identities used by the workloads have a consistent
2. Pods with suspicious configurations behavior.
(sensitive volume mounts, privileged etc.)
2. Suspicious access to sensitive cloud services (e.g. storage,
3. Reconnaissance activity secret store etc.)
(for example: SelfSubjectRulesReview API call).
4. Sensitive API calls, such as “get secret”

Azure: Activity Log

AWS: CloudTrail
GCP: Cloud Audit Logs

78
#RSAC

Mitigations

• In December, a new version of the Threat Matrix for Kubernetes was released (v3): an open-source
knowledge base of attacking techniques of K8s.

• The new version includes now also mitigation techniques.

• http://aka.ms/KubernetesThreatMatrix

79
#RSAC

Mitigations

80
#RSAC

Mitigations

81
#RSAC

Mitigations

82
#RSAC

Mitigations

83
#RSAC

Mitigations

84
#RSAC

Mitigations

85
#RSAC

Mitigations

86
#RSAC

Mitigations

87
#RSAC

Mitigations

88
#RSAC

Mitigations

89
#RSAC

Mitigations

90
#RSAC

Mitigations

91
Key takeaways

#RSAC
#RSAC

Key takeaways

• Implement a holistic strategy for K8s security by considering both the cluster and cloud levels.

• Identities are a key aspect of K8s security: Monitor their activity using auditing tools.

• Adhere to the least-privilege principle.

• Use mitigation measures to prevent potential attacks.

93
Thank you!

#RSAC