Kubernetes’ Architecture

Deep Dive
Lucas Käldström - CNCF Ambassador
7th of May 2019 - Umeå
1 Image credit: @ashleymcnamara
$ whoami
Lucas Käldström, High School Student, 19 years old

CNCF Ambassador, Certified Kubernetes

Administrator and Kubernetes SIG Lead

KubeCon Speaker in Berlin, Austin,

Copenhagen, Shanghai & Seattle

Kubernetes approver and subproject owner, active in

the community for ~3 years. Got kubeadm to GA.

Driving luxas labs which currently performs

contracting for Weaveworks

A guy that has never attended a computing class

2
An intro to CNCF
Cloud Native Computing Foundation helps us all succeed
= OPEN SOURCE CLOUD COMPUTING FOR APPLICATIONS

4
Source

Cloud Native Computing Foundation

• Nonprofit, part of the Linux Foundation; founded Dec 2015
Graduated Incubating

Orchestration Monitoring Distributed Tracing Logging Remote Container Runtime Container Runtime Networking API Distributed Tracing Software Update
API Procedure Call Spec

Service Proxy Security Storage Service Discovery Messaging Service Mesh Package Storage Registry Key/Value
Management Store

• Platinum members:

5
Source
CNCF Project Maturities
LATE MAJORITY
GRADUATED “CONSERVATIVES”

“THE CHASM”

INCUBATING
INNOVATORS
“TECHIES”

SANDBOX

EARLY ADOPTERS EARLY MAJORITY LAGGARDS

“VISIONARIES” “PRAGMATISTS” “SKEPTICS”

SANDBOX

Identity Spec Identity Policy Serverless Tooling Metrics Spec Key/Value Store Monitoring Packaging Container Image Nodeless
6 Spec Security Distribution
7
Cloud Native
Trail Map

Trail Map: l.cncf.io

8 Source
Source

CNCF Cloud Native Definition v1.0

Cloud native technologies empower organizations to build and run scalable

applications in modern, dynamic environments such as public, private, and
hybrid clouds. Containers, service meshes, microservices, immutable
infrastructure, and declarative APIs exemplify this approach.
These techniques enable loosely coupled systems that are resilient,
manageable, and observable. Combined with robust automation, they
allow engineers to make high-impact changes frequently and predictably
with minimal toil.
The Cloud Native Computing Foundation seeks to drive adoption of this
paradigm by fostering and sustaining an ecosystem of open source,
vendor-neutral projects. We democratize state-of-the-art patterns to make
these innovations accessible for everyone.

9
Online, Proctored Kubernetes Exams

• Certified Kubernetes Administrator (CKA)

– Over 3,500 registrations already
– https://www.cncf.io/certification/expert/cka/
• Certified Kubernetes Application Developer
(CKAD)
– Certifies that users can design, build, configure, and expose
cloud native applications for Kubernetes
– https://www.cncf.io/certification/expert/cka/ckad/
• Both tests
– Tests consist of a set of scenarios to resolve from the
command line over 3 hours; there is no multiple choice
– Each exam is $300
– Quarterly exam updates to match K8s releases`

10 Source
Certified Kubernetes Conformance

• CNCF runs a software conformance

program for Kubernetes
– Implementations run conformance tests and
upload results
– New mark and more flexible use of
Kubernetes trademark for conformant
implementations
– cncf.io/ck

11 Source
75 Certified Kubernetes Partners

12 Source
KubeCon + CloudNativeCon
• Europe 2019 (sponsorships and CFP open)
– Barcelona: May 20-23, 2019

• China 2019 (sponsorships open)

– Shanghai: June 24-26, 2019

• North America 2019 (sponsorships open)

– San Diego: November 18-21, 2019

13 Source
KubeCon + CloudNativeCon Attendance

First CNCF-organized event

14 Source
Kubernetes on a
high-level
Kubernetes lets you efficiently declaratively manage your apps at any scale
Most importantly: What does “Kubernetes” mean?

= Greek for “pilot” or

“helmsman of a ship”

16
What is Kubernetes?
= A Production-Grade Container Orchestration System

● A project that was spun out of Google as an open source

container orchestration platform (~2 billion containers/week).
● Built from the lessons learned in the experiences of
developing and running Google’s Borg and Omega.
● Designed from the ground-up as a loosely coupled collection
of components centered around deploying, maintaining and
scaling workloads.
17
What Does Kubernetes do?

● Known as the linux kernel of distributed systems.

● Abstracts away the underlying hardware of the
nodes and provides a uniform interface for
workloads to be both deployed and consume the
shared pool of resources.
● Works as an engine for resolving state by
converging the actual and the desired state of the
system.
18
Kubernetes is self-healing

Kubernetes will ALWAYS try and steer the cluster to its

desired state.

● Me: “I want 3 healthy instances of Redis to always

be running.”
● Kubernetes: “Okay, I’ll ensure there are always 3
instances up and running.”
● Kubernetes: “Oh look, one has died. I’m going to
19
attempt to spin up a new one.”
What can Kubernetes REALLY do?

● Autoscale Workloads
● Blue/Green Deployments
● Fire off Jobs and scheduled CronJobs
● Manage Stateless and Stateful Applications
● Built-in Service Discovery
● ~Easily integrate and support 3rd party apps~

20
Most Importantly...

Use the SAME API

across bare metal and
EVERY cloud provider!!!

21
Kubernetes’ incredible velocity (last 365 days!)

32 000+ 15 000+
human commits
50 000+ contributors
55 000+ edX course enrolls
users on Slack
51 000+
opened
Pull Requests
318 000+
35 000+ Github comments
Kubernetes jobs 73 000+
opened issues

88 000+
Kubernetes
professionals Last updated: 09.01.2019

22 Source 1 Source 2 Source 3 Source 4 Source 5

Kubernetes is a “platform-platform”

23
Kubernetes’
Architecture
Kubernetes’ high-level component architecture
Control Plane etcd (key-value DB, SSOT)

Controller Manager Scheduler

User (Controller Loops) API Server (REST API) (Bind Pod to Node)

Nodes
Legend: Networking Networking Networking
CNI
CRI
Kubelet Kubelet Kubelet
OCI Container Container Container
Protobuf
gRPC Runtime Runtime Runtime
JSON
OS OS OS
Node 1 Node 2 Node 3

25
kube-apiserver, the heart of the cluster
● Provides a forward facing REST interface into the
Kubernetes control plane and datastore.
● All clients and other applications interact with
Kubernetes strictly through the API Server.
● Acts as the gatekeeper to the cluster by handling
authentication and authorization, request
validation, mutation, and admission control in
addition to being the front-end to the backing
26 datastore.
etcd, the key-value datastore

● etcd acts as the cluster datastore.

● A standalone incubating CNCF project
● Purpose in relation to Kubernetes is to provide a
strong, consistent and highly available key-value
store for persisting all cluster state.
● Uses “Raft Consensus” among a quorum of
systems to create a fault-tolerant
27
consistent “view” of the cluster.
kube-controller-manager, the reconciliator

● Serves as the primary daemon that manages all

core components’ reconcilation loops.
● Handles a lot of the business logic of Kubernetes.
● Monitors the cluster state via the API Server and
steers the cluster towards the desired state.
● List of core controllers

28
kube-scheduler, the placement engine

● Verbose policy-rich engine that evaluates workload

requirements and attempts to place it on a
matching resource.
● The default scheduler uses the “binpacking” mode.
● Workload Requirements can include: general
hardware requirements, affinity/anti-affinity, labels,
and other various custom resource requirements.
● Is swappable, you can create your own scheduler
29
kubelet, the node agent

● Acts as the node agent responsible for managing

the lifecycle of every pod on its host.
● Kubelet understands JSON/YAML container
manifests that it can read from several sources:
○ Watching the API server (the primary mode)
○ A directory with files
○ A HTTP Endpoint
○ HTTP Server mode accepting container
30
manifests over a simple API.
Container Runtime, the executor

● A container runtime is a CRI (Container Runtime

Interface) compatible application that executes and
manages containers.
○ Docker (default, built into the kubelet atm)
○ containerd
○ cri-o
○ rkt
○ Kata Containers (formerly clear and hyper)
31
○ Virtlet (VM CRI compatible runtime)
Container Network Interface (CNI)

● Pod networking within Kubernetes is plumbed via

the Container Network Interface (CNI).
● Functions as an interface between the container
runtime and a network implementation plugin.
● CNCF Project
● Uses a simple JSON Schema.

32
Kubernetes Networking

● Pod Network (third-party implementation)

○ Cluster-wide network used for pod-to-pod
communication managed by a CNI (Container
Network Interface) plugin.
● Service Network (kube-proxy)
○ Cluster-wide range of Virtual IPs managed by
kube-proxy for service discovery.

33
kube-proxy, the Service proxier

● Manages the network rules for Services on each

node.
● Performs connection forwarding or load balancing
for Kubernetes Services.
● Available Proxy Modes:
○ ipvs (default if supported)
○ iptables (default fallback)
○ userspace (legacy)
34
Third-party CNI Plugins for Pod Networking

● Amazon ECS ● GCE

● Calico ● kube-router
● Cillium ● Multus
● Contiv ● OpenVSwitch
● Contrail ● Romana
● Flannel ● Weave Net

35
Cluster DNS, today CoreDNS

● Provides Cluster Wide DNS for Kubernetes Services.

○ CoreDNS (current default)
○ kube-dns (default pre-1.13)
● Resolves `{name}.{namespace}.svc.cluster.local`
queries to the Service Virtual IPs.

36
The Kubernetes Dashboard

A limited, general purpose

web front end for the
Kubernetes Cluster.

37
Kubernetes is extensible (!)
● CNI (Container Network Interface) - write your own network plugin for the
cluster
● CRI (Container Runtime Interface) - write your own container runtime for
Kubernetes
● CSI (Container Storage Interface) - write your own persistent storage plugin
● Device Plugins - register custom devices present on your Node
● Pluggable cloud providers - write your own extension that integrates with your
cloud
● CustomResourceDefinitions - create your own Kubernetes Resources easily
● API Aggregation - proxy a new API group from the core API server to your
38 extension
Kubernetes Applied
Dive into how to use Kubernetes in practice
Create a cluster with kubeadm
1. Provision a Linux machine
2. Install kubeadm from package repos or by hand:
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
echo “deb http://apt.kubernetes.io/ kubernetes-xenial main” > /etc/apt/sources.list.d/k8s.list
apt-get update && apt-get install -y kubeadm docker.io

3. Bootstrap the Control Plane Node easily:

kubeadm init

4. Install a 3rd-party Pod Network Provider (e.g. Weave Net):

kubectl apply -f https://git.io/weave-kube-1.6

5. Repeat step 1 & 2 on any Node to join to the cluster:

40
kubeadm join --token <token> <master-ip>:6443
kubeadm
= The official tool to bootstrap a minimum viable, best-practice Kubernetes cluster

Cluster API Cluster API Spec Cluster API Implementation

Layer 3
Addons Addon Operators
Cloud Provider Load Balancers Monitoring Logging

Kubernetes API
Layer 2
Bootstrapping kubeadm
kubeadm kubeadm kubeadm kubeadm

Machines Master 1 Master N Node 1 Node N

Layer 1
Cluster API
Infrastructure

41
kubeadm vs an “end-to-end solution”
kubeadm is built to be part of a higher-level solution

end-to-end solution

Cluster API Cluster API Spec Cluster API Implementation

Addons Cloud Provider Load Balancers Monitoring Logging

Kubernetes API

Bootstrapping kubeadm kubeadm kubeadm kubeadm

Machines Master 1 Master N Node 1 Node N

Infrastructure

42
75 Certified Kubernetes Partners

43 Source
The core primitive: A Pod
apiVersion: v1
● The basic, atomically deployable unit. kind: Pod
metadata:
name: nginx
● Consists of one or many co-located namespace: default
labels:
containers. app: nginx
spec:
containers:
● Represents a single instance of an - image: nginx:1.13.9
name: nginx
application. ports:
- name: http
containerPort: 80
44
The core primitive: A Pod
apiVersion: v1
kind: Pod
metadata:
● Has its own random internal IP address name: nginx
namespace: default
● Containers share network & volumes labels:
app: nginx

● Is non-mutable, hence mortal; dies on spec:

containers:
- image: nginx:1.13.9
Node shutdown or rolling upgrades name: nginx
ports:
- name: http
containerPort: 80
45
A replicated, upgradeable set of Pods: A
Deployment apiVersion: apps/v1
kind: Deployment
metadata:
labels:
● Manages Pods in a app: nginx
name: nginx
spec:
declarative & upgradable manner replicas: 3
selector:
matchLabels:
● Creates $replicas Pods from a template app: nginx
template:
metadata:
Pod Template

● Is mutable, performs a rolling upgrade of labels:

app: nginx
spec:

the Pods when changed

containers:
- image: nginx:1.13.9
name: nginx
ports:
- name: http
46 containerPort: 80
Various possible Deployment upgrade strategies

The built-in Deployment

behavior

The other strategies

can be implemented
fairly easily by talking to
the API.

47 Picture source: Kubernetes effect by Bilgin Ibryam

Access your replicated Pods via a Service
apiVersion: v1
● Provides a stable, immortal front-end IP kind: Service
metadata:
name: nginx
address for a set of Pods. namespace: default
labels:
● Loadbalances traffic to any Pod app: nginx
spec:
type: ClusterIP
matching the key-value label selector ports:
- name: http
(here: app=nginx) port: 80
targetPort: 80
selector: Pod Selector

48
app: nginx
Access your replicated Pods via a Service
apiVersion: v1
kind: Service
● Accessible in the cluster through its IP metadata:
name: nginx
namespace: default
address or via its internal DNS name labels:
app: nginx
(here: nginx.default.svc.cluster.local) spec:
type: ClusterIP
ports:
● Can expose multiple ports - name: http
port: 80
targetPort: 80
selector: Pod Selector

49
app: nginx
Expose your Service to the world with an Ingress
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: nginx
● Expose your Service to the outer namespace: default
labels:

world with Ingress app: nginx

spec:
rules:
● Declare in a Kubernetes-native way - host: nginx.kubernetesfinland.com
http:
paths:
how to proxy traffic to the Service - path: /
backend: Service Reference
serviceName: nginx
servicePort: 80

50
Expose your Service to the world with an Ingress
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: nginx
● The Ingress Implementation is namespace: default
labels:

3rd-party; e.g. Traefik or nginx app: nginx

spec:
rules:
● It’s recommended to run the - host: nginx.kubernetesfinland.com
http:
paths:
implementation itself in Kubernetes - path: /
backend: Service Reference
serviceName: nginx
servicePort: 80

51
Internet

Isolate your stuff in a Namespace

nginx.demo.kubernetesfinland.com

● A logical isolation method for resources; Traefik as Ingress Controller

Namespace: default
used for grouping and policy
nginx Ingress Rule
● Role Based Access Control (RBAC), and
nginx Service
other Policies operate based on
nginx nginx nginx
Namespaces Pod 1 Pod 2 Pod 3

nginx Deployment
52
What happens
when you create
a Deployment?

Note that the API server (and the backing

key-value DB) are involved in virtually any
operation in the cluster.

The “business logic” components (the

controller-manager and scheduler) acts
when something interesting happens.

Finally, the Kubelet receives the message

it should run a Pod; and delegates that
task to a container runtime (like docker)
via a daemon shim implementing the
Container Runtime Interface (CRI).

53
Kubernetes’ Community
Navigate the community with this map
How do I kick the tires with Kubernetes?

Play with Kubernetes right away in your browser!

Create a single-node cluster on your workstation with minikube

Create a production-ready cluster on any machines with kubeadm

Test out Kubernetes easily locally with kind (Kubernetes IN Docker)

Explore Kubernetes Certified Service Providers’ solutions

55
Everything is done in Special Interest Groups
Special Interest Groups (SIGs)
manage Kubernetes’ various
components and features.

All code in the Kubernetes Github

organization should be owned by one
or more SIGs; with directory-level
granularity.

SIGs have regular (often weekly)

video meetings where the attendees
discuss design decisions, new
features, bugs, testing, onboarding or
whatever else that is relevant to the
group. Attending these meetings is
the best way to get to know the
project
56 Image source
Where is the community doing work?
1. On Slack: https://kubernetes.slack.com & http://slack.k8s.io
2. On Github: https://github.com/kubernetes (in ~150 repositories)
3. In weekly Special Interest Group (SIG), Working Group (WG) and Subproject meetings
Check out the community repo: https://github.com/kubernetes/community
Youtube archive of ALL meetings available in the Kubernetes Youtube channel
4. In the Kubernetes Community Meeting every Thursday 6pm - 7pm UTC
https://github.com/kubernetes/community/blob/master/events/community-meeting.md
5. On the Discuss Kubernetes site: https://discuss.kubernetes.io/
6. In Google Groups in #kubernetes-*, e.g. https://groups.google.com/forum/#!forum/kubernetes-dev
7. At meetups worldwide (~170 meetups / ~40 countries): https://www.meetup.com/pro/cncf/
8. In the community Office Hours and Meet Our Contributors - Ask Us Anything series
57
How to start contributing?
1. Read as much of https://github.com/kubernetes/community as you can!!
2. Read the CONTRIBUTORS GUIDE
3. Find a SIG you’re interested to contribute to (e.g. SIG Cluster Lifecycle)
4. Create a Github account if you haven’t done so and sign up to the Kubernetes Slack: slack.k8s.io
5. Join the #sig-foo channels in Slack you’re interested in and #kubernetes-dev
6. Sign up to the #kubernetes-dev, #kubernetes-announce and #kubernetes-sig-foo mailing lists
7. Watch the New Contributor Series on Youtube from a New Contributor Summit
8. Check out the Codebase Tour in the “Meet Our Contributors” Youtube series
9. Learn how to use Kubernetes in more detail in this Free Kubernetes Course on edx.org
10. Join a weekly SIG meeting using Zoom and let them know you’d like to contribute :)

58
Next step: Check out this AWESOME
182-slide Kubernetes workshop presentation,
made by Bob Killen and Jeffrey Sica from the
University of Michigan.

Kubernetes

An Introduction

CC-BY 4.0
59
Kubernetes v1.12 11/2018
Various excellent blog posts
Core Kubernetes: Jazz Improv over Orchestration - Joe Beda, 30th of May
2017
Kubernetes deep dive: API Server, Part 1 - Michael Hausenblas & Stefan
Schimanski, 28th April 2017
Kubernetes deep dive: API Server, Part 2 - Michael & Stefan, 21st July 2017
Kubernetes deep dive: API Server, Part 3 - Michael & Stefan, 15th August
2017
Reasons Kubernetes is cool - Julia Evans, 5th October 2017
How Kubernetes certificate authorities work - Julia Evans, 5th August
2017
Operating a Kubernetes network - Julia Evans, 10th October 2017
60
Join the community in Barcelona, Shanghai and San Diego!

61
Thank you!
@luxas on Github
@luxas on Kubernetes’ Slack
@kubernetesonarm on Twitter
[email protected]