Cilium-SPIFFE/SPIRE: Integration Guide¶
Why CIlium-SPIFFE/SPIRE integration¶
One great way to remove the overhead of secure communication and scaling the security issue will be to use SPIRE, which provides fine-grained, dynamic workload identity management. SPIRE provides the control plane to provision SPIFFE IDs to the workloads. These SPIFFE IDs can then be used for policy authorization. It enables teams to define and test policies for workloads operating on a variety of infrastructure types, including bare metal, public cloud (such as GCP), and container platforms (like Kubernetes).
Note: This integration modifies the following components: cilium-agent, cilium-envoy, and spire-agent.
The image below represents the summary of the actions performed in each of them.
The Scenario Setup¶
1. Create a cluster¶
- GKE
export NAME="test-$RANDOM" gcloud container clusters create "${NAME}" --zone us-west2-a --image-type=UBUNTU gcloud container clusters get-credentials "${NAME}" --zone us-west2-a - MiniKube
minikube start --network-plugin=cni --memory=4096 minikube ssh -- sudo mount bpffs -t bpf /sys/fs/bpf
Note: It is assumed that the k8s cluster is already present/reachable and the user has the rights to create service accounts and cluster-role-bindings.
2. Deploy manifest (cilium-control-plane + spire-control-plane + dependencies).¶
kubectl apply -f https://raw.githubusercontent.com/accuknox/microservices-demo/main/cilium-spire/cilium-gke.yaml \
-f https://raw.githubusercontent.com/accuknox/microservices-demo/main/cilium-spire/spire.yaml
3. Check the status of all the pods.¶
kubectl get pods -A | egrep -i 'spire|cilium'
kube-system cilium-26c86 1/1 Running 0 16h
kube-system cilium-jdx87 1/1 Running 0 16h
kube-system cilium-node-init-2zrjf 1/1 Running 0 16h
kube-system cilium-node-init-7xdl9 1/1 Running 0 16h
kube-system cilium-node-init-lk4kz 1/1 Running 0 16h
kube-system cilium-operator-7c97784647-nk86q 1/1 Running 0 16h
kube-system cilium-operator-7c97784647-xrhd2 1/1 Running 0 16h
kube-system cilium-spsm8 1/1 Running 0 16h
spire spire-agent-2glk2 1/1 Running 0 13m
spire spire-agent-bs679 1/1 Running 0 123m
spire spire-agent-lv66b 1/1 Running 0 5h32m
spire spire-server-0 1/1 Running 0 38m
Note: The spire-control plane (spire-agent and spire-server) should be running as well as the cilium-control plane.
Congratulations! You have a fully functional Kubernetes cluster with Cilium and SPIRE. 🎉
Example Scenario: Upgrading non-secure connections to mTLS¶
The goal of the scenario exposed by the image below is to upgrade the connection from HTTP to HTTPS. This tutorial uses the Star Wars scenario from Cilium, and, based on it, upgrades the connection between the pods xwing to deathstar. A Cilium Network Policy (CNP) is going to be applied to upgrade the connection. For this tutorial the following steps will be performed:
- Deploy the Star Wars scenario;
- Apply a CNP to upgrade the connection between
xwinganddeathstar; - Do an HTTP test connection.
Note: for simplicity, in this tutorial, the upgrade happens just for HTTP connection originating from xwing.
1. Create spire registration entries:¶
curl -s https://raw.githubusercontent.com/accuknox/cilium-spire-tutorials/main/scenario03/0-create_registration_entries.sh | bash
2. Deploy Star Wars scenario (from Cilium tutorials):¶
kubectl apply -f https://raw.githubusercontent.com/accuknox/cilium-spire-tutorials/main/scenario03/1-http-sw-app.yaml
3. Check if the SPIFFE ID was assigned in xwing and deathstar pods.¶
The pod xwing must have a new label called spiffe://example.org/xwing and the pod deathstar a new label called spiffe://example.org/deathstar.
kubectl -n kube-system exec cilium-74m7n -- cilium endpoint list
Defaulted container "cilium-agent" out of: cilium-agent, mount-cgroup (init), clean-cilium-state (init)
ENDPOINT POLICY (ingress) POLICY (egress) IDENTITY LABELS (source:key[=value]) IPv6 IPv4 STATUS
ENFORCEMENT ENFORCEMENT
100 Disabled Disabled 4069 k8s:class=deathstar fd02::8e 10.0.0.176 ready
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=starwars
k8s:io.kubernetes.pod.namespace=default
k8s:org=empire
spiffe://example.org/deathstar
107 Disabled Disabled 1 k8s:minikube.k8s.io/commit=0c397146a6e4f755686a1509562111cba05f46dd ready
k8s:minikube.k8s.io/name=minikube
k8s:minikube.k8s.io/updated_at=2021_07_27T16_23_35_0700
k8s:minikube.k8s.io/version=v1.21.0
k8s:node-role.kubernetes.io/control-plane
k8s:node-role.kubernetes.io/master
reserved:host
732 Disabled Disabled 4 reserved:health fd02::85 10.0.0.178 ready
801 Disabled Disabled 62228 k8s:class=xwing fd02::1c 10.0.0.115 ready
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=starwars
k8s:io.kubernetes.pod.namespace=default
k8s:org=alliance
spiffe://example.org/xwing
871 Disabled Disabled 26062 k8s:io.cilium.k8s.policy.cluster=default 10.0.0.72 ready
k8s:io.cilium.k8s.policy.serviceaccount=coredns
k8s:io.kubernetes.pod.namespace=kube-system
k8s:k8s-app=kube-dns
3362 Disabled Disabled 12147 k8s:app=spire-server 10.0.0.140 ready
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=spire-server
k8s:io.kubernetes.pod.namespace=spire
k8s:statefulset.kubernetes.io/pod-name=spire-server-0
xwing pod to port 80 and downgrade the connection for the ingress traffic to port 80 in deathstar.
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
name: "tls-upgrade-xwing"
spec:
endpointSelector:
matchLabels:
class: xwing
egress:
- toPorts:
- ports:
- port: "80"
protocol: "TCP"
originatingTLS:
spiffe:
peerIDs:
- spiffe://example.org/deathstar
rules:
http:
- {}
---
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
name: "tls-upgrade-deathstar"
spec:
endpointSelector:
matchLabels:
class: deathstar
ingress:
- toPorts:
- ports:
- port: "80"
protocol: "TCP"
terminatingTLS:
spiffe:
peerIDs:
- spiffe://example.org/xwing
rules:
http:
- {}
---
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
name: "enable-xwing-dns"
spec:
description: "Enable DNS traffic for xwing"
endpointSelector:
matchLabels:
org: alliance
class: xwing
egress:
- toPorts:
- ports:
- port: "53"
protocol: UDP
Note: The last policy is related to allowing the DNS traffic for swing. It will be used later for HTTP requests.
4. Apply Cilium Network Policies (CNP):¶
kubectl apply -f https://raw.githubusercontent.com/accuknox/cilium-spire-tutorials/main/scenario03/2-mtls-upgrade.yaml
5. Check if the policies were enforced in the xwing and deathstar endpoints.¶
Just to remember, for pod xwing we apply a policy just for egress and for pod deathstar, just for ingress. Just keep it in mind when looking at the column ENFORCEMENT.
kubectl exec cilium-74m7n -- cilium endpoint list
Defaulted container "cilium-agent" out of: cilium-agent, mount-cgroup (init), clean-cilium-state (init)
ENDPOINT POLICY (ingress) POLICY (egress) IDENTITY LABELS (source:key[=value]) IPv6 IPv4 STATUS
ENFORCEMENT ENFORCEMENT
100 Enabled Disabled 4069 k8s:class=deathstar fd02::8e 10.0.0.176 ready
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=starwars
k8s:io.kubernetes.pod.namespace=default
k8s:org=empire
spiffe://example.org/deathstar
107 Disabled Disabled 1 k8s:minikube.k8s.io/commit=0c397146a6e4f755686a1509562111cba05f46dd ready
k8s:minikube.k8s.io/name=minikube
k8s:minikube.k8s.io/updated_at=2021_07_27T16_23_35_0700
k8s:minikube.k8s.io/version=v1.21.0
k8s:node-role.kubernetes.io/control-plane
k8s:node-role.kubernetes.io/master
reserved:host
732 Disabled Disabled 4 reserved:health fd02::85 10.0.0.178 ready
801 Disabled Enabled 62228 k8s:class=xwing fd02::1c 10.0.0.115 ready
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=starwars
k8s:io.kubernetes.pod.namespace=default
k8s:org=alliance
spiffe://example.org/xwing
871 Disabled Disabled 26062 k8s:io.cilium.k8s.policy.cluster=default 10.0.0.72 ready
k8s:io.cilium.k8s.policy.serviceaccount=coredns
k8s:io.kubernetes.pod.namespace=kube-system
k8s:k8s-app=kube-dns
3362 Disabled Disabled 12147 k8s:app=spire-server 10.0.0.140 ready
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=spire-server
k8s:io.kubernetes.pod.namespace=spire
k8s:statefulset.kubernetes.io/pod-name=spire-server-0
6. Send landing request.¶
The following script is going to perform an HTTP request. This connection is going to be upgraded to HTTPS.
curl -s https://raw.githubusercontent.com/accuknox/cilium-spire-tutorials/main/scenario03/4-curl.sh | bash
If the execution was succeeded, the command will return Ship landed.
Note: one simple way to verify the encryption traffic is the following. Without the policy applied the traffic from this lab doesn't go through cilium_host. After applying the policy, the cilium_host receives the encrypted traffic.
7. Using TCPdump to verify the connection¶
- Login in minikube
minikube ssh; - Inside minikube, download tcpdump
apt-get update & apt-get install tcpdump; - Capture the traffic from the interface cilium_host
tcpdump -i cilium_host port 80.
8. Clean up the pods and CNPs:¶
curl -s https://raw.githubusercontent.com/accuknox/cilium-spire-tutorials/main/scenario03/5-clean-all.sh | bash
Further Reading¶
To learn more about Cilium SPIRE/SPIFFE integrations have a look at our Blogs and GitHub.