Skip to content

Cilium-SPIFFE/SPIRE: Integration Guide

Why CIlium-SPIFFE/SPIRE integration

One great way to remove the overhead of secure communication and scaling the security issue will be to use SPIRE, which provides fine-grained, dynamic workload identity management. SPIRE provides the control plane to provision SPIFFE IDs to the workloads. These SPIFFE IDs can then be used for policy authorization. It enables teams to define and test policies for workloads operating on a variety of infrastructure types, including bare metal, public cloud (such as GCP), and container platforms (like Kubernetes).

Note: This integration modifies the following components: cilium-agent, cilium-envoy, and spire-agent.

The image below represents the summary of the actions performed in each of them.

Alt Text

The Scenario Setup

1. Create a cluster

  • GKE
    export  NAME="test-$RANDOM"
    gcloud container clusters create "${NAME}" --zone us-west2-a --image-type=UBUNTU
    gcloud container clusters get-credentials "${NAME}" --zone us-west2-a
    
  • MiniKube
    minikube start --network-plugin=cni --memory=4096
    minikube ssh -- sudo mount bpffs -t bpf /sys/fs/bpf
    

Note: It is assumed that the k8s cluster is already present/reachable and the user has the rights to create service accounts and cluster-role-bindings.

2. Deploy manifest (cilium-control-plane + spire-control-plane + dependencies).

kubectl apply -f https://raw.githubusercontent.com/accuknox/microservices-demo/main/cilium-spire/cilium-gke.yaml \
              -f https://raw.githubusercontent.com/accuknox/microservices-demo/main/cilium-spire/spire.yaml

3. Check the status of all the pods.

kubectl get pods -A | egrep -i 'spire|cilium'
kube-system   cilium-26c86                                           1/1     Running   0          16h
kube-system   cilium-jdx87                                           1/1     Running   0          16h
kube-system   cilium-node-init-2zrjf                                 1/1     Running   0          16h
kube-system   cilium-node-init-7xdl9                                 1/1     Running   0          16h
kube-system   cilium-node-init-lk4kz                                 1/1     Running   0          16h
kube-system   cilium-operator-7c97784647-nk86q                       1/1     Running   0          16h
kube-system   cilium-operator-7c97784647-xrhd2                       1/1     Running   0          16h
kube-system   cilium-spsm8                                           1/1     Running   0          16h
spire         spire-agent-2glk2                                      1/1     Running   0          13m
spire         spire-agent-bs679                                      1/1     Running   0          123m
spire         spire-agent-lv66b                                      1/1     Running   0          5h32m
spire         spire-server-0                                         1/1     Running   0          38m

Note: The spire-control plane (spire-agent and spire-server) should be running as well as the cilium-control plane.

Congratulations! You have a fully functional Kubernetes cluster with Cilium and SPIRE. 🎉

Example Scenario: Upgrading non-secure connections to mTLS

The goal of the scenario exposed by the image below is to upgrade the connection from HTTP to HTTPS. This tutorial uses the Star Wars scenario from Cilium, and, based on it, upgrades the connection between the pods xwing to deathstar. A Cilium Network Policy (CNP) is going to be applied to upgrade the connection. For this tutorial the following steps will be performed:

  1. Deploy the Star Wars scenario;
  2. Apply a CNP to upgrade the connection between xwing and deathstar;
  3. Do an HTTP test connection.

Note: for simplicity, in this tutorial, the upgrade happens just for HTTP connection originating from xwing.

Alt Text

1. Create spire registration entries:

curl -s https://raw.githubusercontent.com/accuknox/cilium-spire-tutorials/main/scenario03/0-create_registration_entries.sh | bash

2. Deploy Star Wars scenario (from Cilium tutorials):

kubectl apply -f https://raw.githubusercontent.com/accuknox/cilium-spire-tutorials/main/scenario03/1-http-sw-app.yaml

3. Check if the SPIFFE ID was assigned in xwing and deathstar pods.

The pod xwing must have a new label called spiffe://example.org/xwing and the pod deathstar a new label called spiffe://example.org/deathstar.

kubectl -n kube-system exec cilium-74m7n -- cilium endpoint list
Defaulted container "cilium-agent" out of: cilium-agent, mount-cgroup (init), clean-cilium-state (init)                                                                                       
ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                           IPv6       IPv4         STATUS                               
           ENFORCEMENT        ENFORCEMENT                                                                                                                                                     
100        Disabled           Disabled          4069       k8s:class=deathstar                                                   fd02::8e   10.0.0.176   ready                                
                                                           k8s:io.cilium.k8s.policy.cluster=default                                                                                           
                                                           k8s:io.cilium.k8s.policy.serviceaccount=starwars                                                                                   
                                                           k8s:io.kubernetes.pod.namespace=default                                                                                            
                                                           k8s:org=empire                                                                                                                     
                                                           spiffe://example.org/deathstar                                                                                                     
107        Disabled           Disabled          1          k8s:minikube.k8s.io/commit=0c397146a6e4f755686a1509562111cba05f46dd                           ready                                
                                                           k8s:minikube.k8s.io/name=minikube                                                                                                  
                                                           k8s:minikube.k8s.io/updated_at=2021_07_27T16_23_35_0700                                                                            
                                                           k8s:minikube.k8s.io/version=v1.21.0                                                                                                
                                                           k8s:node-role.kubernetes.io/control-plane                                                             
                                                           k8s:node-role.kubernetes.io/master                                                                     
                                                           reserved:host                                                                                          
732        Disabled           Disabled          4          reserved:health                                                       fd02::85   10.0.0.178   ready   
801        Disabled           Disabled          62228      k8s:class=xwing                                                       fd02::1c   10.0.0.115   ready   
                                                           k8s:io.cilium.k8s.policy.cluster=default                                                              
                                                           k8s:io.cilium.k8s.policy.serviceaccount=starwars                                                      
                                                           k8s:io.kubernetes.pod.namespace=default                                                               
                                                           k8s:org=alliance                                                                                       
                                                           spiffe://example.org/xwing                                                                             
871        Disabled           Disabled          26062      k8s:io.cilium.k8s.policy.cluster=default                                         10.0.0.72    ready   
                                                           k8s:io.cilium.k8s.policy.serviceaccount=coredns                                                       
                                                           k8s:io.kubernetes.pod.namespace=kube-system                                                           
                                                           k8s:k8s-app=kube-dns                                                                                   
3362       Disabled           Disabled          12147      k8s:app=spire-server                                                             10.0.0.140   ready   
                                                           k8s:io.cilium.k8s.policy.cluster=default                                                              
                                                           k8s:io.cilium.k8s.policy.serviceaccount=spire-server                                                  
                                                           k8s:io.kubernetes.pod.namespace=spire                                                                 
                                                           k8s:statefulset.kubernetes.io/pod-name=spire-server-0     
Now, we need to enforce a policy to upgrade the connection for the egress traffic from the xwing pod to port 80 and downgrade the connection for the ingress traffic to port 80 in deathstar.

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "tls-upgrade-xwing"
spec:
  endpointSelector:
    matchLabels:
      class: xwing
  egress:
  - toPorts:
    - ports:
      - port: "80"
        protocol: "TCP"
      originatingTLS:
        spiffe:
          peerIDs:
            - spiffe://example.org/deathstar
      rules:
        http:
        - {}
---
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "tls-upgrade-deathstar"
spec:
  endpointSelector:
    matchLabels:
      class: deathstar
  ingress:
  - toPorts:
    - ports:
      - port: "80"
        protocol: "TCP"
      terminatingTLS:
        spiffe:
          peerIDs:
            - spiffe://example.org/xwing
      rules:
        http:
        - {}
---
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "enable-xwing-dns"
spec:
  description: "Enable DNS traffic for xwing"
  endpointSelector:
    matchLabels:
     org: alliance
     class: xwing
  egress:
    - toPorts:
      - ports:
        - port: "53"
          protocol: UDP 

Note: The last policy is related to allowing the DNS traffic for swing. It will be used later for HTTP requests.

4. Apply Cilium Network Policies (CNP):

kubectl apply -f https://raw.githubusercontent.com/accuknox/cilium-spire-tutorials/main/scenario03/2-mtls-upgrade.yaml

5. Check if the policies were enforced in the xwing and deathstar endpoints.

Just to remember, for pod xwing we apply a policy just for egress and for pod deathstar, just for ingress. Just keep it in mind when looking at the column ENFORCEMENT.

kubectl exec cilium-74m7n -- cilium endpoint list
Defaulted container "cilium-agent" out of: cilium-agent, mount-cgroup (init), clean-cilium-state (init)                                                                                       
ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                           IPv6       IPv4         STATUS                               
           ENFORCEMENT        ENFORCEMENT                                                                                                                                                     
100        Enabled            Disabled          4069       k8s:class=deathstar                                                   fd02::8e   10.0.0.176   ready                                
                                                           k8s:io.cilium.k8s.policy.cluster=default                                                                                           
                                                           k8s:io.cilium.k8s.policy.serviceaccount=starwars                                                                                   
                                                           k8s:io.kubernetes.pod.namespace=default                                                                                            
                                                           k8s:org=empire                                                                                                                     
                                                           spiffe://example.org/deathstar                                                                                                     
107        Disabled           Disabled          1          k8s:minikube.k8s.io/commit=0c397146a6e4f755686a1509562111cba05f46dd                           ready                                
                                                           k8s:minikube.k8s.io/name=minikube                                                                                                  
                                                           k8s:minikube.k8s.io/updated_at=2021_07_27T16_23_35_0700                                                                            
                                                           k8s:minikube.k8s.io/version=v1.21.0                                                                                                
                                                           k8s:node-role.kubernetes.io/control-plane                                                                                          
                                                           k8s:node-role.kubernetes.io/master                                                                                                 
                                                           reserved:host                                                                                                                      
732        Disabled           Disabled          4          reserved:health                                                       fd02::85   10.0.0.178   ready                                
801        Disabled           Enabled           62228      k8s:class=xwing                                                       fd02::1c   10.0.0.115   ready                                
                                                           k8s:io.cilium.k8s.policy.cluster=default                                                                                           
                                                           k8s:io.cilium.k8s.policy.serviceaccount=starwars                                                                                   
                                                           k8s:io.kubernetes.pod.namespace=default                                                                                            
                                                           k8s:org=alliance                                                                                                                   
                                                           spiffe://example.org/xwing                                                                                                         
871        Disabled           Disabled          26062      k8s:io.cilium.k8s.policy.cluster=default                                         10.0.0.72    ready                                
                                                           k8s:io.cilium.k8s.policy.serviceaccount=coredns                                                       
                                                           k8s:io.kubernetes.pod.namespace=kube-system                                                           
                                                           k8s:k8s-app=kube-dns                                                                                   
3362       Disabled           Disabled          12147      k8s:app=spire-server                                                             10.0.0.140   ready   
                                                           k8s:io.cilium.k8s.policy.cluster=default                                                              
                                                           k8s:io.cilium.k8s.policy.serviceaccount=spire-server                                                  
                                                           k8s:io.kubernetes.pod.namespace=spire                                                                 
                                                           k8s:statefulset.kubernetes.io/pod-name=spire-server-0  

6. Send landing request.

The following script is going to perform an HTTP request. This connection is going to be upgraded to HTTPS.

curl -s https://raw.githubusercontent.com/accuknox/cilium-spire-tutorials/main/scenario03/4-curl.sh | bash

If the execution was succeeded, the command will return Ship landed.

Note: one simple way to verify the encryption traffic is the following. Without the policy applied the traffic from this lab doesn't go through cilium_host. After applying the policy, the cilium_host receives the encrypted traffic.

7. Using TCPdump to verify the connection

  1. Login in minikube minikube ssh;
  2. Inside minikube, download tcpdump apt-get update & apt-get install tcpdump;
  3. Capture the traffic from the interface cilium_host tcpdump -i cilium_host port 80.

8. Clean up the pods and CNPs:

curl -s https://raw.githubusercontent.com/accuknox/cilium-spire-tutorials/main/scenario03/5-clean-all.sh | bash

Further Reading

To learn more about Cilium SPIRE/SPIFFE integrations have a look at our Blogs and GitHub.

Back to top