Skip to content

KubeArmor and Cilium on MicroK8's Cluster

Overview

This user journey guides you to install and verify the compatibility of Kuberarmor and Cilium on MicroK8's by applying policies on kubernetes workloads.

Step 1: Setup MicroK8's

Clone the Kubearmor Repo:

git clone https://github.com/kubearmor/KubeArmor.git

Alt

cd KubeArmor/contribution/microk8s

Run the script to set up MicroK8's Kubernetes:

./install_microk8s.sh
kubectl get all -A

Alt

Step 2: Setup KubeArmor

Install Karmor CLI:

curl -sfL https://raw.githubusercontent.com/kubearmor/kubearmor-client/main/install.sh | sudo sh -s -- -b /usr/local/bin
karmor install

Alt

Karmor verify:

kubectl get pods -n kube-system | grep kubearmor

Alt

Step 3: Create KubeArmor Policy

1. Create nginx deployment

kubectl create deployment nginx --image nginx
kubectl get pods --show-labels

Alt

2. Apply the following policy

vi ksp-block-untrusted-shell-execution.yaml
# KubeArmor is an open source software that enables you to protect your cloud workload at run-time.
# To learn more about KubeArmor visit: 
# https://www.accuknox.com/kubearmor/ 

apiVersion: security.kubearmor.com/v1
kind: KubeArmorPolicy
metadata:
  name: ksp-block-untrusted-shell-execution
  namespace: default # Change your namespace
spec:
  tags : ["MITRE","D3fend","Execution","Unix Shell"] 
  message: "Bash shells have been accessed"
  selector:
    matchLabels:
      app: nginx
  process:
    severity: 2 # Higher severity for processes 
    matchPaths:
    - path: /bin/bash
    - path: /bin/sh
    - path: /usr/bin/bash
    - path: /usr/bin/env
    - path: /usr/bin/shell
    - path: /bin/ksh
    - path: /etc/init.d      
    - path: /dev/tty 
    - path: /bin/zsh
    - path: /bin/tcsh
    - path: /bin/csh
    action: Block   
  file:
    severity: 10  # lowest severity for processes invoked as child process of bash
    matchPaths:
    - path: /bin/bash
    - path: /bin/sh
    - path: /usr/bin/bash
    - path: /usr/bin/env
    - path: /usr/bin/shell
    - path: /bin/ksh
    - path: /etc/init.d      
    - path: /dev/tty 
    - path: /bin/zsh
    - path: /bin/tcsh
    - path: /bin/csh
      fromSource:
      - path: /bin/bash
    action: Audit

3. Apply the policy

kubectl apply -f ksp-block-untrusted-shell-execution.yaml  

Note: Policy will work based on matched labels Ex:(app: nginx)

kubectl get pods

Alt

4. Violating the Policy

kubectl exec -it <Pod Name> -- bash 

5. run sh, env commands for policy violation

Alt

Note: Kubearmor is working, I can't run commands, which I have blocked in the policy..

Step 4: Getting Alerts/Telemetry from KubeArmor

1. Kubearmor SVC port forward to monitor the logs

kubectl port-forward -n kube-system svc/kubearmor --address 0.0.0.0 --address :: 32767:32767

Alt

2. Verifying policy violation logs

karmor log

Alt

Step 5: Cilium Install

Install Cilium CLI:

curl -L --remote-name-all https://github.com/cilium/cilium-cli/releases/latest/download/cilium-linux-amd64.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-amd64.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin
rm cilium-linux-amd64.tar.gz{,.sha256sum}
cilium install

Above tradition installation method is not working as expected, so installing using Microk8's command.

microk8s enable cilium

Alt

Alt

cilium status 

Alt

Cilium Hubble enable:

cilium hubble enable

Alt

Cilium Hubble verify:

kubectl get pods -n kube-system | grep hubble

Alt

Install the Hubble CLI Client:

export HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
curl -L --remote-name-all https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-amd64.tar.gz{,.sha256sum}
sha256sum --check hubble-linux-amd64.tar.gz.sha256sum
sudo tar xzvfC hubble-linux-amd64.tar.gz /usr/local/bin
rm hubble-linux-amd64.tar.gz{,.sha256sum}

Alt

Step 6: Cilium Policy

1. Create a mysql deployment and verify it

vi mysql.yaml
apiVersion: v1
kind: Service
metadata:
  name: accuknox-mysql-haproxy
spec:
  ports:
  - port: 3306
  selector:
    app: mysql
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: accuknox-mysql
spec:
  selector:
    matchLabels:
      app: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - image: mysql:8.0
        name: mysql
        resources:
          requests:
            memory: 100M
            cpu: 100m
      #        ephemeral-storage: 2G
          limits:
            memory: 1500M
            cpu: 1000m
#        ephemeral-storage: 2G
        env:
          # Use secret in real usage
        - name: MYSQL_ROOT_PASSWORD
          value: password
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysql-pv-claim

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: mysql-pv-volume
  labels:
    type: local
spec:
  storageClassName: standard
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/data"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pv-claim
spec:
  storageClassName: standard
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
kubectl apply -f mysql.yaml
kubectl get pods

Alt

kubectl get pods --show-labels

Alt

2. Apply Cilium policy

vi cnp-mitre-t1571-mysql-ingress.yaml
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: cnp-mitre-t1571-mysql-ingress
  namespace: default        #change default namespace to match your namespace
spec:
  description: "Allow ingress communication only through standard ports of MySQL pods"
  endpointSelector:
    matchLabels:
      app: mysql              # Change label with your own labels
  ingress:
  - toPorts:
    - ports:
      - port: "3306"
        protocol: TCP
      - port: "33060"
        protocol: TCP

3. Apply the policy

kubectl apply -f cnp-mitre-t1571-mysql-ingress.yaml   

Alt

4. Violating the policy

kubectl get pod

Alt

kubectl exec -it <mysql_pod>bash

Alt

5. Delete the policy

kubectl delete cnp rule1-ingress
Back to top