Setup Instructions

Deploying Sample Cluster (skip if you already have a cluster configured)

Local K3s clusterGKE clusterAKS clusterEKS clusterKubeadm

Install K3s

Note: Recommended base OS image is Ubuntu 20.04.

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC='--disable traefik' sh -s - --write-kubeconfig-mode 644

Make K3's cluster config the default

mkdir -p ~/.kube && cp /etc/rancher/k3s/k3s.yaml ~/.kube/config

 - name: create a cluster
   hosts: localhost
   tasks:
    - name: create a cluster
      google.cloud.gcp_container_cluster:
       name: gkecluster
       initial_node_count: 1
       node_config:
         machine_type: e2-medium
         disk_size_gb: 10
         taints:
          - effect: PREFER_NO_SCHEDULE
            key: node.cilium.io/agent-not-ready
            value: "true"
       location: asia-east1
       project: "{{project_id}}"
       auth_kind: serviceaccount
       service_account_file: "{{service_account_file}}"
       state: present

sudo apt-get install python3 -y

sudo apt-get install ansible

ansible-galaxy collection install google.cloud

ansible-playbook kube-cluster.yaml

 - name: Create a managed Azure Container Services (AKS) instance
   hosts: localhost
   tasks:
    - name:
      azure_rm_aks:
       name: myAKS
       location: eastus
       resource_group: myResourceGroup
       dns_prefix: akstest
       kubernetes_version: 1.24.3
       linux_profile:
        admin_username: azureuser
        ssh_key: "{{local_ssh_key}}"
       service_principal:
        client_id: "{{azure_account_client_id}}"
        client_secret: "{{azure_account_client_secret}}"
       agent_pool_profiles:
        - name: default
          count: 2
          vm_size: Standard_D2_v2

sudo apt-get install python3 -y

sudo apt-get install ansible

ansible-galaxy collection install azure.azcollection

ansible-playbook aks-cluster.yaml

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: kubearmor-ub20
  region: us-east-2

nodeGroups:
  - name: ng-1
    amiFamily: "Ubuntu2004"
    privateNetworking: true
    desiredCapacity: 2
    # taint nodes so that application pods are
    # not scheduled until Cilium is deployed.
    taints:
     - key: "node.cilium.io/agent-not-ready"
       value: "true"
       effect: "NoSchedule"
    ssh:
      allow: true
    preBootstrapCommands:
      - "sudo apt install linux-headers-$(uname -r)"

eksctl create cluster -f sample-ubuntu-18.04-cluster.yaml

aws eks --region us-east-1 update-kubeconfig --name kubearmor-ub20

Install Pre-requisites

• VirtualBox

• Vagrant

• kubectl

• Helm

Download the Vagrant setup

• Click here to download

Untar and goto the Vagrant setup directory, Run the below command

vagrant up

Ref: KubeArmor support matrix

1. Install kubearmor cli tool, daemonsets and services

Install kubearmor cli tool

curl -sfL http://get.kubearmor.io/ | sudo sh -s -- -b /usr/local/bin

Install DaemonSets and Services

# Install KubeArmor
karmor install

# Install Discovery-Engine
kubectl apply -f https://raw.githubusercontent.com/kubearmor/discovery-engine/dev/deployments/k8s/deployment.yaml

Output from kubectl get pods -A

NAMESPACE         NAME                                                 READY   STATUS    RESTARTS   AGE
accuknox-agents   discovery-engine-7b6ddbd7d7-swk7j                    1/1     Running   0          3m58s
kube-system       kubearmor-78tnh                                      1/1     Running   0          4m7s
kube-system       kubearmor-annotation-manager-797c848b9c-vxq8c        2/2     Running   0          4m
kube-system       kubearmor-host-policy-manager-766447b4d7-fr5m4       2/2     Running   0          4m6s
kube-system       kubearmor-policy-manager-54ffc4dc56-8szmn            2/2     Running   0          4m6s
kube-system       kubearmor-relay-645667c695-bfwcn                     1/1     Running   0          4m7s
...

We have following installed:

• KubeArmor Protection Engine
• Discovery Engine
• KubeArmor Relay

2. Install Sample Application

Install the following app (WordPress) or you can try your own K8s app.

kubectl apply -f https://raw.githubusercontent.com/kubearmor/KubeArmor/main/examples/wordpress-mysql/wordpress-mysql-deployment.yaml

Output from kubectl get pods -n wordpress-mysql

NAME                        READY   STATUS    RESTARTS   AGE
mysql-58cdf6ccf-kzbp8       1/1     Running   0          12s
wordpress-bf95888cb-2kx65   1/1     Running   0          13s

Keep a note of these pods name mysql-xxxxxxxxx-xxxxx & wordpress-xxxxxxxxx-xxxxx, it'll be different for your environment

The use-cases described in subsequent step uses this sample application.

3. Demo Scenario & Use-cases

Use-case 1: Audit access to sensitive data paths

MySQL keeps all its database tables as part of /var/lib/mysql folder path. Audit access to this folder path recursively (sub-folders inclusive).

apiVersion: security.kubearmor.com/v1
kind: KubeArmorPolicy
metadata:
  name: ksp-mysql-audit-dir
  namespace: wordpress-mysql
spec:
  severity: 5
  selector:
    matchLabels:
      app: mysql
  file:
    matchDirectories:
      - dir: /var/lib/mysql/
        recursive: true
  action: Audit

Executing inside MySQL pod: Before applying policy

kubectl exec -it mysql-xxxxxxxxx-xxxxx -n wordpress-mysql -- bash
[email protected]:/# touch /var/lib/mysql/test

NOTE 01: Replace mysql-xxxxxxxxx-xxxxx with pod name from Step #2

Applying Policy: Applying above policy to deployed application

kubectl apply -f ksp-mysql-audit-dir.yaml

Port Forwarding: We'll be using KubeArmor relay to forward logs to our local system

kubectl -n kube-system port-forward service/kubearmor --address 0.0.0.0 --address :: 32767:32767

Realtime Logs Streaming:

karmor log

NOTE 02: Above 2 commands will be common for all use cases, keep this open in separate terminals (right section of screenshot)

Executing inside MySQL pod: After applying policy

kubectl exec -it mysql-xxxxxxxxx-xxxxx -n wordpress-mysql -- bash
[email protected]:/# touch /var/lib/mysql/test-2

Use-case 2: Block access to files containing sensitive data

WordPress pod contains a file wp-config.php that has sensitive auth credentials. This use-case is to Block access to this file from unknown processes.

Executing inside WordPress pod: Before applying policy

kubectl exec -it wordpress-xxxxxxxxx-xxxxx -n wordpress-mysql -- bash
[email protected]:/var/www/html# cat /var/www/html/wp-config.php

apiVersion: security.kubearmor.com/v1
kind: KubeArmorPolicy
metadata:
  name: ksp-wordpress-block-config
  namespace: wordpress-mysql
spec:
  severity: 10
  selector:
    matchLabels:
      app: wordpress
  file:
    matchPaths:
      - path: /var/www/html/wp-config.php
        fromSource:
          - path: /bin/cat
  action: Block

Applying Policy: Applying above policy to deployed application

kubectl apply -f ksp-wordpress-block-config.yaml

Executing inside WordPress pod: After applying policy

kubectl exec -it wordpress-xxxxxxxxx-xxxxx -n wordpress-mysql -- bash
[email protected]:/var/www/html# cat /var/www/html/wp-config.php
cat: /var/www/html/wp-config.php: Permission denied

Use-case 3: Block access to K8s service account token

A pod is the primary execution unit in K8s. One problem with this approach is that all the processes within that pod have unrestricted access to the pod's volume mounts. One such volume mount is a service account token. Thus, accessing a service account token using an injected binary is a common attack pattern in K8s. This use-case explains how you can protect access (Block) to the service account token through known processes only.

Executing inside WordPress pod: Before applying policy

kubectl exec -it wordpress-xxxxxxxxx-xxxxx -n wordpress-mysql -- bash
[email protected]:/var/www/html# cat /run/secrets/kubernetes.io/serviceaccount/token

apiVersion: security.kubearmor.com/v1
kind: KubeArmorPolicy
metadata:
  name: ksp-wordpress-block-sa
  namespace: wordpress-mysql
spec:
  severity: 7
  selector:
    matchLabels:
      app: wordpress
  file:
    matchDirectories:
      - dir: /run/secrets/kubernetes.io/serviceaccount/
        recursive: true
  action: Block

Applying Policy: Applying above policy to deployed application

kubectl apply -f ksp-wordpress-block-sa.yaml

Executing inside WordPress pod: After applying policy

kubectl exec -it wordpress-xxxxxxxxx-xxxxx -n wordpress-mysql -- bash
[email protected]:/var/www/html# cat /run/secrets/kubernetes.io/serviceaccount/token
cat: /run/secrets/kubernetes.io/serviceaccount/token: Permission denied

Use-case 4: Block execution of unwanted processes

A container image might get shipped with binaries that are not supposed to be executed in production environments. For e.g., WordPress contains apt, apt-get binaries that are used for dynamic package management. These should never be used in the production environment since it will create drift (change) in the container contents i.e., introduce new files/binaries that might increase the attack surface. The following policy Block the execution of such processes.

Executing inside WordPress pod: Before applying policy

kubectl exec -it wordpress-xxxxxxxxx-xxxxx -n wordpress-mysql -- bash
[email protected]:/var/www/html# apt
[email protected]:/var/www/html# apt-get update

apiVersion: security.kubearmor.com/v1
kind: KubeArmorPolicy
metadata:
  name: ksp-wordpress-block-process
  namespace: wordpress-mysql
spec:
  severity: 3
  selector:
    matchLabels:
      app: wordpress
  process:
    matchPaths:
      - path: /usr/bin/apt
      - path: /usr/bin/apt-get
  action: Block

Applying Policy: Applying above policy to deployed application

kubectl apply -f ksp-wordpress-block-process.yaml

Executing inside WordPress pod: After applying policy

kubectl exec -it wordpress-5679855487-58rfz -n wordpress-mysql –- bash
[email protected]:/var/www/html# apt
bash: /usr/bin/apt: Permission denied
[email protected]:/var/www/html# apt-get update
bash: /usr/bin/apt-get: Permission denied

4. Get Recommended Policies

In the above Demo Scenario, we had to explicitly write KubeArmor policies. But with the new KubeArmor recommendation it is easy to get a set of security best practice policies tailored to your environment.

karmor recommend --namespace wordpress-mysql --labels app=wordpress

INFO[0000] pulling image                                 image="wordpress:4.8-apache"
4.8-apache: Pulling from library/wordpress
Digest: sha256:6216f64ab88fc51d311e38c7f69ca3f9aaba621492b4f1fa93ddf63093768845
Status: Image is up to date for wordpress:4.8-apache
INFO[0015] dumped image to tar                           tar=/tmp/karmor4070582578/GwoIiuRV.tar
Distribution debian
INFO[0018] No runtime policy generated for wordpress-mysql/wordpress/wordpress:4.8-apache 
created policy out/wordpress-mysql-wordpress/wordpress-4-8-apache-maintenance-tool-access.yaml ...
created policy out/wordpress-mysql-wordpress/wordpress-4-8-apache-cert-access.yaml ...
created policy out/wordpress-mysql-wordpress/wordpress-4-8-apache-system-owner-discovery.yaml ...
created policy out/wordpress-mysql-wordpress/wordpress-4-8-apache-system-monitoring-deny-write-under-bin-directory.yaml ...
created policy out/wordpress-mysql-wordpress/wordpress-4-8-apache-system-monitoring-write-under-dev-directory.yaml ...
created policy out/wordpress-mysql-wordpress/wordpress-4-8-apache-least-functionality-execute-package-management-process-in-container.yaml ...
output report in out/report.txt ...

Using recommended policies

The recommended policy xx-xx-cert-access.yaml is a powerful policy which enables read access to trusted certificates but denied any form of write to it. This inturn enables the integrity of the file and denies malware/adware to have Adversary-in-the-Middle capability (Ref MITRE-T1553).

Executing inside WordPress pod: Before applying policy

kubectl exec -it wordpress-xxxxxxxxx-xxxxx -n wordpress-mysql -- bash
[email protected]:/var/www/html# cd /etc/ssl/
[email protected]:/etc/ssl# echo "new private key" > myssl.pem
[email protected]:/etc/ssl# cat myssl.pem 
new private key
[email protected]:/etc/ssl#

apiVersion: security.kubearmor.com/v1
kind: KubeArmorPolicy
metadata:
name: wordpress-wordpress-4-8-apache-cert-access
namespace: wordpress-mysql
spec:
action: Block
file:
    matchDirectories:
    - dir: /etc/ssl/
    readOnly: true
    recursive: true
    - dir: /etc/pki/
    readOnly: true
    recursive: true
    - dir: /usr/local/share/ca-certificates/
    readOnly: true
    recursive: true
message: Credentials modification denied
selector:
    matchLabels:
    app: wordpress
severity: 1
tags:
- MITRE
- MITRE_T1552_unsecured_credentials

Applying Policy: Applying above recommended policy to deployed application

kubectl apply -f out/wordpress-mysql-wordpress/wordpress-4-8-apache-cert-access.yaml

Executing inside WordPress pod: After applying policy

kubectl exec -it wordpress-cb9c668d4-zgczt -n wordpress-mysql –- bash
[email protected]:/var/www/html# cd /etc/ssl/
[email protected]:/var/www/html# cat myssl.pem 
new private key
[email protected]:/var/www/html# echo "updated ssl key" >> myssl.pem 
bash: /etc/ssl/myssl.pem: Permission denied
[email protected]:/var/www/html#

5. Get Auto-Discovered Policies

In the above Demo Scenario (last 3 use-cases), we had explicit Deny based policies. KubeArmor also supports Allow based policies i.e., allow only specific actions and audit/deny everything else. Also the allow-policies are auto-discovered by examining the workloads at runtime.

To retrieve the auto discovered policies you can use:

karmor discover -n wordpress-mysql -l "app=wordpress" -f yaml

This discovers the policies for a workload in wordpress-mysql namespace having label app=wordpress.

Output from karmor discover -n wordpress-mysql -l "app=wordpress" -f yaml

apiVersion: security.kubearmor.com/v1
kind: KubeArmorPolicy
metadata:
  name: autopol-system-3960684242
  namespace: wordpress-mysql
spec:
  action: Allow
  file:
    matchPaths:
    - fromSource:
      - path: /usr/sbin/apache2
      path: /dev/urandom
    - fromSource:
      - path: /usr/local/bin/php
      path: /etc/hosts
  network:
    matchProtocols:
    - fromSource:
      - path: /usr/local/bin/php
      protocol: tcp
    - fromSource:
      - path: /usr/local/bin/php
      protocol: udp
  process:
    matchPaths:
    - path: /usr/sbin/apache2
    - path: /usr/local/bin/php
  selector:
    matchLabels:
      app: wordpress
  severity: 1

6. Uninstall

karmor uninstall

# If there is no other workloads deployed in accuknox-agents namesapce apart from DE, this can be removed
kubectl delete ns accuknox-agents

References

1. AccuKnox Splunk App for Compliance and K8s Events
2. KubeArmor support matrix
3. Integrating Kubearmor with Prometheus and Grafana