Skip to content

Setup Instructions

Alt

Deploying Sample Cluster (skip if you already have a cluster configured)

Install K3s

Note: Recommended base OS image is Ubuntu 20.04.

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC='--disable traefik' sh -s - --write-kubeconfig-mode 644

Make K3's cluster config the default

mkdir -p ~/.kube && cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
 - name: create a cluster
   hosts: localhost
   tasks:
    - name: create a cluster
      google.cloud.gcp_container_cluster:
       name: gkecluster
       initial_node_count: 1
       node_config:
         machine_type: e2-medium
         disk_size_gb: 10
         taints:
          - effect: PREFER_NO_SCHEDULE
            key: node.cilium.io/agent-not-ready
            value: "true"
       location: asia-east1
       project: "{{project_id}}"
       auth_kind: serviceaccount
       service_account_file: "{{service_account_file}}"
       state: present
sudo apt-get install python3 -y

sudo apt-get install ansible
ansible-galaxy collection install google.cloud

ansible-playbook kube-cluster.yaml
 - name: Create a managed Azure Container Services (AKS) instance
   hosts: localhost
   tasks:
    - name:
      azure_rm_aks:
       name: myAKS
       location: eastus
       resource_group: myResourceGroup
       dns_prefix: akstest
       kubernetes_version: 1.24.3
       linux_profile:
        admin_username: azureuser
        ssh_key: "{{local_ssh_key}}"
       service_principal:
        client_id: "{{azure_account_client_id}}"
        client_secret: "{{azure_account_client_secret}}"
       agent_pool_profiles:
        - name: default
          count: 2
          vm_size: Standard_D2_v2
sudo apt-get install python3 -y
sudo apt-get install ansible
ansible-galaxy collection install azure.azcollection
ansible-playbook aks-cluster.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: kubearmor-ub20
  region: us-east-2

nodeGroups:
  - name: ng-1
    amiFamily: "Ubuntu2004"
    privateNetworking: true
    desiredCapacity: 2
    # taint nodes so that application pods are
    # not scheduled until Cilium is deployed.
    taints:
     - key: "node.cilium.io/agent-not-ready"
       value: "true"
       effect: "NoSchedule"
    ssh:
      allow: true
    preBootstrapCommands:
      - "sudo apt install linux-headers-$(uname -r)"
eksctl create cluster -f sample-ubuntu-18.04-cluster.yaml
aws eks --region us-east-1 update-kubeconfig --name kubearmor-ub20

Install Pre-requisites

Download the Vagrant setup

Untar and goto the Vagrant setup directory, Run the below command

vagrant up

Ref: KubeArmor support matrix

1. Install kubearmor cli tool, daemonsets and services

Install kubearmor cli tool

curl -sfL http://get.kubearmor.io/ | sudo sh -s -- -b /usr/local/bin

Install DaemonSets and Services

karmor install
kubectl apply -f https://raw.githubusercontent.com/kubearmor/discovery-engine/dev/deployments/k8s/deployment.yaml -n explorer
Output from kubectl get pods -A
NAMESPACE     NAME                                                 READY   STATUS    RESTARTS   AGE
explorer      knoxautopolicy-7d77f658fb-m2g8q                      1/1     Running   0          3m58s
kube-system   kubearmor-78tnh                                      1/1     Running   0          4m7s
kube-system   kubearmor-annotation-manager-797c848b9c-vxq8c        2/2     Running   0          4m
kube-system   kubearmor-host-policy-manager-766447b4d7-fr5m4       2/2     Running   0          4m6s
kube-system   kubearmor-policy-manager-54ffc4dc56-8szmn            2/2     Running   0          4m6s
kube-system   kubearmor-relay-645667c695-bfwcn                     1/1     Running   0          4m7s
...

We have following installed:

  • KubeArmor Protection Engine
  • Discovery Engine
  • KubeArmor Relay

2. Install Sample Application

Install the following app (WordPress) or you can try your own K8s app.

kubectl apply -f https://raw.githubusercontent.com/kubearmor/KubeArmor/main/examples/wordpress-mysql/wordpress-mysql-deployment.yaml
Output from kubectl get pods -n wordpress-mysql
NAME                        READY   STATUS    RESTARTS   AGE
mysql-58cdf6ccf-kzbp8       1/1     Running   0          12s
wordpress-bf95888cb-2kx65   1/1     Running   0          13s

Keep a note of these pods name mysql-xxxxxxxxx-xxxxx & wordpress-xxxxxxxxx-xxxxx, it'll be different for your environment

The use-cases described in subsequent step uses this sample application.

3. Demo Scenario & Use-cases

Alt

Use-case 1: Audit access to sensitive data paths

MySQL keeps all its database tables as part of /var/lib/mysql folder path. Audit access to this folder path recursively (sub-folders inclusive).

apiVersion: security.kubearmor.com/v1
kind: KubeArmorPolicy
metadata:
  name: ksp-mysql-audit-dir
  namespace: wordpress-mysql
spec:
  severity: 5
  selector:
    matchLabels:
      app: mysql
  file:
    matchDirectories:
      - dir: /var/lib/mysql/
        recursive: true
  action: Audit

Executing inside MySQL pod: Before applying policy

kubectl exec -it mysql-xxxxxxxxx-xxxxx -n wordpress-mysql -- bash
[email protected]:/# touch /var/lib/mysql/test

NOTE 01: Replace mysql-xxxxxxxxx-xxxxx with pod name from Step #2

Alt

Applying Policy: Applying above policy to deployed application

kubectl apply -f ksp-mysql-audit-dir.yaml

Port Forwarding: We'll be using KubeArmor relay to forward logs to our local system

kubectl -n kube-system port-forward service/kubearmor --address 0.0.0.0 --address :: 32767:32767

Realtime Logs Streaming:

karmor log

NOTE 02: Above 2 commands will be common for all use cases, keep this open in separate terminals (right section of screenshot)

Executing inside MySQL pod: After applying policy

kubectl exec -it mysql-xxxxxxxxx-xxxxx -n wordpress-mysql -- bash
[email protected]cdf6ccf-kzbp8:/# touch /var/lib/mysql/test-2
Alt

Use-case 2: Block access to files containing sensitive data

WordPress pod contains a file wp-config.php that has sensitive auth credentials. This use-case is to Block access to this file from unknown processes.

Executing inside WordPress pod: Before applying policy

kubectl exec -it wordpress-xxxxxxxxx-xxxxx -n wordpress-mysql -- bash
[email protected]:/var/www/html# cat /var/www/html/wp-config.php

Alt

apiVersion: security.kubearmor.com/v1
kind: KubeArmorPolicy
metadata:
  name: ksp-wordpress-block-config
  namespace: wordpress-mysql
spec:
  severity: 10
  selector:
    matchLabels:
      app: wordpress
  file:
    matchPaths:
      - path: /var/www/html/wp-config.php
        fromSource:
          - path: /bin/cat
  action: Block

Applying Policy: Applying above policy to deployed application

kubectl apply -f ksp-wordpress-block-config.yaml

Executing inside WordPress pod: After applying policy

kubectl exec -it wordpress-xxxxxxxxx-xxxxx -n wordpress-mysql -- bash
[email protected]:/var/www/html# cat /var/www/html/wp-config.php
cat: /var/www/html/wp-config.php: Permission denied

Alt

Use-case 3: Block access to K8s service account token

A pod is the primary execution unit in K8s. One problem with this approach is that all the processes within that pod have unrestricted access to the pod's volume mounts. One such volume mount is a service account token. Thus, accessing a service account token using an injected binary is a common attack pattern in K8s. This use-case explains how you can protect access (Block) to the service account token through known processes only.

Executing inside WordPress pod: Before applying policy

kubectl exec -it wordpress-xxxxxxxxx-xxxxx -n wordpress-mysql -- bash
[email protected]rdpress-bf95888cb-2kx65:/var/www/html# cat /run/secrets/kubernetes.io/serviceaccount/token

Alt

apiVersion: security.kubearmor.com/v1
kind: KubeArmorPolicy
metadata:
  name: ksp-wordpress-block-sa
  namespace: wordpress-mysql
spec:
  severity: 7
  selector:
    matchLabels:
      app: wordpress
  file:
    matchDirectories:
      - dir: /run/secrets/kubernetes.io/serviceaccount/
        recursive: true
  action: Block

Applying Policy: Applying above policy to deployed application

kubectl apply -f ksp-wordpress-block-sa.yaml

Executing inside WordPress pod: After applying policy

kubectl exec -it wordpress-xxxxxxxxx-xxxxx -n wordpress-mysql -- bash
[email protected]:/var/www/html# cat /run/secrets/kubernetes.io/serviceaccount/token
cat: /run/secrets/kubernetes.io/serviceaccount/token: Permission denied

Alt

Use-case 4: Block execution of unwanted processes

A container image might get shipped with binaries that are not supposed to be executed in production environments. For e.g., WordPress contains apt, apt-get binaries that are used for dynamic package management. These should never be used in the production environment since it will create drift (change) in the container contents i.e., introduce new files/binaries that might increase the attack surface. The following policy Block the execution of such processes.

Executing inside WordPress pod: Before applying policy

kubectl exec -it wordpress-xxxxxxxxx-xxxxx -n wordpress-mysql -- bash
[email protected]:/var/www/html# apt
[email protected]:/var/www/html# apt-get update

Alt

apiVersion: security.kubearmor.com/v1
kind: KubeArmorPolicy
metadata:
  name: ksp-wordpress-block-process
  namespace: wordpress-mysql
spec:
  severity: 3
  selector:
    matchLabels:
      app: wordpress
  process:
    matchPaths:
      - path: /usr/bin/apt
      - path: /usr/bin/apt-get
  action: Block

Applying Policy: Applying above policy to deployed application

kubectl apply -f ksp-wordpress-block-process.yaml

Executing inside WordPress pod: After applying policy

kubectl exec -it wordpress-5679855487-58rfz -n wordpress-mysql –- bash
[email protected]:/var/www/html# apt
bash: /usr/bin/apt: Permission denied
[email protected]:/var/www/html# apt-get update
bash: /usr/bin/apt-get: Permission denied

Alt

4. Get Auto-Discovered Policies

In the above Demo Scenario (last 3 use-cases), we had explicit Deny based policies. KubeArmor also supports Allow based policies i.e., allow only specific actions and audit/deny everything else. Also the allow-policies are auto-discovered by examining the workloads at runtime.

To retrieve the auto discovered policies you can use:

karmor discover -n wordpres-mysql -l "app=wordpress" -f yaml
This discovers the policies for a workload in wordpress-mysql namespace having label app=wordpress.

Output from karmor discover -n wordpress-mysql -l "app=wordpress" -f yaml
apiVersion: security.kubearmor.com/v1
kind: KubeArmorPolicy
metadata:
  name: autopol-system-3960684242
  namespace: wordpress-mysql
spec:
  action: Allow
  file:
    matchPaths:
    - fromSource:
      - path: /usr/sbin/apache2
      path: /dev/urandom
    - fromSource:
      - path: /usr/local/bin/php
      path: /etc/hosts
  network:
    matchProtocols:
    - fromSource:
      - path: /usr/local/bin/php
      protocol: tcp
    - fromSource:
      - path: /usr/local/bin/php
      protocol: udp
  process:
    matchPaths:
    - path: /usr/sbin/apache2
    - path: /usr/local/bin/php
  selector:
    matchLabels:
      app: wordpress
  severity: 1

5. Uninstall

karmor uninstall
kubectl delete ns explorer

References

  1. Elastic and Splunk Integration Guide
  2. KubeArmor support matrix
  3. Integrating Kubearmor with Prometheus and Grafana
Back to top