Setup Instructions¶
Deploying Sample Cluster (skip if you already have a cluster configured)
Install K3s
Note: Recommended base OS image is Ubuntu 20.04.
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC='--disable traefik' sh -s - --write-kubeconfig-mode 644
Make K3's cluster config the default
mkdir -p ~/.kube && cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
- name: create a cluster
hosts: localhost
tasks:
- name: create a cluster
google.cloud.gcp_container_cluster:
name: gkecluster
initial_node_count: 1
node_config:
machine_type: e2-medium
disk_size_gb: 10
taints:
- effect: PREFER_NO_SCHEDULE
key: node.cilium.io/agent-not-ready
value: "true"
location: asia-east1
project: "{{project_id}}"
auth_kind: serviceaccount
service_account_file: "{{service_account_file}}"
state: present
sudo apt-get install python3 -y
sudo apt-get install ansible
ansible-galaxy collection install google.cloud
ansible-playbook kube-cluster.yaml
- name: Create a managed Azure Container Services (AKS) instance
hosts: localhost
tasks:
- name:
azure_rm_aks:
name: myAKS
location: eastus
resource_group: myResourceGroup
dns_prefix: akstest
kubernetes_version: 1.24.3
linux_profile:
admin_username: azureuser
ssh_key: "{{local_ssh_key}}"
service_principal:
client_id: "{{azure_account_client_id}}"
client_secret: "{{azure_account_client_secret}}"
agent_pool_profiles:
- name: default
count: 2
vm_size: Standard_D2_v2
sudo apt-get install python3 -y
sudo apt-get install ansible
ansible-galaxy collection install azure.azcollection
ansible-playbook aks-cluster.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: kubearmor-ub20
region: us-east-2
nodeGroups:
- name: ng-1
amiFamily: "Ubuntu2004"
privateNetworking: true
desiredCapacity: 2
# taint nodes so that application pods are
# not scheduled until Cilium is deployed.
taints:
- key: "node.cilium.io/agent-not-ready"
value: "true"
effect: "NoSchedule"
ssh:
allow: true
preBootstrapCommands:
- "sudo apt install linux-headers-$(uname -r)"
eksctl create cluster -f sample-ubuntu-18.04-cluster.yaml
aws eks --region us-east-1 update-kubeconfig --name kubearmor-ub20
Install Pre-requisites
Download the Vagrant setup
Untar and goto the Vagrant setup directory, Run the below command
vagrant up
1. Install kubearmor cli tool, daemonsets and services¶
Install kubearmor cli tool¶
curl -sfL http://get.kubearmor.io/ | sudo sh -s -- -b /usr/local/bin
Install DaemonSets and Services¶
karmor install
kubectl apply -f https://raw.githubusercontent.com/kubearmor/discovery-engine/dev/deployments/k8s/deployment.yaml -n explorer
Output from kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
explorer knoxautopolicy-7d77f658fb-m2g8q 1/1 Running 0 3m58s
kube-system kubearmor-78tnh 1/1 Running 0 4m7s
kube-system kubearmor-annotation-manager-797c848b9c-vxq8c 2/2 Running 0 4m
kube-system kubearmor-host-policy-manager-766447b4d7-fr5m4 2/2 Running 0 4m6s
kube-system kubearmor-policy-manager-54ffc4dc56-8szmn 2/2 Running 0 4m6s
kube-system kubearmor-relay-645667c695-bfwcn 1/1 Running 0 4m7s
...
We have following installed:
- KubeArmor Protection Engine
- Discovery Engine
- KubeArmor Relay
2. Install Sample Application¶
Install the following app (WordPress) or you can try your own K8s app.
kubectl apply -f https://raw.githubusercontent.com/kubearmor/KubeArmor/main/examples/wordpress-mysql/wordpress-mysql-deployment.yaml
Output from kubectl get pods -n wordpress-mysql
NAME READY STATUS RESTARTS AGE
mysql-58cdf6ccf-kzbp8 1/1 Running 0 12s
wordpress-bf95888cb-2kx65 1/1 Running 0 13s
Keep a note of these pods name mysql-xxxxxxxxx-xxxxx & wordpress-xxxxxxxxx-xxxxx, it'll be different for your environment
The use-cases described in subsequent step uses this sample application.
3. Demo Scenario & Use-cases¶
Use-case 1: Audit access to sensitive data paths
MySQL keeps all its database tables as part of /var/lib/mysql folder path. Audit access to this folder path recursively (sub-folders inclusive).
apiVersion: security.kubearmor.com/v1
kind: KubeArmorPolicy
metadata:
name: ksp-mysql-audit-dir
namespace: wordpress-mysql
spec:
severity: 5
selector:
matchLabels:
app: mysql
file:
matchDirectories:
- dir: /var/lib/mysql/
recursive: true
action: Audit
Executing inside MySQL pod: Before applying policy
kubectl exec -it mysql-xxxxxxxxx-xxxxx -n wordpress-mysql -- bash
[email protected]:/# touch /var/lib/mysql/test
NOTE 01: Replace
mysql-xxxxxxxxx-xxxxxwith pod name from Step #2
Applying Policy: Applying above policy to deployed application
kubectl apply -f ksp-mysql-audit-dir.yaml
Port Forwarding: We'll be using KubeArmor relay to forward logs to our local system
kubectl -n kube-system port-forward service/kubearmor --address 0.0.0.0 --address :: 32767:32767
Realtime Logs Streaming:
karmor log
NOTE 02: Above 2 commands will be common for all use cases, keep this open in separate terminals (right section of screenshot)
Executing inside MySQL pod: After applying policy
kubectl exec -it mysql-xxxxxxxxx-xxxxx -n wordpress-mysql -- bash
[email protected]cdf6ccf-kzbp8:/# touch /var/lib/mysql/test-2
Use-case 2: Block access to files containing sensitive data
WordPress pod contains a file wp-config.php that has sensitive auth credentials. This use-case is to Block access to this file from unknown processes.
Executing inside WordPress pod: Before applying policy
kubectl exec -it wordpress-xxxxxxxxx-xxxxx -n wordpress-mysql -- bash
[email protected]:/var/www/html# cat /var/www/html/wp-config.php
apiVersion: security.kubearmor.com/v1
kind: KubeArmorPolicy
metadata:
name: ksp-wordpress-block-config
namespace: wordpress-mysql
spec:
severity: 10
selector:
matchLabels:
app: wordpress
file:
matchPaths:
- path: /var/www/html/wp-config.php
fromSource:
- path: /bin/cat
action: Block
Applying Policy: Applying above policy to deployed application
kubectl apply -f ksp-wordpress-block-config.yaml
Executing inside WordPress pod: After applying policy
kubectl exec -it wordpress-xxxxxxxxx-xxxxx -n wordpress-mysql -- bash
[email protected]:/var/www/html# cat /var/www/html/wp-config.php
cat: /var/www/html/wp-config.php: Permission denied
Use-case 3: Block access to K8s service account token
A pod is the primary execution unit in K8s. One problem with this approach is that all the processes within that pod have unrestricted access to the pod's volume mounts. One such volume mount is a service account token. Thus, accessing a service account token using an injected binary is a common attack pattern in K8s. This use-case explains how you can protect access (Block) to the service account token through known processes only.
Executing inside WordPress pod: Before applying policy
kubectl exec -it wordpress-xxxxxxxxx-xxxxx -n wordpress-mysql -- bash
[email protected]rdpress-bf95888cb-2kx65:/var/www/html# cat /run/secrets/kubernetes.io/serviceaccount/token
apiVersion: security.kubearmor.com/v1
kind: KubeArmorPolicy
metadata:
name: ksp-wordpress-block-sa
namespace: wordpress-mysql
spec:
severity: 7
selector:
matchLabels:
app: wordpress
file:
matchDirectories:
- dir: /run/secrets/kubernetes.io/serviceaccount/
recursive: true
action: Block
Applying Policy: Applying above policy to deployed application
kubectl apply -f ksp-wordpress-block-sa.yaml
Executing inside WordPress pod: After applying policy
kubectl exec -it wordpress-xxxxxxxxx-xxxxx -n wordpress-mysql -- bash
[email protected]:/var/www/html# cat /run/secrets/kubernetes.io/serviceaccount/token
cat: /run/secrets/kubernetes.io/serviceaccount/token: Permission denied
Use-case 4: Block execution of unwanted processes
A container image might get shipped with binaries that are not supposed to be executed in production environments. For e.g., WordPress contains apt, apt-get binaries that are used for dynamic package management. These should never be used in the production environment since it will create drift (change) in the container contents i.e., introduce new files/binaries that might increase the attack surface. The following policy Block the execution of such processes.
Executing inside WordPress pod: Before applying policy
kubectl exec -it wordpress-xxxxxxxxx-xxxxx -n wordpress-mysql -- bash
[email protected]:/var/www/html# apt
[email protected]:/var/www/html# apt-get update
apiVersion: security.kubearmor.com/v1
kind: KubeArmorPolicy
metadata:
name: ksp-wordpress-block-process
namespace: wordpress-mysql
spec:
severity: 3
selector:
matchLabels:
app: wordpress
process:
matchPaths:
- path: /usr/bin/apt
- path: /usr/bin/apt-get
action: Block
Applying Policy: Applying above policy to deployed application
kubectl apply -f ksp-wordpress-block-process.yaml
Executing inside WordPress pod: After applying policy
kubectl exec -it wordpress-5679855487-58rfz -n wordpress-mysql –- bash
[email protected]:/var/www/html# apt
bash: /usr/bin/apt: Permission denied
[email protected]:/var/www/html# apt-get update
bash: /usr/bin/apt-get: Permission denied
4. Get Auto-Discovered Policies¶
In the above Demo Scenario (last 3 use-cases), we had explicit Deny based policies. KubeArmor also supports Allow based policies i.e., allow only specific actions and audit/deny everything else. Also the allow-policies are auto-discovered by examining the workloads at runtime.
To retrieve the auto discovered policies you can use:
karmor discover -n wordpres-mysql -l "app=wordpress" -f yaml
wordpress-mysql namespace having label app=wordpress.
Output from karmor discover -n wordpress-mysql -l "app=wordpress" -f yaml
apiVersion: security.kubearmor.com/v1
kind: KubeArmorPolicy
metadata:
name: autopol-system-3960684242
namespace: wordpress-mysql
spec:
action: Allow
file:
matchPaths:
- fromSource:
- path: /usr/sbin/apache2
path: /dev/urandom
- fromSource:
- path: /usr/local/bin/php
path: /etc/hosts
network:
matchProtocols:
- fromSource:
- path: /usr/local/bin/php
protocol: tcp
- fromSource:
- path: /usr/local/bin/php
protocol: udp
process:
matchPaths:
- path: /usr/sbin/apache2
- path: /usr/local/bin/php
selector:
matchLabels:
app: wordpress
severity: 1
5. Uninstall¶
karmor uninstall
kubectl delete ns explorer