KubeArmor and Cilium on Ubuntu 20.04

Overview¶

This user journey guides you to install and verify the compatibility of Kuberarmor and Cilium on Ubuntu 20.04 with 5.13 Kernel Version by applying policies on kubernetes workloads.

Step 1: Install etcd in control plane VM¶

sudo su

apt update

apt-get install etcd

Once etcd is installed, configure the following values in /etc/default/etcd as shown below.

vim /etc/default/etcd

ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379
ETCD_ADVERTISE_CLIENT_URLS=http://0.0.0.0:2379

Restart & check the status of etcd:

service etcd restart

service etcd status

service etcd enable

Alt

Step 2: Installing BCC¶

apt install -y bison build-essential cmake flex git libedit-dev \
>   libllvm7 llvm-7-dev libclang-7-dev python zlib1g-dev libelf-dev libfl-dev python3-distutils

git clone --depth 1 --branch v0.24.0 https://github.com/iovisor/bcc.git

Alt

mkdir bcc/build; cd bcc/build

cmake ..

make

make install

cmake -DPYTHON_CMD=python3 ..

pushd src/python/ && make

make install

Alt

Step 3: Installing Kubearmor¶

Install pre-requisites:

apt install bpfcc-tools linux-headers-$(uname -r)

wget https://github.com/kubearmor/KubeArmor/releases/download/v0.2.1/kubearmor_0.2.1_linux-amd64.deb && dpkg -i kubearmor_0.2.1_linux-amd64.deb

Alt

If above error occurs, Run:

apt --fix-broken install

Alt

Start & check the status of Kubearmor:

sudo systemctl start kubearmor

sudo systemctl enable kubearmor

sudo systemctl status kubearmor

Alt

Step 4: Install KVM-Service in control plane¶

Pre-requisites: Download & Install Go

Visit Go website for latest version

wget https://go.dev/dl/go1.18.1.linux-amd64.tar.gz

Untar file:

rm -rf /usr/local/go && tar -C /usr/local -xzf go1.18.1.linux-amd64.tar.gz

Paste the below path in /etc/profile:

vim /etc/profile

export PATH=$PATH:/usr/local/go/bin

Run the following command:

source /etc/profile

Note: KVM-Service requires that all the managed VMs should be within the same network.

git clone https://github.com/kubearmor/kvm-service.git

cd kvm-service && git checkout non-k8s

cd src/service/ && make

./kvmservice --non-k8s 2> /dev/null

Alt

Note: Let it keep running & continue in new terminal.

Step 5: Install Karmor in control plane¶

curl -sfL https://raw.githubusercontent.com/kubearmor/kubearmor-client/main/install.sh | sudo sh -s -- -b /usr/local/bin

Alt

Step 6: Onboard VMs using Karmor¶

Let's use kvmpolicy1.yaml and onboard a VM.

cat kvmpolicy1.yaml

Alt

karmor vm add kvmpolicy1.yaml

Alt

karmor vm list

Alt

Step 7: Generate installation scripts for configured VM¶

karmor vm --kvms getscript -v testvm1

Alt

Step 8: Execute the installation script in VM¶

sudo su

apt update

Note: Docker needs to be installed before runing the script.

apt install docker.io

chmod 666 /var/run/docker.sock

Copy the generated installation scripts to appropriate VM:

scp -r testvm1.sh [[email protected]:/path]

chmod +x testvm1.sh

./testvm1.sh

docker ps

Alt

Step 9: Apply and Verify Kubearmor system policy¶

1. Apply the policy

cat khp-example-vmname.yaml

Alt

karmor vm --kvms policy add khp-example-vmname.yaml

Alt

Note: With the above mentioned policy enforced in the VM, if a user tries to access /proc/cpuinfo file, user will see permission denied error and karmor log will show the alert log for blocking the file access.

2. Violating the policy

cat /proc/cpuinfo

Alt

Verifying Policy Violation Logs:

curl -sfL https://raw.githubusercontent.com/kubearmor/kubearmor-client/main/install.sh | sudo sh -s -- -b /usr/local/bin

karmor log

Alt

Step 10: Apply and Verify Cilium network policy¶

1. Allow connectivity with the control plane ( and port 2379)

vim vm-allow-control-plane.yaml

Copy and post the following in vm-allow-control-plane.yaml and save it,

kind: CiliumNetworkPolicy
metadata:
  name: "vm-allow-control-plane"
spec:
  description: "Policy to allow traffic to kv-store"
  nodeSelector:
    matchLabels:
      name: vm1
  egress:
  - toCIDR:
    - 10.128.0.6/32
    toPorts:
    - ports:
      - port: "2379"
        protocol: TCP

karmor vm --kvms policy add vm-allow-control-plane.yaml

Alt

Note: With the above mentioned policy enforced in the VM, a user cannot access any port of the vm. SSH connection of port22 gets an error connection refused .

Alt

2. For SSH connectivity allow port 22 and 169.254.169.254 port 80

vim vm-allow-ssh.yaml

Copy and post the following in vm-allow-ssh.yaml and save it,

kind: CiliumNetworkPolicy
metadata:
  name: "vm-allow-ssh"
spec:
  description: "Policy to allow SSH"
  nodeSelector:
    matchLabels:
      name: vm1
  egress:
  - toPorts:
    - ports:
      - port: "22"
        protocol: TCP
  - toCIDR:
    - 169.254.169.254/32
    toPorts:
    - ports:
      - port: "80"
        protocol: TCP

karmor vm --kvms policy add vm-allow-ssh.yaml

Alt

3. This policy block the DNS access in VM

vim vm-dns-visibility.yaml

Copy and post the following in vm-dns-visibility.yaml and save it,

kind: CiliumNetworkPolicy
metadata:
  name: "vm-dns-visibility"
spec:
  description: "Policy to enable DNS visibility"
  nodeSelector:
    matchLabels:
      name: vm1
  egress:
  - toPorts:
    - ports:
      - port: "53"
        protocol: ANY
      rules:
        dns:
        - matchPattern: "*"

karmor vm --kvms policy add vm-dns-visibility.yaml

Alt

4. This policy allow access of “www.google.co.in” alone in VM

vim vm-allow-www-google-co-in.yaml

Copy and post the following in vm-allow-www-google-co-in.yaml and save it,

kind: CiliumNetworkPolicy
metadata:
  name: "vm-allow-www.google.co.in"
spec:
  description: "Policy to allow traffic to www.google.co.in"
  nodeSelector:
    matchLabels:
      name: vm1
  egress:
  - toFQDNs:
    - matchName: www.google.co.in
    toPorts:
    - ports:
      - port: "80"
        protocol: TCP
      - port: "443"
        protocol: TCP

karmor vm --kvms policy add vm-allow-www-google-co-in.yaml

Alt

Note: With the above mentioned policies enforced in the VM, a user can access only http://www.google.co.in in the vm. Any Other DNS gets an error connection timed out

Violating the Policy:

curl http://www.google.co.in/

Alt

curl https://go.dev/

Alt

Verifying Policy Violation Logs:

docker exec -it cilium hubble observe -f -t policy-verdict

Alt