Skip to content

How to ensure enforcing policies will not break my application?

Introduction

When we using KubeArmor and Cilium policies to allow the functionality of particular functions in deployment. These policies should not obstruct the deployment process and our application should work as intended without any disturbance with applied policies.

AccuKnox provides fine-grained access control for workloads at runtime allowing DevSecOps to control what resources, files, networks, and processes a workload can access. With AccuKnox the DevSecOps can create runtime policies to make sure always verify and never trust that's how zero trust model can be created. AccuKnox allows SecOps to restrict the following types of behavior on the cloud workloads. File access - A typical file access by a process or a network can be allowed or denied on specific paths - Process execution - The ability to allow or deny a Process execution or forking can be achieved for specific processes or all the processes on a directory - Network connection - It is easy to allow or deny any network-based communication from a workload using AccuKnox. The requests can be TCP, UDP, or even ICMP packets can be denied or allowed. - Capabilities - A workload can share the capabilities of the host if and when allowed. Such capabilities can enable additional types of malicious behavior. With AccuKnox we can allow or deny workloads to request other capabilities with the host OS.

For example,An online book-store app made with MySQL as a database. Scenerio is we want to keep the logs from being exposed to the public and also allowing them to function. We need to develop regulations that merely block the exposure, not the entire functionality. We'll go over how to construct rules with suitable permissions and use cases in this document.

Sample Application Deployment

AccuKnox sample library has various deployments for your needs to deploy vulnerable applications and use cases and Proof-of-concept, etc., Check out here. AccuKnox maintains repository that contains various sample deployment that one can use to play with for various PoC scenarios.

In this use case we are going to deploy an Online Book Store app made with mysql deployment.

To create this mysql deployment into your cluster run this below command:

kubectl apply -f https://raw.githubusercontent.com/accuknox/samples/main/OnlineBookStore/mysql-deployment.yaml

Output:

service/mysql created
deployment.apps/mysql created
persistentvolume/mysql-pv-volume created
persistentvolumeclaim/mysql-pv-claim created

Deployment state:

NAME                         READY   STATUS    RESTARTS   AGE
pod/mysql-68579b78bb-rttvt   1/1     Running   0          24s

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
service/kubernetes   ClusterIP   10.44.0.1    <none>        443/TCP    3h3m
service/mysql        ClusterIP   None         <none>        3306/TCP   27s

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/mysql   1/1     1            1           26s

NAME                               DESIRED   CURRENT   READY   AGE
replicaset.apps/mysql-68579b78bb   1         1         1       26s

Policy-Templates

AccuKnox policy-templates includes ready-made policy templates for all of our needs, from NIST, PCI-DSS, and STIG compliances to all CVEs and everything in between. That is the repository we will use in our use case. Check out policy-template for further information.

Policy Creation

To apply this KubeArmor Policy run this below command:

[Note: matchLabels and namespace may vary check with your deployment]

  1. In this policy we are going to audit the config files for any unusual activity.
kubectl apply -f https://raw.githubusercontent.com/kubearmor/policy-templates/main/MySQL/system/ksp-restrict-access-mysql-server-config-files.yaml

Policy YAML:

# KubeArmor is an open source software that enables you to protect your cloud workload at run-time.
# To learn more about KubeArmor visit:
# https://www.accuknox.com/kubearmor/

apiVersion: security.kubearmor.com/v1
kind: KubeArmorPolicy
metadata:
  name: ksp-restrict-access-mysql-server-config
  namespace: default # Change your namespace
spec:
  tags: ["MYSQL", "config-files", "mysqldump"]
  message: "Alert! mysql configuration files has been accessed and/or mysqldump command is has been used."
  selector:
    matchLabels:
      app: mysql8 # Change your labels
  file:
    severity: 5
    matchPaths:
    - path: /etc/mysql/my.cnf
      ownerOnly: true
    matchDirectories:
    - dir: /etc/mysql/
      recursive: true
      ownerOnly: true
    - dir: /var/lib/mysql/
      readOnly: true
      recursive: true
    - dir: /var/log/mysql/
      recursive: true
    action: Audit
  process:
    severity: 10
    matchPaths:
    - path: /usr/bin/mysqldump
    action: Block

In our policy we have mentioned the:

- dir: /var/lib/mysql/
      readOnly: true
      recursive: true

readOnly: true this rule will give appropriate permissions to the deployed application. \ So we can stop the logs from exposing and also our policy won’t break the applications functionality.

  1. In this policy we are going to protect the mysqldump to get accessed and to apply this policy run this command.
kubectl apply -f https://raw.githubusercontent.com/kubearmor/policy-templates/main/mitre/system/ksp-block-mysql-dump-in-pods.yaml

Policy YAML:

# KubeArmor is an open source software that enables you to protect your cloud workload at run-time.
# To learn more about KubeArmor visit:
# https://www.accuknox.com/kubearmor/

apiVersion: security.kubearmor.com/v1
kind: KubeArmorPolicy
metadata:
  name: ksp-block-mysql-dump-in-pods
  namespace: default        # Change your namespace
spec:
  tags: ["mysql","system","K8s"]
  message: "Warning! MySQLdump is blocked"
  selector:
    matchLabels:
      app: testpod    #change with your own label
  process:
    matchPaths:
    - path: /usr/bin/mysqldump
    action: Block
    severity: 6
  1. In this policy we are protecting the configuration files from exposure. To apply the policy run this below command.
kubectl apply -f https://raw.githubusercontent.com/kubearmor/policy-templates/main/stigs/system/ksp-block-stigs-mysql-config-directory-access.yaml

Policy YAML:

# KubeArmor is an open source software that enables you to protect your cloud workload at run-time.
# To learn more about KubeArmor visit:
# https://www.accuknox.com/kubearmor/

apiVersion: security.kubearmor.com/v1
kind: KubeArmorPolicy
metadata:
  name: ksp-block-stigs-mysql-config-directory-access
  namespace: default # Change your namespace
spec:
  tags: ["STIGS", "MYSQL"]
  message: "Alert! Access to mysql conf files has been denied"
  selector:
    matchLabels:
      pod: mysql-1 # Change your match labels
  file:
    severity: 5
    matchDirectories:
    - dir: /etc/mysql/
      recursive: true
      ownerOnly: true
    - dir: /usr/lib/mysql/
      recursive: true
      ownerOnly: true
    - dir: /usr/share/mysql/
      recursive: true
      ownerOnly: true
    - dir: /var/lib/mysql/
      recursive: true
      ownerOnly: true
    - dir: /var/log/mysql/
      recursive: true
      ownerOnly: true
    - dir: /usr/local/mysql/
      recursive: true
      ownerOnly: true
    action: Block
  1. In this Cilium policy we are going to protect the unused ports from accessing the workloads. To apply the policy run this below command.
kubectl apply -f https://raw.githubusercontent.com/kubearmor/policy-templates/main/stigs/network/cnp-mysql-stig-v-235146.yaml

Policy YAML:

# https://www.stigviewer.com/stig/oracle_mysql_8.0/2021-12-10/finding/V-235146

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: cnp-mysql-stig-v-235146
  namespace: default        #change default namespace to match your namespace
spec:
  description: "Restrict access to unused ports"
  endpointSelector:
    matchLabels:
      app: mysql #change label app: mysql with your own labels.
  ingress:
    - fromEndpoints:
        - {}
      toPorts:
        - ports:
            - port: "3306"
        - ports:
            - port: "33060"
  1. In this cilium policy we are protecting from external access to our mysql workload. To apply the policy run the below command.
kubectl apply -f https://raw.githubusercontent.com/kubearmor/policy-templates/main/mitre/network/cnp-egress-deny-external-communication-from-mysql-pod.yaml

Policy YAML:

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: cnp-egress-deny-external-communication-from-mysql-pod
  namespace: default        #change default namespace to match your namespace
spec:
  description: "To block all external world communication from mysql pod and limit it to specific pods"
  endpointSelector:
    matchLabels:
      app: mysql      #change app: mysql to match your label
  egressDeny:
  - toEntities:
    - "world"
  egress:
  - toEndpoints:
    - matchLabels:
        app: frontend   #change app: frontend to match your label

Verification: After applying multiple policies from policy-template our deployment is still running and without any issues

❯ kubectl apply -f https://raw.githubusercontent.com/kubearmor/policy-templates/main/MySQL/system/ksp-restrict-access-mysql-server-config-files.yaml
kubearmorpolicy.security.kubearmor.com/ksp-restrict-access-mysql-server-config unchanged
❯ kubectl apply -f https://raw.githubusercontent.com/kubearmor/policy-templates/main/mitre/system/ksp-block-mysql-dump-in-pods.yaml
kubearmorpolicy.security.kubearmor.com/ksp-block-mysql-dump-in-pods unchanged
❯ kubectl apply -f https://raw.githubusercontent.com/kubearmor/policy-templates/main/stigs/system/ksp-block-stigs-mysql-config-directory-access.yaml
kubearmorpolicy.security.kubearmor.com/ksp-block-stigs-mysql-config-directory-access unchanged
❯ kubectl apply -f https://raw.githubusercontent.com/kubearmor/policy-templates/main/stigs/network/cnp-mysql-stig-v-235146.yaml
ciliumnetworkpolicy.cilium.io/cnp-mysql-stig-v-235146 unchanged
❯ kubectl apply -f https://raw.githubusercontent.com/kubearmor/policy-templates/main/mitre/network/cnp-egress-deny-external-communication-from-mysql-pod.yaml
ciliumnetworkpolicy.cilium.io/cnp-egress-deny-external-communication-from-mysql-pod unchanged

Deployment state after applied the policies:

❯ kubectl get all -A | grep sql
default       pod/wordpress-mysql-55556f7-nf5fm              1/1     Running    0            32m
explorer      pod/mysql-0                                    1/1     Running    0            17m
default       service/mysql                                          ClusterIP  None         <none>          3306/TCP       5h23m
default       service/wordpress-mysql                                ClusterIP  None         <none>          3306/TCP       32m
explorer      service/mysql                                          ClusterIP  10.52.4.111  <none>          3306/TCP       17m
explorer      service/mysql-headless                                 ClusterIP  None         <none>          3306/TCP       17m
default       deployment.apps/wordpress-mysql                 1/1     1         1             32m
default       replicaset.apps/wordpress-mysql-55556f7                 1         1             32m
default       replicaset.apps/wordpress-mysql-7fc5cb7ccc              0         0             32m
explorer      statefulset.apps/mysql                          1/1                             17m

Verifying logs using Cilium:

List all cilium pods

❯ kubectl -n kube-system get pods -l k8s-app=cilium
NAME           READY   STATUS    RESTARTS   AGE
cilium-6jbcf   1/1     Running   0          115m
cilium-pj86d   1/1     Running   0          115m
cilium-qp649   1/1     Running   0          115m

Run this command

kubectl -n kube-system exec -it cilium-6jbcf -- bash

[Note: Pods name may vary check with your deployment]

After entering into the pod, Run this below command:

❯ kubectl -n kube-system exec -it cilium-6jbcf -- bash
Defaulted container "cilium-agent" out of: cilium-agent, wait-for-node-init (init), ebpf-mount (init), clean-cilium-state (init)
[email protected]:/home/cilium# cilium monitor
Listening for events on 2 CPUs with 64x4096 of shared memory
Press Ctrl-C to quit
level=info msg="Initializing dissection cache..." subsys=monitor
-> endpoint 1 flow 0x0 identity remote-node->health state new ifindex 0 orig-ip 10.128.15.192: 10.128.15.192 -> 10.40.2.23 EchoRequest
-> stack flow 0x0 identity health->remote-node state reply ifindex 0 orig-ip 0.0.0.0: 10.40.2.23 -> 10.128.15.192 EchoReply
-> endpoint 1 flow 0x0 identity remote-node->health state established ifindex 0 orig-ip 10.128.15.192: 10.128.15.192:58714 -> 10.40.2.23:4240 tcp ACK
-> stack flow 0x1e62a16c identity health->remote-node state reply ifindex 0 orig-ip 0.0.0.0: 10.40.2.23:4240 -> 10.128.15.192:58714 tcp ACK
-> stack flow 0xd663c7fb identity health->remote-node state reply ifindex 0 orig-ip 0.0.0.0: 10.40.2.23:4240 -> 10.128.0.63:34796 tcp ACK
-> endpoint 1 flow 0x0 identity remote-node->health state established ifindex 0 orig-ip 10.128.0.63: 10.128.0.63:34796 -> 10.40.2.23:4240 tcp ACK
-> stack flow 0x44777c79 identity health->host state reply ifindex 0 orig-ip 0.0.0.0: 10.40.2.23:4240 -> 10.128.0.62:36604 tcp ACK
-> endpoint 1 flow 0xf6e1704e identity host->health state established ifindex 0 orig-ip 10.128.0.62: 10.128.0.62:36604 -> 10.40.2.23:4240 tcp ACK
-> stack flow 0x1e62a16c identity health->remote-node state reply ifindex 0 orig-ip 0.0.0.0: 10.40.2.23:4240 -> 10.128.15.192:58714 tcp ACK
-> endpoint 1 flow 0x0 identity remote-node->health state established ifindex 0 orig-ip 10.128.15.192: 10.128.15.192:58714 -> 10.40.2.23:4240 tcp ACK
-> endpoint 1 flow 0x0 identity remote-node->health state established ifindex 0 orig-ip 10.128.0.63: 10.128.0.63:34796 -> 10.40.2.23:4240 tcp ACK
-> stack flow 0xd663c7fb identity health->remote-node state reply ifindex 0 orig-ip 0.0.0.0: 10.40.2.23:4240 -> 10.128.0.63:34796 tcp ACK
-> endpoint 1 flow 0xf6e1704e identity host->health state established ifindex 0 orig-ip 10.128.0.62: 10.128.0.62:36604 -> 10.40.2.23:4240 tcp ACK
-> stack flow 0x44777c79 identity health->host state reply ifindex 0 orig-ip 0.0.0.0: 10.40.2.23:4240 -> 10.128.0.62:36604 tcp ACK
-> endpoint 1 flow 0x0 identity remote-node->health state established ifindex 0 orig-ip 10.128.15.192: 10.128.15.192:58714 -> 10.40.2.23:4240 tcp ACK
-> stack flow 0x1e62a16c identity health->remote-node state reply ifindex 0 orig-ip 0.0.0.0: 10.40.2.23:4240 -> 10.128.15.192:58714 tcp ACK
-> stack flow 0xd663c7fb identity health->remote-node state reply ifindex 0 orig-ip 0.0.0.0: 10.40.2.23:4240 -> 10.128.0.63:34796 tcp ACK
-> endpoint 1 flow 0x0 identity remote-node->health state established ifindex 0 orig-ip 10.128.0.63: 10.128.0.63:34796 -> 10.40.2.23:4240 tcp ACK
-> stack flow 0x44777c79 identity health->host state reply ifindex 0 orig-ip 0.0.0.0: 10.40.2.23:4240 -> 10.128.0.62:36604 tcp ACK
-> endpoint 1 flow 0xf6e1704e identity host->health state established ifindex 0 orig-ip 10.128.0.62: 10.128.0.62:36604 -> 10.40.2.23:4240 tcp ACK
>> IPCache entry upserted: {"cidr":"10.128.0.62/32","id":1,"old-id":1,"encrypt-key":0}
>> IPCache entry upserted: {"cidr":"10.40.2.1/32","id":1,"old-id":1,"encrypt-key":0}
>> IPCache entry upserted: {"cidr":"10.40.2.1/32","id":1,"old-id":1,"encrypt-key":0}
>> IPCache entry upserted: {"cidr":"10.40.2.1/32","id":1,"old-id":1,"encrypt-key":0}
>> IPCache entry upserted: {"cidr":"10.40.2.40/32","id":1,"old-id":1,"encrypt-key":0}

Conclusion

In this document, we tested and proven several KubeArmor and cilium rules in our OnlineBookStore (mysql) implementation. In this section, we introduced rules such as readOnly: true and ownerOnly: true to allow certain functionality while blocking unauthorized access and processing for KubeArmor policies. Cilium is a white-list model. As a result, we restrict access to our programme to specific ports and access points. We can safeguard workloads and operate workloads smoothly in our environment with these two accuknox open-source solutions. \

We used KubeArmor Policy-Templates and AccuKnox Sample Library to deploy applications and apply policies to our workload.

Back to top