Germán M. Yébenes

, ,

Security supply chain through image verification and signature in Kubernetes

Introduction

Image signature verification is an important step if you want to deploy applications securely on Kubernetes clusters. It allows customers to ensure no unexpected or malicious changes affect the images along the entire CI/CD flow.

Images stored in a registry can be signed using several tools like the Notary Server and the Cosign/Sigstore, both sign digitally an specific tag or, as a best practice, push a signature of specific SHA ID

There are several tools to sign and verify images, some of them includes Notary Server and Cosign/Sigstore.

Notary Server

Notary is an Open Source project to sign, store and verify signatures securely with delegated servers for verification and authentication.

Notary is a project developed by Docker which provides identification and authentication for Docker Images. Notary was designed with the idea to ensure identification, integrity and authentication of the images.

Cosign/Sigstore

Cosign is designed to enable developers to create and push signatures keyless, using created keys or through the KMS system

Image signatures can be verified and integrated by several other tools in Kubernetes like Kyverno, Connaisseur or Rekor through the usage of admission controllers.

In conjunction with Sigstore, provides authentication with GitHub OAuth, Gmail and more, allowing identification and authentication to images

Deploying applications securely

Deploying applications securely challenges developers in a number of ways. First, developers need to be aware of the potential threats posed to their application and the data it stores. This includes both external threats, such as malicious actors trying to gain access to the application, and internal threats, such as employees inadvertently exposing sensitive data.

Kubernetes relies on external tools and sources to ensure the image did not suffer any unexpected or malicious changes during the promotion workflow but it does not have any native system to verify images or signatures.

Trusted images in Kubernetes

The implementation of these solutions will use Kyverno and Cosign to verify images preventing non-trusted images from being deployed in production.

Building the environment where only trusted images can be deployed in production will use this application

Building and signing images with Cosign

With Cosign images can be signed and this signature can be used to verify images before being deployed.

This Proof of Concept uses the Github repository kyverno_sigstore_verification to follow up the steps along this article moving along the concepts

Once the image is ready went through all the required security standards, images can be signed using Cosign:

$ cosign sign --key <KEYNAME>.key DOCKERUSER/test:signed

The image is now signed using the provided keys and can be pushed to the repository, signatures and tags can be looked at the repository 

The application image are there with both tags and the signatures with the format of sha256-<id>

Policy Management System – Kyverno

Kyverno is a Policy Management Tool using a Kubernetes Admission Controller to ensure and/or audit policy violation,It consists of a set of Custom Resource Definitions (CRDs) and an admission controller defining cluster policies, policies reports and cluster policies reports.

Helm can deploy Kyverno deployed as explained here the deploy should finish soon with the pods running on the namespace kyverno

NAME                                         READY   STATUS    RESTARTS   AGE
kyverno-696d66bc7b-4xx4q                     1/1     Running   0          20h
kyverno-696d66bc7b-c8css                     1/1     Running   0          20h
kyverno-696d66bc7b-xrr66                     1/1     Running   0          20h
kyverno-cleanup-controller-5dc5d7774-jwpjt   1/1     Running   0          20h

Deploy Kyverno policy from the repository

kubectl create -f cpolicy.yaml

Looking at the policy details:

[...]
spec:
  background: false
  failurePolicy: Fail
  rules:
  - match:
      any:
      - resources:
          kinds:
          - Pod
    name: verify-image
    verifyImages:
    - image: '*'
      key: |-
        -----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE/IjWWdh7Jd6p72Y/1D+TmVGjIp2cukXhWknErGhBIwnM5p9wc69S8+/bETXbgUfU7Y9N5grQo9x/u+qZDlHoA==
        -----END PUBLIC KEY-----
      verifyDigest: true
  validationFailureAction: Audit
  validationFailureActionOverrides:
  - action: Enforce
    namespaces:
    - prod
[...]

The above policy takes effect immediately, so let´s review.

Verify Kyverno and policies

New objects are validated against the Kyverno policies to pass, fail, warn, error or skip triggering a policy report.

Every time a new object is created it passes throughout the policies triggering a policy report.

Let’s see kyverno on action!

error: failed to create deployment: admission webhook "mutate.kyverno.svc-fail" denied the request: 
policy Deployment/prod/unsigned for resource violation: 
check-image-signature:
  autogen-verify-image: |
    failed to verify image docker.io/gmontalvoy/test:unsigned: .attestors[0].entries[0].keys: no matching signatures

The admission controller prevented the the unsigned image to be deployed on a production namespace but allows the trusted ones

❯ kc create deployment signed --image=gmontalvoy/test:signed -n prod
deployment.apps/signed created

The Kyverno Admission Controller intercepted the application based on the policy declared above.

Image by Sysdig article: https://sysdig.com/blog/image-scanning-admission-controller/

Audit events

What about the non-production namespaces? The kyverno policy includes an audit policy targeting other namespaces than ´prod´.When an unsigned image is deployed on a non-production namespace targeted by an audit-type policy (validationFailureAction: Audit) it will trigger a policyReport or polr kubernetes objects with the related violation even though the object was allowed to deploy.

❯ kc get polr -A
NAMESPACE   NAME                         PASS   FAIL   WARN   ERROR   SKIP   AGE
dev         cpol-check-image-signature   0      1      0      0       0      19m
prod        cpol-check-image-signature   1      0      0      0       0      19m

As shown above, there are two policy reports, one informing about the signed image was deployed on prod as specify by the policy:

results:
- message: image verified
  policy: check-image-signature
  resources:
  - apiVersion: v1
    kind: Pod
    name: signed-5b6c94c79b-jkwhh
    namespace: prod
    uid: 69c13f62-f8e0-4be7-8665-c6943fb31639
  result: pass
  rule: verify-image
  scored: true
  severity: medium
  source: kyverno

In the results section of the policy report, there is the validation performed on a prod namespace.

The results of the non-production reports produce a result showing the resource is violating the trusted image policy but it is just a warning not preventing the image to be deployed on a non-production namespace.

results:
- message: unverified image docker.io/gmontalvoy/test@sha256:6234a29e5fe3235109059262e86d4ffddcc5785ce5abbf73973aa25cc9fb3915
  policy: check-image-signature
  resources:
  - apiVersion: v1
    kind: Pod
    name: unsigned-85755b54c-flw7x
    namespace: dev
    uid: a8e4f91e-cbea-4fb5-9bb4-53e0cbd9f28c
  result: fail

Conclusion

As more and more institutions and customers are being targeted and attacked, the importance of security fully integrated within the CI/CD workflow is more important than ever before.

Applications in kubernetes needs to be compliant and secure and image verification through signatures, Cosign and Sigstore are projects into the Kubernetes framework ensuring compliance and security standards when it comes to deploy new applications.

While enforcing and auditing policies takes more importance, these events can be pushed to observability tools like Grafana, Prometheus and others providing observability layer to admission controller tools.

References

Leave a comment