conndeck blog

Supply Chain Security: Sigstore, SLSA, and What's Actually Practical

In April, one of our engineers asked an innocent question in Slack: "How do we know the images in prod are the ones CI built?" The channel went quiet for a long time.

The honest answer was: we didn't. CI built an image, pushed it with a tag, and the cluster pulled whatever that tag pointed at. Nothing technically stopped someone -- or something -- with registry credentials from pushing a different image over the same tag, and nothing on the cluster side would have noticed. We had branch protection, code review, MFA everywhere, and then a gap at the exact seam where code becomes running software. That's supply chain security in one sentence: not the exotic nation-state stuff, just the boring question of whether the thing running is the thing you built.

Here's what we actually implemented, in the order that gave us the most assurance per week of work.

Quick answer: Sign every image in CI with cosign keyless signing -- OIDC identity, no keys to manage -- and verify signatures at admission with a Kyverno verifyImages policy or the Sigstore Policy Controller, pinning images to digests. Attach SBOMs with syft and cosign attest so the next zero-day is a query, not a scavenger hunt. Aim for SLSA level 3 -- hosted build platform with tamper-resistant provenance -- and treat level 4 as optional homework. Start every admission policy in Audit mode.

Sign with cosign, keyless, from CI

Cosign's original workflow was cosign generate-key-pair, which handed you a private key and, with it, a permanent key-management problem. Keyless signing removed that: your CI job authenticates to Fulcio (Sigstore's CA) with its OIDC token, gets a short-lived certificate bound to the workflow identity, signs the image, and the whole event lands in Rekor, the public transparency log. The key exists for minutes and is never stored anywhere.

In GitHub Actions it's a couple of lines after your push step:

- uses: sigstore/cosign-installer@v3
- name: Sign image
  run: cosign sign --yes registry.example.com/api@${{ steps.build.outputs.digest }}

Two details matter. First, sign the digest, not the tag -- a signature over api:v2.14.3 signs whatever that tag pointed at that second, while the digest is immutable. Second, the --yes just skips the confirmation prompt, because CI can't press Y. What you get is a signature stored as an artifact next to the image in your registry, verifiable by anyone who knows what identity to expect:

$ cosign verify registry.example.com/api@sha256:4d3c8f... \
    --certificate-identity "https://github.com/example-org/api/.github/workflows/release.yml@refs/heads/main" \
    --certificate-oidc-issuer "https://token.actions.githubusercontent.com"
Verification for registry.example.com/api@sha256:4d3c8f... --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The code-signing certificate was verified using trusted certificate authority certificates

That identity string is the whole trust model: "this image was built by this workflow, in this repo, on this branch." An attacker with registry push credentials can't forge it, because they can't mint a Fulcio certificate for your repo's workflow.

Verify at admission, or it didn't happen

Signatures nobody checks are decoration. The enforcement point is admission, and the common options are a Kyverno ClusterPolicy or the Sigstore Policy Controller. Kyverno, since you probably already run it:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-image-signatures
spec:
  validationFailureAction: Audit
  background: true
  webhookTimeoutSeconds: 30
  failurePolicy: Fail
  rules:
    - name: require-cosign-signature
      match:
        any:
          - resources:
              kinds: ["Pod"]
      mutate:
        imageRegistry:
          reference: "registry.example.com/*"
      verifyImages:
        - imageReferences:
            - "registry.example.com/*"
          mutateDigest: true
          verifyDigest: true
          required: true
          attestors:
            - entries:
                - keyless:
                    subject: "https://github.com/example-org/*"
                    issuer: "https://token.actions.githubusercontent.com"
                    rekor:
                      url: https://rekor.sigstore.dev

mutateDigest rewrites tags to digests at admission so the verified artifact is exactly what runs -- this closes the tag-swap hole even for images that were signed. failurePolicy: Fail means a webhook outage blocks deploys rather than silently allowing them, which is what you want for a security control and will also ruin one of your days when Kyverno has a bad morning; that's the trade, and it's the right one. And as always: Audit first, read the PolicyReports for two weeks, then Enforce. During the audit phase I tend to watch for blocked pods in a live cluster view (conndeck is what I use) because a policy that's wrong shows up as deploys that never produce pods, and you want to see that in seconds.

SBOMs: build them for the bad day

An SBOM -- a software bill of materials listing every package in an image -- feels like compliance homework right up until the next log4shell, when it becomes the only way to answer "are we affected?" before lunch. The toolchain is painless:

$ syft registry.example.com/api@sha256:4d3c8f... -o cyclonedx-json > sbom.json
$ cosign attest --yes --predicate sbom.json --type cyclonedx registry.example.com/api@sha256:4d3c8f...

Syft generates the SBOM, cosign attaches it to the image as a signed attestation in the registry. Later, when a CVE drops, you pull attestations across your estate and grep -- or feed them into a scanner like Grype, which can scan the SBOM directly. The whole pipeline is an afternoon of CI work. The query capability on the bad day is priceless, and I say that having done the bad day the manual way.

SLSA: pick level 3 and move on

SLSA is the framework that describes how tamper-resistant your build process is. The levels, roughly:

  • Level 1: the build produces provenance -- a record of how the artifact was built.
  • Level 2: the build runs on a hosted platform that signs the provenance.
  • Level 3: the build platform is hardened and isolated, so provenance can't be forged even by someone with repo write access.
  • Level 4: hermetic, reproducible builds with two-party review. Rebuilding the same source produces bit-identical output.

The practical reading: if you build on GitHub Actions or GitLab CI with the SLSA provenance generator (or cosign attestations like above), you're at or near level 3 for a few hours of work. Level 4 is where you discover that your Go build embeds timestamps and your Java build depends on which mirror answered first. Most organizations should bank level 3, verify provenance at admission the same way they verify signatures, and revisit level 4 when a regulator makes them.

Frequently asked questions

How do I sign container images with cosign?

The modern way is keyless signing: run cosign sign against your image digest from CI, and cosign uses an OIDC identity from your CI provider to get a short-lived certificate from Fulcio, recording everything in the Rekor transparency log. There is no private key to store, rotate, or lose, which is why keyless largely replaced the old cosign generate-key-pair workflow. Sign digests, not tags, because tags move and signatures don't follow them.

How do I enforce image signature verification in Kubernetes?

Use an admission policy, typically a Kyverno ClusterPolicy with a verifyImages rule or the Sigstore Policy Controller, that requires every image to carry a valid cosign signature from an identity you trust before the pod is admitted. Configure it to verify keyless identities by issuer and subject, pin images to digests so the verified artifact is exactly what runs, and roll the policy out in Audit mode before enforcing.

What is SLSA and which level should I aim for?

SLSA is a framework describing how hardened your build pipeline is against tampering, with levels from provenance existing at all up to hermetic, reproducible, two-party-reviewed builds. For most teams, level 3 is the sweet spot: builds run on a hosted, isolated platform that generates tamper-resistant provenance, which GitHub Actions with the SLSA generator or GitLab's equivalent gets you mostly for free. Level 4 requires reproducible builds and is a research project for most organizations.

Do I need SBOMs for my container images?

Yes, but generate them for the day you need to answer where is log4j in our estate, not because a framework says so. Syft generates an SBOM in seconds, cosign attest attaches it to the image in your registry, and that combination turns the next zero-day scramble from a week of guesswork into a query. An SBOM nobody can query is compliance wallpaper.

Is Sigstore production-ready for private images?

Yes, and it works fine for private registries since signatures and attestations are just registry artifacts alongside the image. The caveat is that keyless signing leans on Sigstore's public Fulcio and Rekor instances, which some organizations aren't comfortable with for internal identities; if that's you, use cosign with your own keys managed in a KMS, or run the private Sigstore infrastructure. Verification at admission works identically either way.

The rules I run with

Supply chain security isn't one big system, it's three small ones that compound: sign what you build, verify at the door, and keep an inventory for the bad day. Cosign keyless plus a Kyverno policy plus syft is genuinely a week of work for most platform teams.

Start with the signatures. The Slack question deserves a better answer than silence.