Keyless Container Signing With Cosign 2.2, A Setup That Survives an Audit

TL;DR — Cosign 2.2 keyless signing replaces long-lived keys with short-lived OIDC certs from Fulcio, logged to Rekor / The trust root is your OIDC issuer plus the Sigstore TUF root, not a private key you have to rotate / Air-gapped clusters need a private Fulcio+Rekor or you cannot verify offline — plan for that before the audit lands.

Long-lived signing keys are a problem nobody wants to admit. They sit in HSMs or, more often, in environment variables called COSIGN_KEY. They get exfiltrated, rotated badly, or forgotten about until someone leaves the team. Sigstore’s keyless flow makes the key ephemeral: a workload identity (OIDC token) gets a code-signing certificate from Fulcio that is valid for ten minutes, the signature is logged to Rekor, and the certificate expires. There is nothing to rotate because there is nothing to steal.

I have rolled this out across three production clusters this year. The mechanics are clean. The trust story takes a beat to internalise, and there are two or three operational gotchas you only discover at scale. Cosign 2.2 (released August 2023) is the current line and what I am using here.

How the Trust Actually Works

When you run cosign sign --yes ghcr.io/me/app@sha256:... with no key flag, Cosign does this:

1. Gets an OIDC token from a configured provider (GitHub Actions, Google, the interactive browser flow, etc.).
2. Generates an ephemeral keypair locally.
3. Sends the public key plus OIDC token to Fulcio, which issues an X.509 cert binding the public key to the OIDC identity. Cert validity: 10 minutes.
4. Signs the image digest with the ephemeral private key.
5. Pushes the signature, cert, and bundle to the OCI registry (as a .sig tag).
6. Logs the signing event to Rekor, the transparency log. You get back a log index.
7. Discards the private key.

When you verify, you check: the signature is valid against the cert’s public key, the cert chains to Fulcio’s root, the OIDC identity in the cert matches what you expect, and the Rekor entry exists and proves the signature was created during the cert’s validity window. If any of those fail, verification fails.

The trust root is therefore the Sigstore TUF repository (which pins the Fulcio and Rekor public keys) plus your policy about which OIDC identities are allowed to sign what. There is no private key in your possession.

Signing From GitHub Actions

This is the most common case and the one with the least friction, because GitHub provides OIDC tokens natively.

name: build-sign-push
on:
  push:
    branches: [main]

jobs:
  publish:
    runs-on: ubuntu-22.04
    permissions:
      contents: read
      packages: write
      id-token: write
    steps:
      - uses: actions/checkout@v4

      - uses: sigstore/cosign-installer@v3.1.2
        with:
          cosign-release: v2.2.0

      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build and push
        id: build
        run: |
          IMAGE=ghcr.io/${{ github.repository }}:${{ github.sha }}
          docker build -t "$IMAGE" .
          docker push "$IMAGE"
          DIGEST=$(docker inspect --format='{{index .RepoDigests 0}}' "$IMAGE")
          echo "digest=$DIGEST" >> "$GITHUB_OUTPUT"

      - name: Sign
        env:
          COSIGN_EXPERIMENTAL: "1"
        run: cosign sign --yes "${{ steps.build.outputs.digest }}"

Two non-obvious points. First, id-token: write is the magic permission that lets the runner mint OIDC tokens — without it, keyless signing fails with an unhelpful error. Second, always sign by digest (@sha256:...), never by tag. Tags are mutable; signing a tag signs whatever it points to now, which is not what you want.

Verifying at Admission Time

Signing without verification is theatre. The verification has to happen at the gate to the cluster, not on a developer’s laptop. I use sigstore/policy-controller (the Sigstore project’s own admission controller). Connaisseur and Kyverno work too — pick one.

apiVersion: policy.sigstore.dev/v1beta1
kind: ClusterImagePolicy
metadata:
  name: require-ghcr-signed-by-ci
spec:
  images:
    - glob: "ghcr.io/myorg/**"
  authorities:
    - keyless:
        url: https://fulcio.sigstore.dev
        identities:
          - issuer: https://token.actions.githubusercontent.com
            subjectRegExp: "https://github.com/myorg/.+/.github/workflows/.+@refs/heads/main"
      ctlog:
        url: https://rekor.sigstore.dev

The subjectRegExp is the policy that matters. It says: I will only accept images signed by a GitHub Actions workflow in the myorg org running on the main branch. A signed image from a feature branch, a fork, or a developer laptop will be rejected. This is the bit people miss — keyless signing is only as good as the identity policy you enforce on the verify side.

SBOM Attestations Ride the Same Rail

Cosign does not just sign images. It signs attestations — typed claims about an image, including SBOMs, SLSA provenance, and vulnerability reports. They go to the same registry, get logged to the same Rekor, and are verified the same way.

syft ghcr.io/myorg/app@sha256:abc... -o cyclonedx-json > sbom.json

cosign attest --yes \
  --predicate sbom.json \
  --type cyclonedx \
  ghcr.io/myorg/app@sha256:abc...

At verify time:

cosign verify-attestation \
  --type cyclonedx \
  --certificate-identity-regexp "https://github.com/myorg/.+" \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  ghcr.io/myorg/app@sha256:abc... | jq '.payload | @base64d | fromjson'

This is the foundation of every supply-chain claim you might want to make later. SLSA provenance, vulnerability scan results, internal review approvals — all of it is “signed attestation of type X, attached to a digest”. I will go deeper on SBOMs in a follow-up post on Syft and CycloneDX.

Air-Gapped and Self-Hosted

The public Sigstore instance is fine for most teams. Regulated environments often cannot reach fulcio.sigstore.dev, and that is when you discover keyless signing has a different operational profile.

You have two options. Run your own Fulcio and Rekor (the sigstore/scaffolding Helm charts work, with caveats around the trust root) and configure your OIDC issuer to talk to them. Or fall back to long-lived keys, where Cosign supports cosign generate-key-pair --kms ... against AWS KMS, GCP KMS, Azure Key Vault, or HashiCorp Vault. The KMS-backed key never leaves the HSM and you get most of the benefit of keyless without the OIDC infrastructure.

I have seen teams try to bridge the gap by signing with public Sigstore and then mirroring images into an air-gapped registry. This works for the image but not for verification — policy-controller in the air-gapped cluster cannot reach Rekor to check the transparency proof. The workaround is offline bundle verification with cosign verify --offline ... and a bundled Rekor entry, which is functional but adds operational steps.

Common Pitfalls

Signing tags instead of digests. cosign sign ghcr.io/me/app:latest will sign whatever :latest currently points to. Tomorrow it points to something else and your signature is meaningless. Always resolve to digest first.

Permissive identity regex. subjectRegExp: ".*" accepts a signature from any GitHub identity, including a fork of your repo someone made yesterday. The regex needs to pin the org, repo pattern, and ideally the branch.

TUF root expiry. The Sigstore TUF root rotates periodically. Cosign caches it in ~/.sigstore/. CI runners with ephemeral filesystems re-fetch it every time, which is fine. Long-lived nodes with stale caches occasionally fail verification after a root rotation. cosign initialize re-pulls.

Rekor downtime. The public Rekor has had outages. During an outage, signing succeeds (cert from Fulcio is independent) but the Rekor entry write fails, and Cosign defaults to failing the sign command. Verification of previously-signed images keeps working because the Rekor entry already exists. You can sign with --rekor-url pointing at a private instance, or temporarily with --tlog-upload=false if your verifier policy allows unlogged signatures (mine does not).

Cosign 2.x breaking changes. Cosign 2.0 removed COSIGN_EXPERIMENTAL as the keyless flag (it became default), changed the bundle format, and tightened the verification flags. If you have a 1.x pipeline, the upgrade is not a drop-in. Read the Sigstore migration guide before bumping.

Wrapping Up

Keyless signing is the rare case where the security improvement and the operational improvement point in the same direction — no keys to manage, ephemeral identity, public transparency log, and an admission policy that ties everything to your CI provenance. The work is in the verification policy, not the signing. Get the identity regex right and the rest follows. Next on my list is making the SBOM that rides alongside these signatures actually accurate, which is harder than it looks.