Schema First API Development with buf, A Step by Step Tutorial

TL;DR — buf 1.47 makes schema-first development pleasant in a way raw protoc never did. Lint, breaking-change check, generate, publish, and consume — all from one CLI with sane defaults. The workflow below is the one I ship to every team that starts using protobuf seriously.

I’ve watched three teams adopt protobuf the hard way. They start with a shared repo full of .proto files, a Makefile that nobody understands invoking protoc with a string of plugin flags, and a CI step that breaks every time a contributor’s local toolchain drifts. Six months in, breaking changes ship undetected, codegen artifacts get checked in, and somebody is editing the generated files because that’s how they unblock themselves.

buf is the tool that turns all of this around. The CLI takes opinionated stances on layout, linting, and breaking-change detection. The Buf Schema Registry (BSR) is the npm-equivalent for protobuf. The remote codegen plugin marketplace removes the toolchain-drift problem entirely. None of this is new in 2025, but buf 1.47 polished enough rough edges that I now consider it the default.

This tutorial covers the workflow I use for a multi-team protobuf monorepo. If you’ve read the Connect-Go tutorial, the buf config there was the short version. This is the long version, including BSR, CI, and the few traps that catch people.

1. Project Layout

There are two viable layouts: per-service repos or a single proto monorepo. I default to monorepo for proto definitions, with services depending on it. This makes breaking-change detection trivial because you have one source of truth.

proto/
  buf.yaml
  buf.gen.yaml
  buf.lock
  acme/
    orders/
      v1/
        order_service.proto
        order.proto
    payments/
      v1/
        payment_service.proto
  buf.work.yaml      (only if you split into multiple modules)

buf.yaml declares the module:

version: v2
modules:
  - path: .
name: buf.build/acme/proto
deps:
  - buf.build/googleapis/googleapis
breaking:
  use:
    - FILE
lint:
  use:
    - DEFAULT
  except:
    - PACKAGE_VERSION_SUFFIX  # we use v1, v2 — disable if your convention differs
  service_suffix: Service

A few opinions to copy:

• One package per service. The package name encodes the version suffix.
• File names follow snake_case.proto. Service files are <thing>_service.proto.
• Common types (errors, pagination) go in a shared package, versioned independently.

2. Linting

The default lint set catches the worst protobuf style mistakes:

buf lint

Common findings to expect on a legacy codebase:

• FIELD_LOWER_SNAKE_CASE — field is customerID instead of customer_id
• RPC_REQUEST_RESPONSE_UNIQUE — two RPCs share a request type (breaks evolution)
• PACKAGE_DIRECTORY_MATCH — package doesn’t match the file path
• ENUM_ZERO_VALUE_SUFFIX — Status.ACTIVE instead of Status.STATUS_UNSPECIFIED

The ENUM_ZERO_VALUE_SUFFIX rule is the one teams resist most and the one they regret skipping. A zero-valued enum is the default-on-the-wire, and giving it a meaningful name means you can’t add a “no value set” sentinel later.

3. Breaking-Change Detection

This is buf’s killer feature. Run against a baseline:

buf breaking --against '.git#branch=main'

It detects:

• Field number reuse with a different type
• Removed fields that were not reserved
• Removed RPCs
• Required fields added (proto2)
• Wire-incompatible type changes

The --against target can be a git ref, a BSR module version, or a local directory. In CI, you compare PR against the target branch. In release, you compare the release tag against the previous release tag.

# .github/workflows/buf.yml
name: buf
on: [pull_request]
jobs:
  buf:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - uses: bufbuild/buf-setup-action@v1.45.0
        with: { version: "1.47.0" }
      - run: buf lint
      - run: buf breaking --against "https://github.com/${{ github.repository }}.git#branch=main"

4. Code Generation

buf.gen.yaml lives next to buf.yaml. The simplest version:

version: v2
plugins:
  - remote: buf.build/protocolbuffers/go:v1.34.2
    out: gen
    opt: paths=source_relative
  - remote: buf.build/grpc/go:v1.5.1
    out: gen
    opt:
      - paths=source_relative
      - require_unimplemented_servers=false

Run it:

buf generate

Remote plugins mean nobody needs protoc-gen-go installed. The plugin runs on Buf’s hosted infra. For air-gapped environments, you can run a local plugin instead:

- local: protoc-gen-go
  out: gen
  opt: paths=source_relative

4.1 Multi-Target Generation

A single module often needs Go, TypeScript, and Python clients:

version: v2
plugins:
  - remote: buf.build/protocolbuffers/go:v1.34.2
    out: gen/go
    opt: paths=source_relative
  - remote: buf.build/bufbuild/es:v2.2.0
    out: gen/ts
    opt: target=ts
  - remote: buf.build/protocolbuffers/python:v28.2
    out: gen/python

For Connect-Go specifically:

  - remote: buf.build/connectrpc/go:v1.18.1
    out: gen/go
    opt: paths=source_relative

I usually keep one buf.gen.yaml per target ecosystem (e.g., buf.gen.go.yaml, buf.gen.web.yaml) so each consumer runs only what it needs.

5. The Buf Schema Registry (BSR)

The BSR is where modules live after buf push. Think npm for protobuf. You push from CI on merge to main:

# .github/workflows/push.yml
name: buf-push
on:
  push:
    branches: [main]
jobs:
  push:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: bufbuild/buf-setup-action@v1.45.0
        with: { version: "1.47.0", buf_user: ${{ secrets.BUF_USER }}, buf_api_token: ${{ secrets.BUF_TOKEN }} }
      - uses: bufbuild/buf-push-action@v1
        with:
          buf_token: ${{ secrets.BUF_TOKEN }}

Consumers depend on the module by reference, not by copying .proto files around:

# consumer's buf.yaml
version: v2
deps:
  - buf.build/acme/proto:v1.4.0

buf dep update resolves and writes buf.lock. The semantics are similar to Go modules.

5.1 Remote Codegen Without Local Sources

The BSR offers SDKs: buf.build/gen/go/acme/proto/protocolbuffers/go is an importable Go module of pre-generated code. Your service just imports it:

import ordersv1 "buf.build/gen/go/acme/proto/protocolbuffers/go/acme/orders/v1"

This eliminates the need to run buf generate in your service repo at all. The trade-off is that you’re pinned to whatever generator versions buf chose; if you need a specific protoc-gen-go release, run it locally.

6. CI Pipeline End to End

Putting it together, a real proto repo’s CI looks like this:

PR opened
  └─> buf lint          (fails on style issues)
  └─> buf format -d     (fails if not formatted)
  └─> buf breaking      (fails on breaking changes vs main)
  └─> buf build         (fails on parse errors)

main merged
  └─> buf push          (publishes vX.Y.Z to BSR)
  └─> tag release       (semver from commit messages)
  └─> notify consumers  (open PRs in dependent repos to bump versions)

The “notify consumers” step is where teams diverge. Some use renovate to auto-PR version bumps. Some use a custom action that opens a PR in each dependent repo. Pick one, automate it, never bump manually.

7. Common Pitfalls

7.1 Editing Generated Files

Someone will. They’ll add a struct method, an import, or “fix” a generated comment. Then buf generate overwrites it and they’re confused. Set .gitignore patterns or, better, don’t check generated files in at all — generate on build.

7.2 No reserved After Field Removal

When you remove a field, reserved 7; (or the field name) prevents the number from being reused. Skip this and someone in six months will add a new field at number 7 with a different type, and old clients will explode in colorful ways.

7.3 Using the Same Request Type for Two RPCs

rpc GetOrder(IdRequest) returns (Order);
rpc DeleteOrder(IdRequest) returns (Empty);

This breaks the day you need to add an option to one of them. buf lint catches it with RPC_REQUEST_RESPONSE_UNIQUE. Don’t disable that rule.

7.4 Pinning Plugin Versions to “Latest”

remote: buf.build/protocolbuffers/go:latest looks convenient. It’s a time bomb. Pin to specific versions and bump deliberately. The same advice applies to local plugins.

7.5 Breaking-Change Checks Without a Baseline

buf breaking --against '' will silently pass. Always provide a real baseline (a branch, a tag, or a BSR version). The CI snippet above does this; copy it.

8. Troubleshooting

8.1 dependency cycle detected

You imported a file that imports back, transitively. buf build prints the cycle. Usually solved by extracting common types into a separate package.

8.2 BSR Push Fails With module already exists at this commit

You’re trying to push without changes. Either bump a file, or skip the push step. Don’t re-tag the same content.

8.3 Generated Code Diff in CI Despite No Schema Changes

Plugin versions drifted between local and CI. Pin both to the same versions. If you’re using remote plugins, the BSR generates deterministic output for a given input and plugin version, so this means the input changed.

9. Wrapping Up

Schema-first development with buf is the lowest-friction way to keep API contracts honest across teams. The lint and breaking-change checks pay rent every week. The BSR removes the toolchain-drift class of incidents. The remote codegen plugins remove the protoc-installation class of onboarding pain.

The buf docs are well-maintained and worth reading if you’re going beyond the basics in this post (especially the sections on managed mode and the buf curl command). With contracts under control, the next thing to tighten up is observability — specifically OpenTelemetry for gRPC services, which is up next.