gRPC for Internal Services in Go, A buf Powered Workflow

TL;DR — For internal service-to-service traffic in 2024, gRPC plus buf is the default in my book. Schema in version control, lint and breaking checks in CI, code generation that doesn’t fight you, and HTTP/2 multiplexing for free.

The case for gRPC over HTTP+JSON for internal services has been made enough times that I won’t re-litigate it. Smaller payloads, native streaming, bidirectional flows, generated clients in every language you care about. What I want to talk about is the workflow that makes it actually pleasant to live with in 2024.

The workflow centers on buf. If you are still using protoc directly with a Makefile full of plugin invocations, you are doing it the hard way. buf CLI 1.34 replaces all of that, plus it gives you linting, breaking change detection, and a module system for sharing proto across repos.

What follows is the setup I use on every new Go service that exposes gRPC. It takes maybe twenty minutes to bootstrap and it pays off on the first schema change.

Repository layout

I keep proto files in a top-level proto/ directory, separate from Go code. This makes it easier to share schemas with non-Go consumers later — TypeScript, Python, whatever — without anyone reaching into your Go module.

.
├── buf.yaml
├── buf.gen.yaml
├── proto
│   └── orders
│       └── v1
│           └── orders.proto
├── gen
│   └── go
│       └── orders
│           └── v1
│               ├── orders.pb.go
│               └── orders_grpc.pb.go
├── internal
│   └── ordersvc
│       └── server.go
└── cmd
    └── ordersvc
        └── main.go

The gen/ directory is checked in. I know there’s a school of thought that says generated code shouldn’t be committed. I disagree for proto: it makes go install work, it makes IDE jump-to-definition work, and it makes code review of schema changes show you exactly what the wire format changed to.

buf.yaml and buf.gen.yaml

# buf.yaml
version: v2
modules:
  - path: proto
lint:
  use:
    - STANDARD
  except:
    - PACKAGE_VERSION_SUFFIX
breaking:
  use:
    - FILE

# buf.gen.yaml
version: v2
managed:
  enabled: true
  override:
    - file_option: go_package_prefix
      value: github.com/yourorg/orders/gen/go
plugins:
  - remote: buf.build/protocolbuffers/go:v1.34.2
    out: gen/go
    opt: paths=source_relative
  - remote: buf.build/grpc/go:v1.4.0
    out: gen/go
    opt:
      - paths=source_relative
      - require_unimplemented_servers=true

require_unimplemented_servers=true is non-negotiable. It forces every server implementation to embed UnimplementedOrdersServiceServer, which means adding a new RPC to the proto doesn’t break existing servers at compile time. They return Unimplemented until you fill them in. This is the difference between a smooth schema rollout and a Friday afternoon panic.

The managed block lets buf set Go package paths consistently without polluting your proto files with option go_package = "..." lines. Worth using.

A proto file that ages well

// proto/orders/v1/orders.proto
syntax = "proto3";

package orders.v1;

import "google/protobuf/timestamp.proto";

service OrdersService {
  rpc GetOrder(GetOrderRequest) returns (GetOrderResponse);
  rpc ListOrders(ListOrdersRequest) returns (ListOrdersResponse);
  rpc CreateOrder(CreateOrderRequest) returns (CreateOrderResponse);
}

message Order {
  string id = 1;
  string customer_id = 2;
  repeated LineItem items = 3;
  Money total = 4;
  OrderStatus status = 5;
  google.protobuf.Timestamp created_at = 6;
}

message LineItem {
  string sku = 1;
  int32 quantity = 2;
  Money unit_price = 3;
}

message Money {
  string currency_code = 1; // ISO 4217
  int64 units = 2;          // whole units
  int32 nanos = 3;          // fractional, 10^-9
}

enum OrderStatus {
  ORDER_STATUS_UNSPECIFIED = 0;
  ORDER_STATUS_PENDING = 1;
  ORDER_STATUS_PAID = 2;
  ORDER_STATUS_FULFILLED = 3;
  ORDER_STATUS_CANCELLED = 4;
}

message GetOrderRequest { string id = 1; }
message GetOrderResponse { Order order = 1; }

message ListOrdersRequest {
  string customer_id = 1;
  int32 page_size = 2;
  string page_token = 3;
}

message ListOrdersResponse {
  repeated Order orders = 1;
  string next_page_token = 2;
}

message CreateOrderRequest {
  string customer_id = 1;
  repeated LineItem items = 2;
  string idempotency_key = 3;
}

message CreateOrderResponse { Order order = 1; }

A few patterns worth lifting:

• Always wrap requests and responses in messages. Even if GetOrderRequest has one field today, the wrapper means you can add a fields_mask or read_consistency parameter later without a breaking change.
• Enums always have _UNSPECIFIED = 0. Zero is the default proto3 value. If you don’t reserve it for “unspecified”, you can’t distinguish “client didn’t set this” from “client set it to the first enum value”.
• Money as units + nanos. Following Google’s common types convention. Don’t put currency in a double.
• Idempotency key on writes. Every mutating RPC takes one. The server uses it for dedup. This is cheap to add now and impossible to retrofit cleanly.

CI for proto

This is the part that earns its keep. In every PR, run:

buf lint
buf breaking --against '.git#branch=main,subdir=.'
buf generate
git diff --exit-code gen/

buf lint catches style issues. buf breaking catches wire-incompatible changes — renaming a field, changing a type, removing an enum value. buf generate regenerates the code. The final git diff --exit-code ensures the committed generated code matches the proto, so reviewers always see schema and code together.

I run this as a single make target:

.PHONY: proto-check
proto-check:
	buf lint
	buf breaking --against '.git#branch=main'
	buf generate
	@git diff --exit-code gen/ || (echo "gen/ out of date, run 'buf generate'"; exit 1)

Server implementation in Go

The grpc-go boilerplate has gotten a lot leaner. Here’s a server I’d actually ship:

package main

import (
    "context"
    "log/slog"
    "net"
    "os"
    "os/signal"
    "syscall"
    "time"

    "google.golang.org/grpc"
    "google.golang.org/grpc/health"
    healthpb "google.golang.org/grpc/health/grpc_health_v1"
    "google.golang.org/grpc/keepalive"

    ordersv1 "github.com/yourorg/orders/gen/go/orders/v1"
    "github.com/yourorg/orders/internal/ordersvc"
)

func main() {
    log := slog.New(slog.NewJSONHandler(os.Stdout, nil))
    lis, err := net.Listen("tcp", ":9090")
    if err != nil {
        log.Error("listen", "err", err)
        os.Exit(1)
    }

    s := grpc.NewServer(
        grpc.KeepaliveParams(keepalive.ServerParameters{
            MaxConnectionIdle: 5 * time.Minute,
            Time:              30 * time.Second,
            Timeout:           5 * time.Second,
        }),
        grpc.MaxRecvMsgSize(4*1024*1024),
    )

    ordersv1.RegisterOrdersServiceServer(s, ordersvc.New(log))

    hs := health.NewServer()
    hs.SetServingStatus("", healthpb.HealthCheckResponse_SERVING)
    healthpb.RegisterHealthServer(s, hs)

    go func() {
        sig := make(chan os.Signal, 1)
        signal.Notify(sig, syscall.SIGINT, syscall.SIGTERM)
        <-sig
        log.Info("shutting down")
        ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
        defer cancel()
        done := make(chan struct{})
        go func() { s.GracefulStop(); close(done) }()
        select {
        case <-done:
        case <-ctx.Done():
            s.Stop()
        }
    }()

    log.Info("listening", "addr", lis.Addr().String())
    if err := s.Serve(lis); err != nil {
        log.Error("serve", "err", err)
    }
}

Health server, keepalives, graceful shutdown, message size limit. That’s the bare minimum I’d want in any service. Add interceptors for tracing, metrics, and auth on top — I do those in a separate internal/interceptors package so they’re testable in isolation.

When not to use gRPC

Despite all this enthusiasm, gRPC isn’t universal. Don’t reach for it when:

• The consumer is a browser. gRPC-Web works, but it’s a degraded experience compared to REST or GraphQL. If your client is a SPA, see GraphQL vs REST in 2024 for what to use instead.
• You need broad debuggability with curl. Yes, grpcurl exists. No, it is not as universal as curl. Operations teams who don’t write Go will be unhappy.
• You have public third-party API consumers. They want REST.

Common Pitfalls

• Skipping UnimplementedXxxServer embedding. Adding an RPC becomes a breaking change for every server in your fleet. Always require it.
• Putting business logic in the generated server method. The grpc method is a transport adapter. It should validate, call a domain service, and translate errors. Nothing else.
• Using google.protobuf.Any casually. It’s an escape hatch, not a data modeling tool. If you find yourself reaching for it, your schema is wrong.
• Forgetting deadlines. Every RPC client call should have a context deadline. If not, one slow downstream takes your whole service down via goroutine pileup.
• Mutually-incompatible TLS configs. If your client uses grpc.WithTransportCredentials and your server is plain HTTP/2, you get cryptic errors. Pick one stance per environment and document it.

Wrapping Up

gRPC in Go in 2024 is mature, fast, and finally has the tooling story it deserves. buf does the heavy lifting on schema discipline, grpc-go does the heavy lifting on wire performance, and the result is a setup that scales from a single service to a hundred without changing the workflow.

The thing I’d encourage anyone adopting this to do first: get buf breaking running in CI before you have more than one consumer of your service. Once you have ten clients depending on a proto, every change becomes a coordination problem. Catching breaking changes in PR review is the lowest-effort win in the whole stack, and you only get to install it once cleanly.