Context, Deadlines, and Cancellation in gRPC Microservices

TL;DR — context.Context is the deadline-and-cancellation contract that ties a Go program together / gRPC propagates deadlines across the wire automatically; cancellation is automatic too / The hard part isn’t using context — it’s not subverting it.

Context is the boring part of Go that turns out to be load-bearing. In a single-process program it’s a nice convenience. Across a gRPC mesh, it’s the only thing keeping your services from holding onto work that no caller wants anymore. I’ve debugged enough cascading slowdowns to be religious about this.

The setup: a request hits service A, which calls B and C in parallel, B calls D. Each hop should have a budget. When the user closes the tab, every downstream call should unwind. When a downstream is slow, the upstream should fail fast instead of holding the call indefinitely. Context, used correctly, gives you all of this almost for free.

This post builds on patterns from the goroutine post and assumes you’ve seen gRPC-Go basics. I’ll focus on the propagation rules and the failure modes I see most.

Deadlines Are Wire-Level in gRPC

This is the part that surprises people coming from REST. When you call a gRPC method with a context that has a deadline, the deadline is encoded into the request and the server-side context inherits it.

ctx, cancel := context.WithTimeout(ctx, 200*time.Millisecond)
defer cancel()

resp, err := client.GetInvoice(ctx, &billingv1.GetInvoiceRequest{InvoiceId: id})

On the server, the handler receives a context whose deadline is the smaller of:

1. The caller’s deadline (sent as grpc-timeout header)
2. The server’s own configured deadline (if any)

The server-side context is canceled when the client disconnects. If the client times out, the server context is canceled. If the client cancels explicitly, the server context is canceled. This is automatic.

The implication: your handler should pass ctx to every downstream operation — database, gRPC, HTTP. Once you do, you’ve inherited the caller’s budget end to end.

func (s *invoiceServer) GetInvoice(
    ctx context.Context,
    req *billingv1.GetInvoiceRequest,
) (*billingv1.Invoice, error) {
    // ctx already carries the caller's deadline
    inv, err := s.repo.Find(ctx, req.GetInvoiceId())
    if err != nil {
        if errors.Is(err, context.DeadlineExceeded) {
            return nil, status.Error(codes.DeadlineExceeded, "repo lookup timed out")
        }
        return nil, status.Errorf(codes.Internal, "repo: %v", err)
    }
    return toProto(inv), nil
}

Per-RPC Deadlines, Not Global Ones

A common mistake: setting a single timeout on the gRPC client connection and assuming it applies to every call. It doesn’t — connection-level options are about dialing and keepalive, not per-call deadlines. Each call needs its own context.

I push teams to set per-RPC deadlines explicitly, sized to the operation. A health check is 100 ms. A list query is 500 ms. A heavy report is 30 s. Don’t reuse a 30 s context for everything; you’ll never notice when a “fast” operation goes slow.

// at the call site
ctx, cancel := context.WithTimeout(parentCtx, 500*time.Millisecond)
defer cancel()
res, err := client.ListInvoices(ctx, req)

The defer cancel() is not optional. context.WithTimeout returns a cancel function whose contract is “you must call this.” If you don’t, the context’s resources (a timer, a goroutine reference) live until the deadline fires. In a hot path you’ll leak.

go vet catches the obvious cases; staticcheck catches more. Run both.

Budget Propagation Across Fan-Out

When service A calls B and C concurrently, both should run under A’s budget — but you might want to give each a slightly shorter slice so you have time to assemble the response and reply within A’s deadline.

func (s *Service) Compose(ctx context.Context, req *Req) (*Resp, error) {
    // Reserve 50ms for our own work
    deadline, ok := ctx.Deadline()
    var dctx context.Context
    var cancel context.CancelFunc
    if ok {
        dctx, cancel = context.WithDeadline(ctx, deadline.Add(-50*time.Millisecond))
    } else {
        dctx, cancel = context.WithTimeout(ctx, 2*time.Second)
    }
    defer cancel()

    g, gctx := errgroup.WithContext(dctx)
    var b *BResp
    var c *CResp
    g.Go(func() error {
        var err error
        b, err = s.bClient.GetB(gctx, req.ToB())
        return err
    })
    g.Go(func() error {
        var err error
        c, err = s.cClient.GetC(gctx, req.ToC())
        return err
    })
    if err := g.Wait(); err != nil {
        return nil, err
    }
    return assemble(b, c), nil
}

The “reserve some time for your own work” pattern is the difference between gracefully replying with what you have and forcing the upstream to time out on you. Don’t be the service that uses every millisecond of its budget on downstream calls and then can’t respond.

Cancellation Is Not Magic

<-ctx.Done() is the signal. Code that doesn’t check it doesn’t get canceled. Long-running loops, blocking syscalls, channel sends — all need explicit cancellation handling.

The patterns:

// Loop with cancellation check
for {
    select {
    case <-ctx.Done():
        return ctx.Err()
    default:
    }
    if err := step(ctx); err != nil {
        return err
    }
}

// Or, even better, a select on the work and the done channel
for {
    select {
    case <-ctx.Done():
        return ctx.Err()
    case item := <-input:
        if err := process(ctx, item); err != nil {
            return err
        }
    }
}

For blocking I/O, the standard library is mostly cooperative now. net.Conn operations honor context via the *WithContext variants. database/sql honors context on QueryContext, ExecContext. The HTTP client respects context on *http.Request. Anything you wrote yourself, you have to wire up.

The wrong pattern, which I still see:

// DON'T: time.Sleep ignores context
time.Sleep(5 * time.Second)

// DO:
select {
case <-time.After(5 * time.Second):
case <-ctx.Done():
    return ctx.Err()
}

Note: time.After leaks a timer until it fires. For long sleeps in tight loops, use time.NewTimer and stop it on cancel.

Storing Values in Context

Context can carry request-scoped values via WithValue. The pattern is widely abused.

Reasonable uses:

• Request ID for logging correlation
• Authenticated user identity
• Trace span (handled by OpenTelemetry SDK)

Unreasonable uses:

• Database handles
• Service dependencies
• Anything you could pass as a function argument

The rule I keep teaching juniors: if it’s in context, it’s because it’s tied to the request, not the program. The Go blog post on context covers the reasoning and remains the canonical reference.

For type safety, use unexported key types:

type ctxKey string

const userIDKey ctxKey = "user_id"

func WithUserID(ctx context.Context, id string) context.Context {
    return context.WithValue(ctx, userIDKey, id)
}

func UserIDFrom(ctx context.Context) (string, bool) {
    id, ok := ctx.Value(userIDKey).(string)
    return id, ok
}

Never use a string literal as a context key. Two packages using "user_id" will collide silently.

Common Pitfalls

The recurring offenders:

• context.Background() deep in business logic. If you find yourself calling Background() anywhere except main or top-level handlers, you’ve broken the chain. The whole point is that the caller’s context propagates.
• context.TODO() left in production. It’s meant as a placeholder during refactoring. Reviewing PRs with TODO() calls is a useful signal — usually the author meant to thread context through and didn’t.
• Forgetting defer cancel(). Leaks the context. go vet warns on the common cases.
• Reusing a context after cancel. Once a context is done, derived contexts are also done immediately. New calls fail before they start. Don’t carry a request-scoped context into background work; create a new one (often context.Background() with a fresh deadline) for fire-and-forget jobs.
• Trusting downstream to honor deadlines. Even with gRPC propagating headers, a downstream that ignores its own context will keep working. Your timeout protects you; it doesn’t protect them. Plan for downstream tail latency.
• Status code confusion. Return codes.DeadlineExceeded when the deadline fired, codes.Canceled when the client canceled. Both are visible via status.Code(err) on the receiving side. Don’t collapse them into codes.Internal.
• Logging the context. I’ve seen logs with ctx=&{0xc00...} from people who passed context to a log statement. Context is not for logging. Pull the values you want and log those.

Wrapping Up

If you take one thing from this: every blocking operation in a Go microservice should accept a context, every gRPC call should have a deadline sized to the operation, and every context with a cancel should have a defer cancel(). That’s it. The rest is consequences of those rules. The next post moves to interceptors — gRPC’s middleware mechanism, which is where authentication, logging, and observability all hook in.