Chapter 133-4 hours

gRPC & Inter-Service Communication

A first-principles walkthrough of the RPC framework that powers the internals of Google, Netflix, Uber and most modern microservice fleets, from the contract (Protocol Buffers) and the binary wire format , through the HTTP/2 transport and the four streaming shapes, to deadlines, interceptors, mTLS and production resilience. Written to explain not just what each piece does but why it exists and how it works underneath. Implementations in Go, Python, JavaScript, TypeScript and Java.

Part I / Foundations

What gRPC Actually Is

gRPC is a framework for calling a function that lives on another machine as if it were a local function in your own process. The name expands to gRPC Remote Procedure Call (a recursive acronym; the “g” has been backronymed to a different word in every release for fun). The phrase that matters is the middle one, Remote Procedure Call. You write user, err := client.GetUser(ctx, &pb.GetUserRequest{Id: 42}) and it looks like an ordinary method call, but under the hood the arguments are serialized, shipped across the network to a server, executed there, and the return value is shipped back, all hidden behind that one line.

Three things are true about gRPC and you need all three in your head at once. It is (1) a contract-first RPC framework: you describe your API in a .proto file and code is generated from it; (2) it serializes data with Protocol Buffers, a compact binary format, instead of text like JSON; and (3) it transports those bytes over HTTP/2, which gives it multiplexing and real bidirectional streaming. Everything else in this manual is a consequence of those three facts.

REST hands you a building with addressable rooms (resources) and a fixed set of things you may do to each room (GET, POST, …). gRPC hands you a remote control with labelled buttons (methods), you don’t think about rooms, you press GetUser or SendPayment and the work happens elsewhere. One models nouns; the other models verbs.

YOU WORK HEREYour service, generated stubs & serversclient.GetUser(ctx, req) / func (s) GetUser(ctx, req) / the methods you call & implementProtocol Buffers, the contract + binary serializationmessages -> compact bytes / sec 04-05HTTP/2, multiplexed, bidirectional framed transportstreams carry the bytes / enables all 4 call types / sec 06TCP + TLS, the reliable, encrypted pipe
Where gRPC sits

“Procedure call” is the whole idea

In ordinary code, calling a function is invisible plumbing: you pass arguments, the CPU jumps to the function, it returns a value. RPC asks a simple question, what if that function lived on a different computer? The dream of RPC (which dates back to the 1980s) is to make the network disappear, so a distributed system feels like one program. gRPC is the modern, production-grade realization of that dream: you never write socket code, never parse a response by hand, never build a URL. You call a typed method and handle a typed result or a typed error.

The one-sentence definition

gRPC = typed remote method calls, defined by a Protobuf contract, serialized as compact binary, carried over HTTP/2 streams: so a client in any language can call a server in any other language as if it were local.

Why gRPC Exists

gRPC isn’t a replacement for REST everywhere, it was built to solve specific pains that show up when many services (often in different languages) talk to each other constantly, at high volume, inside a system. Google built it (open-sourced in 2015, evolved from an internal system called Stubby) precisely because REST-over-JSON was too slow, too loose, and too limited for service-to-service traffic at their scale. Five concrete problems drove its design.

The problem with REST/JSONWhat gRPC does instead
JSON is bulky & slow. Text, with repeated field names on every object; parsing is CPU-heavy.Protobuf binary: field numbers not names, varint-packed. Often 3-10x smaller and far faster to (de)serialize.
No enforced contract. The shape of a JSON payload lives in docs (or someone’s head); drift causes runtime breakage.The .proto file is the contract. Client & server generate code from the same source, mismatches fail at compile time.
Weak streaming. Plain HTTP/1.1 REST is request->response; streaming needs bolt-ons (SSE, polling, raw WebSockets).Four call types including full bidirectional streaming, native to the framework over HTTP/2.
Polyglot friction. Every language hand-writes its own client & serialization, inconsistently.One .proto generates idiomatic clients/servers for Go, Python, Java, C++, Rust, and more.
HTTP/1.1 connection overhead. One request per connection (or head-of-line blocking), repeated handshakes.HTTP/2 multiplexes many concurrent calls over one long-lived connection.

The honest scope

Those strengths are aimed at internal, machine-to-machine communication, microservices, backend-to-backend, mobile-to-backend where you control both ends. gRPC is weaker than REST for public, browser-facing APIs (browsers can’t speak raw gRPC, see sec 19) and for human-debuggable, cache-friendly, broadly-compatible endpoints. The full comparison is sec 18; for now, hold the frame: gRPC is the internal nervous system; REST is the public front door.

The RPC Mental Model

Before any syntax, internalize the machinery that makes a remote call look local, because every gRPC concept later is just a named part of this picture. A local function call and a remote one differ in exactly one place: between “call” and “execute,” the arguments have to cross a network. RPC inserts two pieces of generated code, a stub on the client and a skeleton/handler on the server, to hide that crossing.

CLIENT PROCESSyour codeclient.GetUser(req)STUB (generated)serialize args -> bytessend over HTTP/2SERVER PROCESSreal method runsquery DB, compute…SKELETON (generated)deserialize bytes -> argsdispatch to handlernetwork, Protobuf over HTTP/2request bytes -><- response bytes travel the reverse path, deserialized back into a typed return value
The anatomy of a remote call

The leaky-abstraction warning

RPC tries to make the network invisible, but the network is never truly invisible, and pretending otherwise is the classic RPC trap. A local call can’t time out, get lost, or be reordered; a remote one can do all three. That’s why gRPC gives first-class tools for the things a local call never needed: deadlines (sec 13), status codes for partial failure (sec 14), retries (sec 17), and cancellation. Treat every remote call as “a local call that can fail in network-shaped ways,” and you’ll design resilient systems instead of brittle ones.

Don’t pretend the wire isn’t there

The single biggest mistake with RPC is writing remote calls as if they were free and infallible, no timeout, no error handling for UNAVAILABLE, no thought about latency in a loop. The abstraction is a convenience, not a guarantee. Always pass a context/deadline and always handle the error.

Part II / The Two Pillars

Protocol Buffers: the Contract

Protocol Buffers (“protobuf”) is two things wearing one name: an IDL (Interface Definition Language) for describing your messages and services, and a binary serialization format for encoding them on the wire. This section is the IDL half, the .proto file that is the single source of truth both sides generate code from. Get the contract right and the client and server literally cannot disagree about the shape of the data.

A .proto file defines messages (the data structures) and services (collections of RPC methods). Here is a complete, realistic example, a user service, annotated with every rule that matters:

syntax = "proto3";              // always declare the syntax; proto3 is current

package user.v1;                // namespace + a versioning convention (v1, v2...)

option go_package = "example.com/gen/userv1;userv1";  // where generated Go lands
option java_package = "com.example.gen.userv1";      // and where generated Java lands
option java_multiple_files = true;                   // one class per message, not one giant outer class

// A message is a typed record. Each FIELD has a type, a name, and a NUMBER.
message User {
  string id          = 1;       // the "= 1" is the FIELD NUMBER, not a value
  string email       = 2;
  string full_name   = 3;
  Role   role        = 4;       // a nested enum (declared below)
  repeated string tags = 5;     // "repeated" = a list/array of strings
  int64  created_at  = 6;       // unix seconds; proto3 has no native date
}

// An enum is a fixed set of named values. The first MUST be 0 (the default).
enum Role {
  ROLE_UNSPECIFIED = 0;         // 0 is the implicit default, reserve it
  ROLE_MEMBER      = 1;
  ROLE_ADMIN       = 2;
}

message GetUserRequest  { string id = 1; }
message GetUserResponse { User user = 1; }   // messages nest inside messages

message CreateUserRequest {
  string email     = 1;
  string full_name = 2;
  optional string phone = 3;    // "optional" tracks presence: set vs unset vs ""
}

// A SERVICE is a set of methods. Each takes one message and returns one message.
service UserService {
  rpc GetUser    (GetUserRequest)    returns (GetUserResponse);
  rpc CreateUser (CreateUserRequest) returns (User);
}

The rules that actually bite

  • Field numbers are the real identity, not names. On the wire, email is transmitted as field 2, never as the string “email” (this is why protobuf is compact, sec 05). The name is for your code; the number is the contract.
  • Never change or reuse a field number. This is the cardinal rule of schema evolution. Renaming a field is safe (names aren’t on the wire); changing its number or type silently corrupts data for any peer still using the old definition. To remove a field, mark it reserved so the number can never be accidentally reused.
  • Adding fields is backward-compatible. Give a new field a fresh number; old clients simply don’t see it, new servers treat it as unset when an old client omits it. This is how a gRPC API evolves without breaking deployed consumers, the same additive-change discipline you know from REST versioning.
  • proto3 has defaults, not nulls. An unset string is "", an unset int is 0, an unset bool is false. If you must distinguish “absent” from “zero,” use optional (which adds presence tracking), the same trap as Go’s zero values, solved the same way (sec 13 of the REST/handlers manuals echoes this).
ConstructMeansNotes
scalarint32 int64 uint32 sint32 fixed64 float double bool string bytesPick sint* for often-negative numbers; bytes for raw binary.
repeatedAn ordered list of the field’s typeThe protobuf equivalent of an array/slice.
enumA fixed value set; first entry must be 0Integrity + self-documentation, like a Postgres enum.
oneofAt most one of several fields is setA tagged union, e.g. a result that is either a value or an error.
map&lt;k,v>An associative arraySugar over a repeated key/value message.
optionalAdds explicit presence tracking to a scalarDistinguishes “unset” from the zero value.

The contract is the API

In REST the contract is prose in a Swagger doc that code may or may not match. In gRPC the .proto is executable truth: both sides generate from it, so a field you added or a type you changed is reflected in both clients and servers the moment they regenerate. The schema can’t silently drift from the implementation.

How Protobuf Encodes: the Wire Format

The reason gRPC is fast and small comes down to how protobuf turns a message into bytes. You don’t hand-write this, but understanding it explains every performance claim and every gotcha. The core trick: each field is written as a tiny tag (which encodes the field number and a wire type) followed by the value, and integers are packed using varints that use fewer bytes for smaller numbers. No field names, no quotes, no commas, no whitespace.

JSON, text, names repeated every time{“id”:“u42”,“role”:2,“created_at”:1717careful: every key, quote, brace and comma is bytes on the wire~52 BProtobuf, field numbers + varints, no names0A 03u 4 220 0230 …varint~20 Btag len value / tag valueSame information, well under half the size, and no text parsing to decode it.
JSON vs Protobuf, same data

The tag: field number + wire type in one byte (usually)

Each field on the wire begins with a tag computed as (field_number << 3) | wire_type. The low 3 bits are the wire type (how to read the bytes that follow: varint, 64-bit, length-delimited, 32-bit); the rest is the field number. So the decoder reads the tag, learns “this is field 2, and it’s length-delimited,” and knows exactly how to consume what comes next, even if it has never seen that field before (it can skip unknown fields, which is what makes forward-compatibility work).

tag byte = (field_number << 3) | wire_typefield numberupper bitswire type3 bits0 = VARINT, int32/64, bool, enum1 = I64, fixed64, double2 = LEN, string, bytes, messages5 = I32, fixed32, floatvarint: small numbers cost fewer bytes1 -> 1 byte  /  300 -> 2 bytes /  70000 -> 3 byteseach byte uses 7 bits for data + 1 “continue” bit, so values grow their encoding only as needed
Decoding one field

The cost of binary: it isn’t human-readable

You can’t curl a gRPC endpoint and eyeball the JSON. Debugging needs tools that understand the schema (grpcurl, reflection, sec 20). That opacity is the price of the speed and size; it’s the main reason public/debuggable APIs often stay on REST.

HTTP/2: the Transport Underneath

gRPC doesn’t invent its own transport, it rides on HTTP/2, and almost every gRPC superpower (concurrency, all four streaming shapes, low overhead) is really an HTTP/2 feature. The thing to understand is HTTP/2’s central idea: a single TCP connection is divided into many independent, interleaved streams, each carrying a sequence of binary frames. That’s what lets one connection carry hundreds of concurrent RPCs, and lets data flow in both directions at once.

HTTP/1.1, one call blocks the line (or needs many connections)call Acall B waitscall C waitsHTTP/2, one connection, 4 streams interleavedABCDframes from different calls share the wire, in any orderNo head-of-line blocking at the HTTP layer: a slow call never freezes the others on the connection.
Multiplexing

How a gRPC call maps onto HTTP/2

Concretely, an RPC is an HTTP/2 POST to a path shaped like /package.Service/Method (e.g. /user.v1.UserService/GetUser). gRPC metadata travels as HTTP/2 headers; the serialized protobuf travels in DATA frames as the body; and the final status (the gRPC status code, sec 14) arrives in HTTP/2 trailers after the body. Streaming simply means more than one message flows in one or both directions on that stream before it closes.

gRPC conceptHTTP/2 mechanism
One RPC callOne HTTP/2 stream (request + response)
Method being called:path header = /pkg.Service/Method
Metadata (auth tokens, trace IDs)HTTP/2 request & response headers
The request/response message(s)length-prefixed protobuf in DATA frames
Final status + messagegrpc-status / grpc-message trailers
Streamingmultiple DATA frames before the stream half-closes

Part III / The Four Call Types

Unary RPC

The simplest and most common shape: one request, one response: exactly like a normal function call. The client sends a single message, the server does its work and returns a single message. ~90% of real-world RPCs are unary. We’ll build the GetUser method from the user.proto in sec 04, in full, in all five languages.

ClientServer1 request1 response
Unary

// ===== SERVER =====
package main

import (
    "context"
    "log"
    "net"

    "google.golang.org/grpc"
    "google.golang.org/grpc/codes"
    "google.golang.org/grpc/status"
    userv1 "example.com/gen/userv1" // generated from user.proto
)

// Embed the generated UnimplementedUserServiceServer for forward-compat.
type server struct {
    userv1.UnimplementedUserServiceServer
}

// The method signature is generated FROM the proto: ctx, *Request -> *Response, error
func (s *server) GetUser(ctx context.Context, req *userv1.GetUserRequest) (*userv1.GetUserResponse, error) {
    if req.GetId() == "" {
        return nil, status.Error(codes.InvalidArgument, "id is required") // typed error, sec 14
    }
    // ...real work: query the DB by req.GetId()...
    u := &userv1.User{Id: req.GetId(), Email: "ada@example.com", FullName: "Ada", Role: userv1.Role_ROLE_ADMIN}
    return &userv1.GetUserResponse{User: u}, nil
}

func main() {
    lis, _ := net.Listen("tcp", ":50051")
    s := grpc.NewServer()
    userv1.RegisterUserServiceServer(s, &server{}) // wire the impl to the service
    log.Println("gRPC on :50051")
    s.Serve(lis)
}

// ===== CLIENT =====
func callGetUser() {
    // NewClient replaces the deprecated grpc.Dial; insecure creds for local dev only
    conn, _ := grpc.NewClient("localhost:50051", grpc.WithTransportCredentials(insecure.NewCredentials()))
    defer conn.Close()

    client := userv1.NewUserServiceClient(conn) // the generated STUB
    ctx, cancel := context.WithTimeout(context.Background(), time.Second) // always a deadline, sec 13
    defer cancel()

    resp, err := client.GetUser(ctx, &userv1.GetUserRequest{Id: "u42"}) // looks local, runs remote
    if err != nil {
        log.Fatalf("GetUser failed: %v", err) // err carries the gRPC status code
    }
    log.Printf("got user: %s", resp.GetUser().GetFullName())
}

Notice the symmetry across languages: a generated stub on the client, a generated servicer/server base class on the server, and a method whose exact signature came from the proto.

Server Streaming

One request, a stream of responses. The client asks once; the server sends back many messages over time, then closes the stream. Perfect for: returning a large result set in chunks, a live feed of events, progress updates on a long job, or paginating without repeated round trips. The proto marks the response as stream.

ClientServer1 requestmany responses, then close
Server streaming

// proto:  rpc ListUsers(ListUsersRequest) returns (stream User);

// ===== SERVER: receive one req, call stream.Send(...) repeatedly =====
func (s *server) ListUsers(req *userv1.ListUsersRequest, stream userv1.UserService_ListUsersServer) error {
    for _, u := range queryUsers(req.GetFilter()) { // imagine this yields a big result set
        if err := stream.Send(u); err != nil {       // push one message down the stream
            return err                                // client gone / cancelled
        }
    }
    return nil // returning nil closes the stream cleanly (sends OK trailer)
}

// ===== CLIENT: call once, then Recv() in a loop until io.EOF =====
func listUsers(client userv1.UserServiceClient) {
    ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
    defer cancel()
    stream, _ := client.ListUsers(ctx, &userv1.ListUsersRequest{Filter: "active"})
    for {
        u, err := stream.Recv()
        if err == io.EOF { break } // server closed the stream, we're done
        if err != nil { log.Fatal(err) }
        log.Printf("user: %s", u.GetFullName())
    }
}

Each language reaches for the streaming idiom it already has: Go an explicit stream.Send/stream.Recv pair, Python a generator (yield) on the server and an iterable on the client, Node a readable stream (or an async iterable), Java a StreamObserver whose onNext is called once per message. Five spellings, identical bytes on the wire.

Client Streaming

A stream of requests, one response. The client sends many messages, then the server replies once with a summary/result. Ideal for uploads, batch ingestion, or aggregating a series of readings into a single computed answer. The proto marks the request as stream.

ClientServermany requests, then close1 response
Client streaming

// proto:  rpc UploadEvents(stream Event) returns (UploadSummary);

// ===== SERVER: Recv() in a loop, then SendAndClose() once at the end =====
func (s *server) UploadEvents(stream userv1.UserService_UploadEventsServer) error {
    count := 0
    for {
        ev, err := stream.Recv()
        if err == io.EOF { // client finished sending, now reply once
            return stream.SendAndClose(&userv1.UploadSummary{Received: int32(count)})
        }
        if err != nil { return err }
        store(ev)
        count++
    }
}

// ===== CLIENT: Send() many, then CloseAndRecv() for the single reply =====
func uploadEvents(client userv1.UserServiceClient, events []*userv1.Event) {
    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()
    stream, _ := client.UploadEvents(ctx)
    for _, ev := range events {
        stream.Send(ev) // push each one up
    }
    summary, err := stream.CloseAndRecv() // close our side, await the summary
    if err != nil { log.Fatal(err) }
    log.Printf("server received %d events", summary.GetReceived())
}

The asymmetry is the point: SendAndClose/CloseAndRecv in Go, a request_iterator plus a normal return in Python, the server consumes the whole request stream before producing its one answer.

Bidirectional Streaming

A stream of requests and a stream of responses, simultaneously and independently. Both sides read and write on the same stream at the same time, in any order, this is what HTTP/2’s full-duplex framing (sec 06) buys you. It’s the shape behind chat, real-time collaboration, live telemetry with control messages, and interactive sessions. Both request and response are stream in the proto.

ClientServerinterleaved & independent, neither side waits its turn
Bidirectional streaming

// proto:  rpc Chat(stream ChatMessage) returns (stream ChatMessage);

// ===== SERVER: loop Recv() and Send() on the same stream =====
func (s *server) Chat(stream userv1.UserService_ChatServer) error {
    for {
        msg, err := stream.Recv()
        if err == io.EOF { return nil } // client closed its send side
        if err != nil { return err }
        // echo back (or broadcast to a room, etc.), can Send anytime, any number
        reply := &userv1.ChatMessage{User: "server", Text: "ack: " + msg.GetText()}
        if err := stream.Send(reply); err != nil { return err }
    }
}

// ===== CLIENT: typically Send in one goroutine, Recv in another =====
func chat(client userv1.UserServiceClient) {
    stream, _ := client.Chat(context.Background())
    go func() { // concurrent receiver
        for {
            in, err := stream.Recv()
            if err != nil { return }
            log.Printf("<< %s", in.GetText())
        }
    }()
    for _, t := range []string{"hi", "how are you", "bye"} {
        stream.Send(&userv1.ChatMessage{User: "ada", Text: t}) // send concurrently
    }
    stream.CloseSend() // signal we're done sending; receiver drains the rest
}

Streaming gotchas to respect

Streams are long-lived, so they consume a connection/goroutine for their lifetime, always set deadlines or idle limits, and handle the client vanishing mid-stream. There’s no automatic back-pressure knob beyond HTTP/2 flow control, so a fast producer can overwhelm a slow consumer if you don’t pace it. And a single stream is ordered but not a transaction, if it breaks halfway, you’ve delivered a prefix, so design messages to be resumable or idempotent.

Unary1 -> 1normal callServer stream1 -> manyfeeds, large resultsClient streammany -> 1uploads, batchBidirectionalmany <-> manychat, realtimeAll four ride the same HTTP/2 stream machinery; the proto’s two stream keywords decide the shape.rpc M(Req) returns (Res) | (stream Req) | returns (stream Res) | both
The four shapes at a glance

Part IV / The Machinery

The .proto -> Code Workflow

You never write the stubs and skeletons by hand, a compiler, protoc (or the buf toolchain), reads your .proto and emits idiomatic source for each target language via a language-specific plugin. This is the step that turns the contract into callable code, and it’s the reason a Go client and a Python server can interoperate flawlessly: both were generated from the same file.

user.protothe contractprotoc+ lang plugins(one per target language)Go: *.pb.gomessages + stub + server basePy: *_pb2*.pymessages + stub + servicerRegenerate after every proto change, the generated files are build artifacts, committed or generated in CI.
Code generation

// Install the two plugins once (they sit on your PATH):
//   go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
//   go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest

// Generate messages (--go_out) AND the service stubs/skeleton (--go-grpc_out):
//   protoc \
//     --go_out=. --go_opt=paths=source_relative \
//     --go-grpc_out=. --go-grpc_opt=paths=source_relative \
//     user.proto

// Produces:
//   user.pb.go, structs for each message (+ getters)
//   user_grpc.pb.go, UserServiceClient (stub) + UserServiceServer (to implement)

// Then in code you simply import the generated package:
//   import userv1 "example.com/gen/userv1"

Channels, Stubs & a Call End-to-End

Two client-side objects matter, and people conflate them. A channel (Go calls it a ClientConn) is the long-lived, reusable connection to a server, it manages the underlying HTTP/2 connection(s), reconnection, and load-balancing state. A stub is the cheap, generated object you create on top of a channel to actually call methods. The rule: create the channel once and share it; create stubs freely. Opening a channel per request destroys performance, you throw away the connection reuse that was the whole point of HTTP/2.

1 / you call stub.GetUser(ctx, req)2 / stub serializes req -> protobuf3 / client interceptors run4 / HTTP/2 stream -> :path, DATA5 / server reads frames6 / deserialize + interceptors7 / YOUR handler runs8 / serialize reply + statusThe reply retraces the path: serialized on the server, framed over the same stream, deserialized by the stub,and handed back to you as a typed value, or a typed error carrying a status code.channel = the reused connection / stub = the cheap per-call caller on top of it
A unary call, end to end

The performance rule

One channel per server, shared across your whole app, for its whole lifetime. Stubs are throwaway. If your latency is mysteriously bad, the first thing to check is whether you’re creating a channel (and thus a fresh HTTP/2 + TLS handshake) on every call.

Metadata, Deadlines & Cancellation

These are the tools that acknowledge the network is real (the sec 03 warning, made concrete). Metadata is gRPC’s key-value side-channel, the equivalent of HTTP headers, for things that aren’t the message itself: auth tokens, trace/request IDs, API versions. Deadlines put an absolute time bound on a call and, crucially, propagate across hops. Cancellation lets a caller (or a broken connection) abort in-flight work so servers don’t toil on results nobody wants.

Deadlines beat timeouts: and they propagate

A gRPC deadline is an absolute point in time, not a per-hop duration. When service A calls B with a 1-second deadline and B calls C, the remaining budget travels along, so C knows it has, say, 600ms left, not a fresh second. This prevents the classic cascade where each layer waits its own full timeout and total latency balloons. Always set a deadline on every call. A call with no deadline can hang forever, pinning resources.


// ===== CLIENT: attach a deadline + metadata =====
ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second) // absolute deadline
defer cancel() // cancel frees resources whether we time out or finish early

ctx = metadata.AppendToOutgoingContext(ctx,
    "authorization", "Bearer "+token,   // auth travels as metadata, sec 16
    "x-request-id", reqID)              // trace id for correlation, like sec 15 of the layers manual

resp, err := client.GetUser(ctx, &userv1.GetUserRequest{Id: "u42"})
if status.Code(err) == codes.DeadlineExceeded {
    log.Println("call timed out") // a network-shaped failure a local call never had
}

// ===== SERVER: read metadata, respect the inherited deadline =====
func (s *server) GetUser(ctx context.Context, req *userv1.GetUserRequest) (*userv1.GetUserResponse, error) {
    md, _ := metadata.FromIncomingContext(ctx)
    auth := md.Get("authorization") // verify token here (or in an interceptor, sec 15)

    // ctx already carries the client's remaining deadline + cancellation, // pass it straight to the DB driver so slow work is abandoned automatically.
    if ctx.Err() != nil { return nil, status.FromContextError(ctx.Err()).Err() }
    _ = auth
    return s.lookup(ctx, req.GetId())
}

Pass the context down, always

In Go, thread the incoming ctx into every downstream call (DB queries, outbound RPCs). That’s what makes deadlines and cancellation actually work, if the client hangs up, the cancellation ripples all the way down and frees everything. A handler that ignores ctx keeps grinding on abandoned work.

Status Codes & Error Handling

gRPC does not use HTTP status codes. It has its own fixed set of status codes: an enum of ~16 values, sent in the grpc-status trailer (sec 06). Every call ends with exactly one: OK for success, or one of the error codes with an optional message. Returning the right code is part of your contract, because clients (and retry policies, sec 17) switch on it.

CodeMeansClosest HTTP analogue
OKSuccess200
INVALID_ARGUMENTClient sent bad input (independent of system state)400
UNAUTHENTICATEDNo / invalid credentials401
PERMISSION_DENIEDAuthenticated but not allowed403
NOT_FOUNDThe requested entity doesn’t exist404
ALREADY_EXISTSCreate conflicts with an existing entity409
FAILED_PRECONDITIONSystem not in a state for the operation400/409
RESOURCE_EXHAUSTEDQuota / rate limit hit429
DEADLINE_EXCEEDEDCall ran past its deadline504
UNAVAILABLETransient, server down/overloaded; safe to retry503
INTERNALA real bug / invariant broken500
UNIMPLEMENTEDMethod not implemented on this server501

import (
    "google.golang.org/grpc/codes"
    "google.golang.org/grpc/status"
)

// SERVER: return a typed status, not a bare error string.
func (s *server) GetUser(ctx context.Context, req *userv1.GetUserRequest) (*userv1.GetUserResponse, error) {
    if req.GetId() == "" {
        return nil, status.Error(codes.InvalidArgument, "id is required")
    }
    u, found := s.db.Find(req.GetId())
    if !found {
        return nil, status.Errorf(codes.NotFound, "no user with id %q", req.GetId())
    }
    return &userv1.GetUserResponse{User: u}, nil
}

// CLIENT: inspect the code to decide what to do.
resp, err := client.GetUser(ctx, req)
if err != nil {
    st := status.Convert(err)        // pull the status out of the error
    switch st.Code() {
    case codes.NotFound:        // expected, show "not found" in UI
    case codes.Unavailable:     // transient, retry with backoff, sec 17
    default:                    log.Printf("unexpected: %v: %s", st.Code(), st.Message())
    }
}

Interceptors: the Middleware of gRPC

If you read the handlers/services/middleware chapter, this is the exact same idea with a different name. An interceptor is a function that wraps every RPC, running before and/or after your handler, the place to centralize cross-cutting concerns so you don’t repeat them in every method: authentication, logging, metrics, tracing, panic recovery, rate limiting. They come in two flavours: unary interceptors (wrap one-shot calls) and stream interceptors (wrap streaming calls), on both the client and the server side.

incoming RPC ->recovercatch panicslogging+ metrics, traceauthverify tokenyour handlerbusiness logicOrder matters (recover outermost, auth nearest the handler), the same ordering discipline as HTTP middleware.<- response unwinds back through each interceptor (logging records status, recover guards the return)
The interceptor chain

// A unary server interceptor: signature is fixed by the framework.
func authInterceptor(ctx context.Context, req any, info *grpc.UnaryServerInfo,
    handler grpc.UnaryHandler) (any, error) {

    md, _ := metadata.FromIncomingContext(ctx)
    tokens := md.Get("authorization")
    if len(tokens) == 0 || !valid(tokens[0]) {
        return nil, status.Error(codes.Unauthenticated, "missing or invalid token")
    }
    // attach the verified identity for the handler to read from ctx
    ctx = context.WithValue(ctx, userKey{}, parse(tokens[0]))
    return handler(ctx, req) // call the next link / the real handler
}

func loggingInterceptor(ctx context.Context, req any, info *grpc.UnaryServerInfo,
    handler grpc.UnaryHandler) (any, error) {
    start := time.Now()
    resp, err := handler(ctx, req)
    log.Printf("%s took %s -> %s", info.FullMethod, time.Since(start), status.Code(err))
    return resp, err
}

// Register the chain when building the server (outermost listed first):
s := grpc.NewServer(
    grpc.ChainUnaryInterceptor(loggingInterceptor, authInterceptor),
)

This is the gRPC home for everything the layers manual put in middleware: auth, logging, tracing, rate limiting, panic recovery, written once, applied to every method.

Part V / Production & Interop

Authentication & Security

gRPC security splits into two orthogonal questions: channel security (is the connection encrypted and is the peer who they claim to be? -> TLS / mTLS) and call credentials (who is the caller for this request? -> a token in metadata). You combine them: TLS protects the pipe, a per-call token identifies the user. The insecure credentials used in earlier examples are for local development only, never ship them.

LayerMechanismAnswers
TransportTLS: server presents a certencrypted? + is the server genuine?
TransportmTLS: both sides present certs+ is the client service genuine? (service-to-service identity)
Per-calltoken in metadata (JWT / OAuth bearer)which user is making this call? (authn/authz, verified in an interceptor sec 15)

import "google.golang.org/grpc/credentials"

// ===== SERVER over TLS =====
creds, _ := credentials.NewServerTLSFromFile("server.crt", "server.key")
s := grpc.NewServer(grpc.Creds(creds)) // every connection is now encrypted

// ===== CLIENT over TLS =====
tlsCreds := credentials.NewTLS(&tls.Config{RootCAs: pool}) // trust this CA
conn, _ := grpc.NewClient("api.example.com:443", grpc.WithTransportCredentials(tlsCreds))

// ===== Per-call token (combine with TLS) =====
// Implement credentials.PerRPCCredentials so a fresh token rides on every call:
type tokenCreds struct{ token string }
func (t tokenCreds) GetRequestMetadata(ctx context.Context, _ ...string) (map[string]string, error) {
    return map[string]string{"authorization": "Bearer " + t.token}, nil
}
func (t tokenCreds) RequireTransportSecurity() bool { return true } // refuse to send token in cleartext

conn, _ = grpc.NewClient("api.example.com:443",
    grpc.WithTransportCredentials(tlsCreds),
    grpc.WithPerRPCCredentials(tokenCreds{token}),
)

Never send tokens over plaintext

Bearer tokens are like passwords, whoever holds one can impersonate the caller (echoing the JWT-theft warning from the auth manual). Always require transport security before attaching credentials. mTLS is the standard for service-to-service identity inside a mesh; user-level authz still rides on a per-call token that an interceptor verifies.

Resilience & Load Balancing

Because every remote call can fail in network-shaped ways, production gRPC leans on a few resilience features, mostly configured, not hand-coded. The non-obvious one is load balancing: gRPC’s long-lived, multiplexed connection (sec 06) breaks naive load balancers, and understanding why is essential to deploying it.

L4 (TCP) balancer, WRONG for gRPCclientL4 LBper-connbackend 1 (hot)backend 2 idleall multiplexed calls land on one boxL7 / client-side balancing, RIGHTclientL7 / meshper-RPCbackend 1backend 2backend 3individual RPCs spread across backends
The load-balancing pitfall

Retries, keepalive, health

gRPC supports declarative retries via a service config (a JSON policy attached to the channel): which status codes are retryable (typically UNAVAILABLE), how many attempts, and exponential backoff, no retry loops in your code. Keepalive pings detect dead connections and keep idle ones alive through NATs/proxies. A standard health-checking service lets load balancers and Kubernetes probes ask “are you ready?” Together these are the resilience baseline.

{
  "methodConfig": [{
    "name": [{ "service": "user.v1.UserService" }],
    "retryPolicy": {
      "maxAttempts": 4,
      "initialBackoff": "0.1s",
      "maxBackoff": "2s",
      "backoffMultiplier": 2,
      "retryableStatusCodes": [ "UNAVAILABLE" ]
    }
  }]
}
// Attach to the channel (Go: grpc.WithDefaultServiceConfig(json);
// Python: grpc.insecure_channel(target, options=[("grpc.service_config", json)]))
// Only retry IDEMPOTENT methods automatically, retrying a non-idempotent
// "charge card" can double-charge (same lesson as POST idempotency keys in REST).

Retries need idempotency

Automatic retries are safe only for idempotent methods. Retrying a “create payment” on a timeout can charge twice, the exact danger the REST manual solved with idempotency keys. Mark which methods are safe, and for the rest, use an idempotency key or accept that they aren’t auto-retried.

gRPC vs REST: Choosing

Not a rivalry, different tools. The honest decision rule: gRPC for internal, high-throughput, typed service-to-service traffic; REST/JSON for public, browser-facing, human-debuggable, cache-friendly APIs. Most real systems run both: gRPC between backend services, a REST (or GraphQL) edge for the outside world.

DimensiongRPCREST / JSON
PayloadBinary protobuf, compact, fastText JSON, bulky, human-readable
ContractEnforced by .proto; codegenConvention + docs (OpenAPI optional)
TransportHTTP/2 only (multiplexed)Any HTTP, incl. 1.1
StreamingFirst-class, bidirectionalBolt-ons (SSE, polling, WebSockets)
Browser supportNo native, needs gRPC-Web (sec 19)Universal
Human-debuggableNeeds tooling (grpcurl)curl, browser, devtools
HTTP cachingNot reallyMature (ETag, Cache-Control)
Best fitMicroservices, internal, low-latency, polyglotPublic APIs, web/mobile front doors, third parties

gRPC-Web & Browser Interop

A hard constraint, and a frequent surprise: browsers cannot speak native gRPC. The reason is from sec 06, gRPC needs fine-grained control over HTTP/2 frames and trailers, and browser fetch/XHR don’t expose that. So talking to gRPC from a web frontend requires a translation layer.

BrowsergRPC-Web clientProxyEnvoy / gatewaygRPC servicenative gRPCgRPC-WebHTTP/1.1-friendlynative gRPCHTTP/2 + trailers
Bridging to the browser

Your options, roughly in order of how much you want gRPC on the frontend:

  • gRPC-Web: a variant protocol plus a generated JS/TS client; a proxy (Envoy has a built-in filter) translates it to real gRPC. Streaming is limited (server-streaming works; full bidirectional generally doesn’t).
  • Connect (connectrpc), a modern protocol family that speaks gRPC, gRPC-Web, and its own HTTP/JSON, often without a separate proxy, from the same handlers. Increasingly the friendliest path.
  • grpc-gateway / transcoding: generate a REST+JSON facade from your .proto (via HTTP annotations). The browser uses plain REST; the gateway transcodes to gRPC. This is the “REST edge from one proto” pattern referenced in sec 18.

Plan the edge before committing

If a browser must call your service directly, decide the bridge strategy up front, you can’t just point fetch at a gRPC port. For internal-only services this never matters; for anything web-facing it’s a first-class design decision.

Observability & Debugging

Binary payloads mean you can’t eyeball traffic like JSON (sec 05), so gRPC ships an ecosystem to see inside. Knowing these turns a gRPC service from a black box into something you can poke, trace, and probe.

Tool / featureWhat it gives you
grpcurlThe curl of gRPC, call methods from the CLI with JSON in/out, list services and methods.
Server reflectionLets clients/tools discover a server’s services & message schemas at runtime, so grpcurl works without the .proto on hand.
Health checkingA standard grpc.health.v1.Health service for readiness/liveness probes (K8s, load balancers).
channelzBuilt-in introspection of live channels, connections, and per-RPC stats for debugging connectivity.
Interceptors (sec 15)The hook for metrics (Prometheus), structured logs, and distributed tracing (OpenTelemetry) on every call.
Trace propagationPass a trace/request ID through metadata (sec 13) so one request is followable across every service it touches.
# List every service the server exposes (needs reflection enabled):
grpcurl localhost:50051 list

# List the methods of one service:
grpcurl localhost:50051 list user.v1.UserService

# Call a unary method with a JSON request, grpcurl turns it into protobuf for you:
grpcurl -d '{"id": "u42"}' localhost:50051 user.v1.UserService/GetUser

# Against a TLS server, drop -plaintext; for local insecure servers, add it:
grpcurl -plaintext -d '{"id":"u42"}' localhost:50051 user.v1.UserService/GetUser

Enable reflection in non-prod

Turn on server reflection in dev/staging so grpcurl and GUI tools (like Postman’s gRPC mode or grpcui) can explore your API without you shipping .proto files around. Many teams disable it in production to avoid advertising the schema, a small attack-surface decision.

Debug Cheat-Sheet

The whole manual compressed to what you reach for under pressure.

ConceptOne-liner
gRPCTyped remote method calls, Protobuf contract, binary serialization, HTTP/2 transport.
Protobuf (IDL)The .proto is the contract; both sides generate code from it.
Field numbersThe real wire identity, never change or reuse one; adding fields is safe.
Wire formattag = (field«3)|wiretype, varint-packed; small + fast, not human-readable.
HTTP/2One connection, many interleaved streams; one RPC = one stream.
Unary1->1, the normal call (~90% of RPCs).
Server stream1->many, feeds, big result sets.
Client streammany->1, uploads, batch aggregation.
Bidi streammany<->many, chat, realtime, full-duplex.
Channel vs stubChannel = reused connection (one, shared); stub = cheap per-call caller.
MetadataKey-value side-channel = HTTP headers; carries tokens & trace IDs.
DeadlineAbsolute time bound, propagates across hops, set one on every call.
Status codesgRPC’s own enum (OK, NOT_FOUND, UNAVAILABLE…), in a trailer, not HTTP codes.
InterceptorsgRPC middleware, auth, logging, tracing, recovery; unary & stream.
SecurityTLS/mTLS for the pipe + token-in-metadata for the user; never insecure in prod.
Load balancingUse L7 / client-side, an L4 TCP balancer pins everything to one backend.
RetriesDeclarative via service config; only auto-retry idempotent methods.
BrowserNo native gRPC, use gRPC-Web, Connect, or a REST gateway.
Debuggrpcurl + reflection; channelz for connections; health service for probes.
vs RESTgRPC = internal/typed/fast; REST = public/debuggable/cacheable. Run both.

The whole topic in one breath: gRPC lets a client call a server’s method as if it were local. You define the API once in a .proto (sec 04), protoc generates a typed stub and server skeleton (sec 11), arguments are serialized as compact protobuf (sec 05) and carried over multiplexed HTTP/2 streams (sec 06) in one of four shapes, unary, server-, client-, or bidirectional-streaming (sec 07-10). A long-lived channel hosts cheap per-call stubs (sec 12); metadata carries auth and trace context, deadlines bound and propagate the call, cancellation reclaims work (sec 13); every call ends with a gRPC status code (sec 14); interceptors centralize cross-cutting logic (sec 15); TLS/mTLS plus per-call tokens secure it (sec 16); declarative retries and L7/client-side balancing make it resilient (sec 17). Reach for gRPC inside your system and REST at the public edge (sec 18-19), and lean on grpcurl, reflection and interceptors to see inside (sec 20).

Grounded in grpc.io & protobuf.dev docs / MDN for HTTP/2 background / Go 1.22+ (google.golang.org/grpc), Python 3.11+ (grpcio), Node.js 20+ (@grpc/grpc-js), TypeScript 5+ (ts-proto) and Java 17+ (io.grpc) examples.

Backend from First Principles / Chapter 13 / gRPC. Code targets Go 1.22+, Python 3.11+, Node.js 20+, TypeScript 5+ and Java 17+.